Content generation device, mixed reality device, content generation system, content generation method, program, and storage medium

Abstract

To provide a content generation device capable of more easily generating an instruction related to a task, a mixed reality device, a content generation system, a content generation method, a program, and a storage medium.SOLUTION: A content generation device according to an embodiment is configured to: recognize a voice of a worker during a task; generate an instruction related to the task on the basis of recognition results of the voice; record the instruction in association with data related to the task; preferably generate a prompt using the recognized voice; and acquire a summary of the voice and record the summary as the instruction by inputting the prompt into a language model including a neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a content generation apparatus, a mixed reality device, a content generation system, a content generation method, a program, and a storage medium. [Background technology]

[0002] When a beginner with little knowledge or experience works, it is desirable for an expert to be able to instruct the beginner. For example, a task may be performed while an expert instructs a newcomer. To reduce the manpower or labor required for instruction, it is preferable that at least a part of the instruction be automated. To automate the instruction, it is necessary to prepare content including instructions regarding the task in advance. There is a demand for technology that can reduce the effort required for preparing content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a content generation device, a mixed reality device, a content generation system, a content generation method, a program, and a storage medium that can more easily generate work instructions. [Means for solving the problem]

[0005] A content generator according to an embodiment recognizes the voice of a worker who is performing a task. The content generator generates instructions related to the task based on the results of the voice recognition. The content generator records the instructions in association with data on the task. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram illustrating the functions of a content generation device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating the operation. [Diagram 3] 3(a) and 3(b) are schematic diagrams illustrating the state of work. [Figure 4] 4(a) and 4(b) are diagrams for explaining the process performed by the content generation device according to the embodiment. [Diagram 5] 5(a) and 5(b) are diagrams for explaining the process performed by the content generation device according to the embodiment. [Figure 6] 6(a) and 6(b) are diagrams for explaining the process performed by the content generation device according to the embodiment. [Figure 7] FIG. 7 is a table showing an example of data registered in the database. [Figure 8] FIG. 8 is a flowchart showing a content generating method according to the embodiment. [Figure 9] 9(a) and 9(b) are diagrams for explaining the process performed by the content generation device according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram illustrating a content generation system according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram illustrating a mixed reality device. [Figure 12] FIG. 12 is a schematic diagram illustrating a work site. [Figure 13] FIG. 13 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Figure 14] FIG. 14 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram illustrating the state of work. [Figure 16] FIG. 16 is a schematic diagram showing an example of a tool. [Figure 17]FIG. 17 is a schematic diagram illustrating the state of work. [Figure 18] FIG. 18 is a diagram for explaining a method of calculating the center coordinates. [Figure 19] FIG. 19 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be shown differently depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] An embodiment of the present invention is directed to generating instructions for a task from the voice of a worker when the task is being performed. After the instructions for the task are generated, another worker can refer to the instructions while performing the task. This allows an inexperienced worker to perform the task more efficiently or safely.

[0009] FIG. 1 is a block diagram illustrating the functions of a content generation device according to an embodiment. As shown in FIG. 1, the content generator 10 has the functions of an acquisition unit 11, a prompt generation unit 12, a prompt processing unit 13, a display unit 14, and a recording unit 15.

[0010] The content generating device 10 is connected to a voice input device such as a microphone. The acquiring unit 11 acquires voice input to the voice input device. The acquiring unit 11 may acquire voice data stored in a server on a network or the like. The voice includes the contents of the speech of the worker while working. The acquired voice may further include voice before the start of work or after the end of work.

[0011] The acquisition unit 11 recognizes the acquired voice. Voice data is converted into text data by voice recognition. For example, the acquisition unit 11 uses a voice recognition model 21 including a neural network. The voice recognition model 21 includes an acoustic model, a language model, and the like, and outputs text data in response to voice input. The acquisition unit 11 inputs voice to the voice recognition model 21, and acquires the text data output from the voice recognition model 21.

[0012] The prompt generation unit 12 and the prompt processing unit 13 are functions for generating easier to understand or more appropriate instructions from the recognized voice. The recognized voice may include calls, responses, calls, etc., in addition to instructions related to the work. Content other than the instructions is not essential for the work. The prompt generation unit 12 generates a prompt to summarize the voice recognition results. The summary succinctly expresses the instructions related to the work.

[0013] For example, the prompt generation unit 12 generates a prompt for summarizing the recognition result according to a preset rule. The prompt processing unit 13 inputs the generated prompt to a language model 22 prepared in advance. As the language model 22, a large-scale language model including a neural network (e.g., ChatGPT) is preferably used. The language model 22 processes the input prompt. The prompt processing unit 13 acquires the summary output from the language model 22.

[0014] The display unit 14 displays the summary on a display device so that the user can confirm the summary. The user is, for example, a person involved in the generation of content and an expert who is accustomed to the work. The user can modify the displayed summary using an input device. The display unit 14 accepts the modifications to the summary.

[0015] The summary obtained by the prompt processing unit 13 or the summary corrected by the user corresponds to an instruction related to the task. If no corrections are made to the summary, the recording unit 15 links the summary output from the language model 22 to the task data and records it in the database 23. If a correction is made to the summary, the recording unit 15 links the corrected summary to the task data and records it in the database 23.

[0016] FIG. 2 is a schematic diagram illustrating the operation. In the example shown in Fig. 2, two workers W1 and W2 are performing a screw tightening operation on an article 100. Cylindrical members 110-130 are placed on the disk-shaped article 100. Fastening points 101 (screw holes) for fixing the members 110-130 are present on the bottom and top surfaces of the article 100. The workers W1 and W2 turn the screws into each fastening point 101 in turn.

[0017] 3(a) and 3(b) are schematic diagrams illustrating the state of work. As an example, worker W1 is an experienced worker who is accustomed to the work, and worker W2 is a newcomer who is not accustomed to the work. During the work, worker W1 proceeds with the work while speaking to worker W2. For example, as shown in FIG. 3(a), worker W1 instructs worker W2 to pay attention to the perpendicularity of the screws by utterance S1. In response to utterance S1, worker W2 pays attention to the screws. As shown in FIG. 3(b), worker W2 responds to utterance S1 by utterance S2.

[0018] FIGS. 4(a), 4(b), 5(a), 5(b), 6(a), and 6(b) are diagrams for explaining the processing by the content generation device according to the embodiment. The utterances S1 and S2 shown in Fig. 3(a) and Fig. 3(b) are acquired by a voice input device. The acquisition unit 11 acquires voice data from the voice input device and inputs it to the voice recognition model 21. The voice data may be added with the date and time when the voices were acquired. When the utterance by the worker W1 and the utterance by the worker W2 are acquired by different voice input devices, the voice data may be added with information of the device from which the voices were acquired. When the worker who uses each device is registered in advance, the acquisition unit 11 can identify the worker who uttered the voice from the information of the device.

[0019] The acquisition unit 11 acquires text output from the speech recognition model 21. The acquisition unit 11 adds information on the date and time when the voice was acquired and the worker who uttered the voice to the acquired text, and generates text TX1 shown in Fig. 4(a). In the text TX1, text data based on utterance S1 is linked to worker W1, and text data based on utterance S2 is linked to worker W2.

[0020] 4(b), the prompt generating unit 12 adds an instruction i1 for generating a summary from the text TX1, thereby generating text (prompt) TX2 for generating a summary.

[0021] The prompt processing unit 13 inputs the text TX2 into the language model 22. The prompt processing unit 13 obtains the result output from the language model 22. As an example, the text TX3 shown in FIG. 5(a) is output from the language model 22. The display unit 14 displays the text TX3 to the user and accepts corrections from the user. For example, the user deletes a part of the text TX3 to shorten the text TX3. As a result, the corrected text TX4 is input as shown in FIG. 5(b). The prompt processing unit 13 accepts the corrections, and the recording unit 15 records the corrected summary in association with the task.

[0022] In modifying the summary, the user may directly edit the summary or may input a prompt for modifying the summary. In this case, the prompt processor 13 accepts the input prompt. The prompt processor 13 inputs the prompt input by the user to the language model 22 and obtains an output result from the language model 22. The display unit 14 displays the output result again to the user. For example, the modification of the summary is repeated until the user determines that no modification is necessary.

[0023] The recorded content (instructions) is used the next time the same work is performed. For example, when worker W2 or another new worker performs work on the item 100, the recorded instructions are automatically output. The instructions may be output by voice or may be displayed.

[0024] When generating a prompt, the prompt generation unit 12 may delete a portion of the text from the speech recognition result. For example, the database 23 registers the start date of each worker's work. The prompt generation unit 12 refers to the start date of each worker for the work being performed. The prompt generation unit 12 calculates the period from the start date to the present, and compares the period with a preset threshold. If the period is less than the threshold, the prompt generation unit 12 determines that the worker is a novice. If the period is equal to or greater than the threshold, the prompt generation unit 12 determines that the worker is an expert. The prompt generation unit 12 deletes the text based on the utterance of the novice from the speech recognition result.

[0025] As an example, as shown in Fig. 6(a), the speech recognition result is associated with the worker, and text TX5 is generated. The prompt generation unit 12 refers to the work start dates of each of workers W1 and W2. As a result, worker W1 is determined to be an expert, and worker W2 is determined to be a novice. The prompt generation unit 12 deletes the speech of worker W2 to generate text TX6, as shown in Fig. 6(b).

[0026] Alternatively, the prompt generation unit 12 may calculate the period from the start date to the present for each worker, and determine the worker with the longer period as an expert and the worker with the shortest period as a novice. When there are three or more workers, the prompt generation unit 12 determines the worker with the longest period as an expert and the worker with the shortest period as a novice. The other workers may be determined as either experts or novice.

[0027] Or, more directly, data indicating the proficiency of each worker may be registered in the database 23. The prompt generation unit 12 refers to the proficiency registered in advance and deletes text based on the speech of a new worker.

[0028] FIG. 7 is a table showing an example of data registered in the database. A table 200 shown in FIG. 7 is stored in the database 23. The content generated by the content generating device 10 is associated with the work in the table 200. Specifically, the table 200 includes a work ID 201, a fastening point ID 202, a position 203, a torque value 204, an order 205, and a content ID 206. The work ID 201 describes an ID of a work including fastening a screw to the fastening point 101. The fastening point ID 202 describes an ID of a point where a screw is fastened. The position 203 indicates the coordinates of the fastening point identified by the fastening point ID. The torque value 204 indicates a torque value required for fastening a screw to each fastening point. The order 205 indicates the order of fastening for a plurality of fastening points that are fastened in one work. The content ID 206 describes an ID for identifying the content generated by the content generating device 10.

[0029] The work to be linked with the generated instruction is specified by the worker. Alternatively, the work being performed may be automatically estimated, and the instruction may be linked with the estimated work. For example, as shown in FIG. 2, a camera device 30 is installed at a work site. The camera device 30 captures images of the work being performed and acquires images. The camera device 30 may acquire continuous images (video). A processing device 40 receives the images acquired by the camera device 30. The processing device 40 estimates the work being performed, the start of the work, the end of the work, etc. from the images. A known method can be used as a method of estimation using images.

[0030] As an example, when the work to be performed is known, the processing device 40 estimates the posture (skeletal information) of the worker from the image. A posture estimation model such as OpenPose, DarkPose, or CenterNet can be used to estimate the posture. The processing device 40 acquires a change in the position of a part of the skeleton as time-series data. The processing device 40 extracts a characteristic pattern (motif) from the time-series data, and estimates the work being performed, the start of the work, and the end of the work using the time-series data and the motif. This estimation method is discussed in JP 2022-003491 A.

[0031] As another example, the processing device 40 estimates the posture of the worker, the position of the item, the orientation of the item, the state of the item, and the like from the image. A posture estimation model is used to estimate the posture. The position of the item is estimated by template matching. The orientation of the item is estimated from the amount of movement of the item relative to a preset initial state, calculated from images repeatedly acquired. The state of the item is estimated using a state estimation model that has been learned in advance. The processing device 40 estimates the work being performed, the start of the work, and the end of the work based on the estimation results. This estimation method is discussed in JP 2023-111521 A.

[0032] The content generator 10 receives estimation results such as the work being performed, the start of the work, and the end of the work from the processing device 40. The content generator 10 generates instructions using the voice from the start of the work to the end of the work. The content generator 10 records the instructions in association with the data of the estimated work.

[0033] The content generation device 10 and the processing device 40 are connected via wired communication, wireless communication, a network, etc. The image capture device 30 and the processing device 40 are connected via wired communication, wireless communication, a network, etc.

[0034] FIG. 8 is a flowchart showing a content generating method according to the embodiment. First, a voice during work is input to the voice input device, and the acquisition unit 11 acquires the voice (step St1). The acquisition unit 11 performs voice recognition (step St2). The prompt generation unit 12 generates a prompt based on the recognized voice, the association between the voice and the worker, etc. (step St3). The prompt processing unit 13 inputs the prompt to the language model 22, and acquires a summary output from the language model 22 (step St4). The display unit 14 displays the summary (step St5). The display unit 14 accepts corrections to the summary as appropriate. The recording unit 15 links the summary (instructions) to the work data and records it in the database 23 (step St6).

[0035] Advantages of the embodiment will be described. When the content creation device 10 according to the embodiment acquires a voice, it generates an instruction for a task based on the result of recognizing the voice. Then, the content creation device 10 records the instruction in association with the task data. This makes it possible to output the instruction associated with the task the next time the task is performed.

[0036] For example, according to an embodiment, as shown in Figs. 3(a) and 3(b), instructions are automatically generated from utterances made by an expert when actually providing instruction. The generated instructions are linked to the work being performed at that time. Instructions for each work are automatically prepared without the expert having to be aware of preparing instructions or linking the instructions to the work. Therefore, even a worker who is not familiar with generating content can easily prepare instructions for the work.

[0037] After the instructions are prepared, the instructions are automatically output when the same task is to be performed. This allows even an inexperienced worker to efficiently perform the task by following the instructions. The task to be performed may be specified by the worker or may be automatically estimated by the method described above.

[0038] The speech recognition result may be recorded as the task instruction, but more preferably, a summary of the recognition result is recorded as the instruction. By generating a summary, at least a portion of the content other than the instruction is omitted from the speech recognition result. The summary can succinctly express the task instruction. This allows the generation of more understandable or more appropriate instructions.

[0039] As shown in Figure 7, the content including the instructions may be more specifically linked to the fastening point. In this case, when a screw is turned into a particular fastening point, the instructions linked to that fastening point are automatically output. This makes it easier for inexperienced workers to understand the instructions and perform the work more efficiently.

[0040] 9(a) and 9(b) are diagrams for explaining the process performed by the content generation device according to the embodiment. The prompt generator 12 may generate a plurality of prompts. For example, when the text TX1 shown in FIG. 4(a) is acquired, the prompt generator 12 adds an instruction i2 different from the example shown in FIG. 4(b) as shown in FIG. 9(a). The instruction i2 is for generating a longer summary (work instruction) than the instruction i1. As a result, the text TX7 is generated.

[0041] The prompt processor 13 inputs the text TX7 to the language model 22 and obtains the text TX8 output from the language model 22, as shown in Fig. 9(b). The display unit 14 displays the text TX8 to the user and accepts corrections from the user. The recording unit 15 records the text TX8 output from the language model 22 or the text TX8 corrected by the user in association with the task.

[0042] The number of characters contained in text TX8 is greater than the number of characters contained in text TX3. Therefore, text TX8 contains more information than text TX3. For example, worker W2 reads text TX8 after completing a task to reconfirm the task. Or, another worker reads text TX8 before the task to reconfirm the task. Text TX8 does not need to be output the next time the same task is performed. According to this method, multiple contents with different uses can be generated from one voice data.

[0043] FIG. 10 is a schematic diagram illustrating a content generation system according to the embodiment. The content generation device according to the embodiment can be used in cooperation with a mixed reality device (MR). As shown in FIG. 10, the content generation system 1 according to the embodiment includes a content generation device 10 and an MR device 300. The MR device 300 acquires the voice of a worker. The content generation device 10 generates content using the acquired voice. The MR device 300 outputs the content to the worker. The MR device 300 has the functions of the imaging device 30 and the processing device 40 shown in FIG. 2.

[0044] FIG. 11 is a schematic diagram illustrating a mixed reality device. The MR device 300 includes a frame 301 , a lens 311 , a lens 312 , a projection device 321 , a projection device 322 , an image camera 331 , a depth camera 332 , a sensor 340 , a microphone 341 , a processing device 350 , a battery 360 , and a storage device 370 .

[0045] In the illustrated example, the MR device 300 is a twin-eye head-mounted display. Two lenses, 311 and 312, are fitted in a frame 301. Projection devices 321 and 322 project information onto the lenses 311 and 312, respectively.

[0046] The projection device 321 and the projection device 322 display the recognition result of the body of the worker, a virtual object, etc. on the lens 311 and the lens 312. Only one of the projection device 321 and the projection device 322 may be provided, and information may be displayed on only one of the lens 311 and the lens 312.

[0047] The lenses 311 and 312 are optically transparent. A person wearing the MR device 300 can view the real situation through the lenses 311 and 312. The person wearing the MR device 300 can also view information projected onto the lenses 311 and 312 by the projection devices 321 and 322. Information (virtual space) is displayed superimposed on the real space by projection by the projection devices 321 and 322.

[0048] The image camera 331 detects visible light and obtains a two-dimensional image. The depth camera 332 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 340 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 341 accepts voice input.

[0049] The processing device 350 controls each element of the MR device 300. For example, the processing device 350 controls the display by the projection device 321 and the projection device 322. The processing device 350 detects a movement of the field of view based on the detection result by the sensor 340. The processing device 350 changes the display by the projection device 321 and the projection device 322 in response to the movement of the field of view. In addition, the processing device 350 can execute various processes using data obtained from the image camera 331 and the depth camera 332, data in the storage device 370, and the like.

[0050] The battery 360 supplies the power required for operation to each element of the MR device 300. The storage device 370 stores data required for processing by the processing device 350, data obtained by processing by the processing device 350, etc. The storage device 370 may be provided outside the MR device 300 and may communicate with the processing device 350.

[0051] The MR device used is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0052] FIG. 12 is a schematic diagram illustrating a work site. Fig. 12 shows the state of work as seen from the worker W2. Each of the workers W1 and W2 wears the MR device 300 shown in Fig. 11. The workers W1 and W2 use a wrench and an extension bar to turn the screws into each fastening point 101 of the article 100.

[0053] A marker 105 is provided near the work target. The marker 105 is an AR marker. As described below, the marker 105 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 105. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 350 sets a three-dimensional coordinate system based on a plurality of points indicated by a hand gesture.

[0054] At the start of the task, the image camera 331 and the depth camera 332 capture images of the marker 105. The processing device 350 recognizes the marker 105 from the captured images. The processing device 350 sets a three-dimensional coordinate system based on the position and orientation of the marker 105.

[0055] 13 and 14 are schematic diagrams showing an example of output by the mixed reality device according to the embodiment. During work, the image camera 331 and the depth camera 332 capture images of the article 100, the worker's left hand 151, and the worker's right hand 152. The processing device 350 recognizes the left hand 151 and the right hand 152 from the captured images by hand tracking. The processing device 350 may display the recognition results on the lenses 311 and 312 by the projection devices 321 and 322. Hereinafter, the processing device displaying information on the lenses using the projection devices will also be simply referred to as "the processing device displaying information."

[0056] 13, for example, the processing device 350 displays the recognition results of the left hand 151 and the right hand 152 superimposed on the hands in real space. In the illustrated example, a plurality of virtual objects 151a and a plurality of virtual objects 152a are displayed as the recognition results of the left hand 151 and the right hand 152. The plurality of virtual objects 151a respectively represent a plurality of joints of the left hand 151. The plurality of virtual objects 152a respectively represent a plurality of joints of the right hand 152. Instead of the joints, virtual objects (meshes) respectively representing the surface shapes of the left hand 151 and the right hand 152 may be displayed.

[0057] When the processing device 350 recognizes the left hand 151 and the right hand 152, it measures the coordinates of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The coordinates of any of these joints are used as the coordinates of the hand. The center of gravity of the multiple joints may be used as the coordinates of the hand. Alternatively, the center coordinates of the entire hand may be used as the coordinates of the hand.

[0058] The processing device 350 may display virtual objects 161a and 161b as shown in Fig. 14. The virtual objects 161a and 161b are displayed corresponding to one fastening point 101. The virtual object 161a is displayed at a position away from the fastening point 101. The virtual object 161b is displayed between the fastening point 101 and the virtual object 161a. The virtual object 161b indicates which fastening point the virtual object 161a corresponds to.

[0059] FIG. 15 is a schematic diagram illustrating the state of work. For example, as shown in FIG. 15 , a wrench 180 and an extension bar 190 are used to turn a screw into a fastening point 101. When tightening a screw into a fastening point 101, an operator places a screw into a threaded hole of the fastening point 101. The operator fits one end of the extension bar 190 onto the screw. The operator fits the head of the wrench 180 onto the other end of the extension bar 190. The operator holds the head of the wrench 180 with one hand and holds the grip of the wrench 180 with the other hand. By turning the wrench 180, the screw is tightened into the fastening point 101 via the extension bar 190.

[0060] At this time, the worker positions the extension bar 190 so that the extension bar 190 is close to or in contact with the virtual object 161b. The worker also grips the head of the wrench 180 so that the hand comes into contact with the virtual object 161a. By displaying the virtual object, the worker can easily grasp the positions of the tool and the hand when turning the screw into the fastening point 101. This can improve the work efficiency.

[0061] In the illustrated example, virtual object 161a is spherical, and virtual object 161b is rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each object is not limited to this example. For example, virtual object 161a may be a cube, and virtual object 161b may be a line.

[0062] The virtual objects 161a and 161b are displayed at pre-registered positions in a three-dimensional coordinate system that is set based on the marker 105. Alternatively, the positions of the fastening points 101, data on the tools to be used, and the like may be registered in advance, and the display positions of the virtual objects 161a and 161b may be calculated using the data. For example, the virtual object 161b is displayed between the fastening point 101 and a position that is separated from the fastening point 101 by the length of the extension bar 190. The virtual object 161a is displayed at a position that is separated from the fastening point 101 by the length of the extension bar 190.

[0063] The virtual objects 161a and 161b are sequentially displayed at each fastening point in accordance with a preset fastening order. That is, after a screw is turned at the fastening point 101 at which the virtual objects 161a and 161b are displayed, the virtual objects 161a and 161b are displayed at another fastening point 101.

[0064] After the virtual object is displayed, the processing device 350 may determine whether a predetermined object has come into contact with the virtual object. For example, the processing device 350 determines whether a hand has come into contact with the virtual object 161a. Specifically, the processing device 350 calculates the distance between the coordinates of the hand and the virtual object 161a. If either distance is less than a preset threshold, the processing device 350 determines that the hand has come into contact with the virtual object. As an example, in FIG. 14, the diameter of the virtual object 161a (sphere) corresponds to the threshold. The sphere indicates the range in which it is determined that the hand has come into contact with the virtual object.

[0065] FIG. 16 is a schematic diagram showing an example of a tool. The processing device 350 may determine whether the tool has come into contact with the virtual object 161a. For example, as shown in FIG. 16, a plurality of markers 181 are attached to the wrench 180. The processing device 350 recognizes the plurality of markers 181 from an image captured by the image camera 331. The processing device 350 measures the coordinates of each marker 181. The positional relationship between the plurality of markers 181 and the head 182 of the wrench 180 is registered in advance. The processing device 350 calculates the coordinates of the head 182 based on the coordinates of at least three recognized markers 181 and the positional relationship registered in advance. The processing device 350 calculates the distance between the coordinates of the head 182 and the virtual object 161a. If any of the distances is less than a preset threshold, the processing device 350 determines that the wrench 180 has come into contact with the virtual object.

[0066] If the distance is less than the threshold, it can be estimated that a screw is being turned at the fastening point 101 corresponding to the one of the virtual objects 161a. In the example shown in Fig. 15, it is estimated that a task is being performed at the fastening point 101 corresponding to the virtual object 161a due to contact between the hand and the virtual object 161a.

[0067] The start of the work may be estimated based on the distance between a predetermined object and a virtual object. When the distance is first less than a threshold, the processing device 350 estimates that the work has started. The content generation device 10 acquires a voice input to the microphone 341 after the work has started. The content generation device 10 links an instruction generated based on the voice to data of the fastening location where the work is estimated to be performed. For example, as shown in FIG. 7, the instruction data is linked to an ID of the fastening location. Alternatively, the content generation device 10 may identify a work ID including tightening a screw into the estimated fastening location, and link the instruction to the work ID.

[0068] When the start of work is estimated, not only the voice after the start of work but also the voice for a predetermined period of time before the start of work may be acquired. The content generating device 10 links the instructions generated based on the voice before the start of work and the voice after the start of work to the data of the fastening location. When an expert is providing guidance, there are cases where instructions are spoken before the start of work. By acquiring the voice before the start of work as well, the instructions issued before the start of work can also be included in the generated content.

[0069] For example, if the state in which the distance is longer than the threshold continues for a period of time longer than a preset period of time after the start of the work is estimated, the processing device 350 estimates that the work is completed. A digital tool capable of detecting torque may be used for the work. In this case, the processing device 350 receives the torque detected from the tool. If a required torque value is preset for each fastening point, the work may be estimated to be completed at the timing when the required torque value is detected by the tool.

[0070] FIG. 17 is a schematic diagram illustrating the state of work. Another method may be used to estimate the location being worked on and the start of work. In another method, the movement of the worker's hand is used. While the worker is turning the wrench 180, the processing device 350 repeatedly measures the coordinates of the hand. At this time, as shown in FIG. 17, the hand is positioned on a circumference with a part of the wrench 180 as the center. The hand is moved to draw an arc. Using this, the processing device 350 calculates the center coordinates of the tool rotation. A specific example of the calculation of the center coordinates will be described below.

[0071] FIG. 18 is a diagram for explaining a method of calculating the center coordinates. The processing device 350 extracts three mutually different coordinates from the measured coordinates. The processing device 350 calculates the circumcenter O of the three coordinates. Here, as shown in FIG. 18, the three coordinates are respectively denoted by P 1 (x 1 ,y 1 ,z 1 ), P 2 (x2 ,y 2 ,z 2 ), and P 3 (x 3 ,y 3 ,z 3 ) The coordinates of the circumcenter O are P 0 (x 0 ,y 0 ,z 0 ) and the coordinate P 1 ~P 3 For the triangle obtained by connecting the two, coordinate P 1 The length of the opposite side of L 1 Coordinate P 2 The length of the opposite side of L 2 Coordinate P 3 The length of the opposite side of L 3 Coordinate P 1 The angle at the coordinate P is α. 2 The angle at the coordinate P is β. 3 The angle at is γ. In this case, the coordinates of the circumcenter O are expressed by the following formula (1). Note that in formula (1), a symbol with an arrow indicates a position vector. Formula (1) can be rewritten as formula (2). Formula (2) can be decomposed into formulas (3) to (5).

number

number

number

number

number

[0072] From formulas (3) to (5), x 0 ,y 0 ,z 0The processing device 350 calculates the coordinates P 0 (x 0 ,y 0 ,z 0 ) is calculated as the center coordinate of rotation of the wrench 180.

[0073] The processing device 350 may extract multiple combinations of the three coordinates. The processing device 350 calculates the coordinates of the circumcenter for each combination. The processing device 350 calculates the median, average, or mode of the multiple coordinates as the central coordinate of the rotation of the wrench 180. This can improve the accuracy of the calculated central coordinates.

[0074] Before executing the above method, a part of coordinates may be selected from the multiple coordinates of the hand. The above-mentioned calculation is performed using the selected coordinates. For example, only the coordinates when the hand is estimated to be moving in an arc are selected, and the coordinates used for the calculation are extracted from these coordinates. By using only the coordinates when the hand is moving in an arc, the accuracy of the center coordinate can be further improved.

[0075] As an example, when a digital tool is used, the processing device 350 selects the hand coordinates obtained at the timing (time) when data is received from the digital tool. When a screw is turned with the digital tool, the torque value, rotation angle, and the like are detected by the digital tool. The processing device 350 receives these detection values ​​detected by the digital tool. Receiving the detection values ​​indicates that the hand is moving in an arc. Therefore, the hand coordinates obtained at the timing (time) when the detection values ​​are received are selected and used to calculate the center coordinates, so that the center coordinates can be calculated more accurately.

[0076] The central coordinate of the rotation of the wrench 180 can be regarded as the position of the screw being tightened by the wrench 180. The screw is fastened to a predetermined location of the article. Therefore, by estimating the position of the screw, it can be estimated that the screw is fastened to the fastening location closest to the central coordinate. For example, the processing device 350 calculates the distance between the central coordinate and the closest fastening location, and compares the distance with a preset threshold. If the state in which the distance is less than the threshold continues for longer than a preset time, the processing device 350 estimates that work is being performed on the closest fastening location. In addition, the timing when the distance first becomes less than the threshold after it is estimated that work is being performed can be estimated as the start of work.

[0077] For example, if the state in which the distance is longer than the threshold continues for a period of time longer than a preset time, the processing device 350 estimates that the work is completed. If a digital tool is used and a required torque value is preset for each fastening point, the work may be estimated to be completed when the required torque value is detected by the tool.

[0078] When a digital tool is used, the MR device 300 may receive rotation angles, torque, time, and the like from the tool and associate these data with the data of the estimated fastening points, allowing a detailed work record to be automatically created.

[0079] In any of the estimation methods described above, an image captured by the image camera 331 is used. The position of the hand or the position of the tool is measured from the image, and the position is used to determine whether a predetermined object contacts a virtual object or to calculate the center coordinates. The result of the contact determination or the result of the center coordinate calculation is used to estimate the work being performed, the start of the work, etc.

[0080] According to the content generation system of the embodiment, by using the MR device 300, even a newcomer can efficiently proceed with the work. In addition, while working, the MR device 300 automatically estimates that the work is being performed and that the work has started. The content generation apparatus 10 uses the estimation results to automatically link the generated instructions to the work. According to the embodiment, even an inexperienced worker can efficiently perform the work and automatically generate content including instructions while performing the work.

[0081] After the instruction is generated by the content generation apparatus 10, the MR device 300 may output the instruction. For example, the MR device 300 projects the instruction onto the lens 311 and the lens 312. Alternatively, the MR device 300 may output the instruction by voice using a speaker (not shown).

[0082] An embodiment of the present invention may be applied to the task of loosening a screw. When loosening a screw, a tool is used to turn the screw as shown in Fig. 15. In this case, the task being performed and the fastening location where the screw is being turned may be estimated using an image. The content generation device 10 generates instructions from the voice during the task based on the estimation results.

[0083] The embodiment of the present invention may be applied to tasks other than screwing. For example, the task may be assembling an item, disassembling an item, or transporting an item. The content generation device 10 recognizes the voice of the worker during the task and generates instructions related to the task. This makes it possible to output instructions linked to the task the next time the task is performed.

[0084] FIG. 19 is a schematic diagram showing a hardware configuration. 19 is used as each of the content generation device 10, the processing device 40, and the processing device 350. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0085] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0086] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0087] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) or the like.

[0088] The input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95.

[0089] An output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the output device via the output I / F 96 and cause the output device to output information.

[0090] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0091] Data processing by the content generation device 10, the processing device 40, and the processing device 350 may be executed by only one computer 90. Part of the data processing may be executed by a server or the like via the communication I / F 97. In the example shown in Fig. 1 and Fig. 2, one computer may perform the processing executed by the content generation device 10 and the processing device 40, and may have the functions of both the content generation device 10 and the processing device 40. Part of the processing executed by the processing device 350 in the example shown in Fig. 10 may be executed by the content generation device 10.

[0092] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0093] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0094] Embodiments of the invention include the following features. (Feature 1) Recognizes the voice of the worker while working, generating instructions regarding the task based on the speech recognition result; A content generating device that records the instruction in association with the work data. (Feature 2) generating a prompt using the recognized speech; obtaining a summary of the speech by inputting the prompt into a language model including a neural network; The content generation device according to feature 1, further comprising: recording the summary as the instructions. (Feature 3) accepting modifications to said summary from a user; 3. The content generating apparatus according to feature 2, further comprising: if the summary is modified, recording the modified summary as the instruction. (Feature 4) In the above operation, a screw is turned into the article, 4. The content generation device according to any one of features 1 to 3, wherein the voice is acquired after the start of the task is estimated. (Feature 5) 5. The content generation device according to feature 4, wherein the start of the task is estimated using an image depicting the task. (Feature 6) The location where the screw is being turned is estimated; 6. The content generating device according to feature 4 or 5, wherein the instruction is linked to data of the estimated location and recorded. (Feature 7) A mixed reality device capable of displaying a virtual object overlaid on a real space, A mixed reality device that outputs the instruction recorded by the content generation apparatus according to any one of features 1 to 6 when the task is performed. (Feature 8) A content generating device according to feature 1; A mixed reality device capable of displaying a virtual object superimposed on a real space, the mixed reality device estimates a location where the screw is being turned based on a result of hand tracking during the task of turning a screw into an object; The content generation device links the instruction to data of the estimated location and records it. (Feature 9) On the computer, Recognize the voice of the worker while working, generating instructions regarding the task based on the speech recognition result; The instruction is recorded in association with the work data. Content generation methods. (Feature 10) A program for causing the computer to execute the content generation method according to feature 9. (Feature 11) A storage medium storing the program according to feature 10.

[0095] According to the embodiments described above, a content generation apparatus, a mixed reality device, a content generation system, a content generation method, a program, and a storage medium are provided that can more easily generate instructions related to a task.

[0096] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0097] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0098] 1: content generation system, 10: content generation device, 11: acquisition unit, 12: prompt generation unit, 13: prompt processing unit, 14: display unit, 15: recording unit, 21: voice recognition model, 22: language model, 23: database, 30: imaging device, 40: processing device, 90: computer, 91: CPU, 92: ROM, 93: RAM, 94: storage device, 95: input interface, 96: output interface, 97: communication interface, 98: system bus, 100: article, 101: fastening point, 105: marker, 110-130: member, 151: left hand, 152: right hand, 151a, 152a, 161a, 161b: virtual object, 180: wrench, 181: marker, 182: head, 190: extension bar, 200: table, 201: task ID, 202: fastening point ID, 203: position, 204: torque value, 205: order, 206: content, 300: MR device, 301: frame, 311, 312: lenses, 321, 322: projection device, 331: image camera, 332: depth camera, 340: sensor, 341: microphone, 350: processing device, 360: battery, 370: storage device, S1, S2: speech, TX1 to TX8: text, W1, W2: worker

Claims

1. Recognizes the voice of the worker while working, generating instructions regarding the task based on the speech recognition result; A content generating device that records the instruction in association with the work data.

2. generating a prompt using the recognized speech; obtaining a summary of the speech by inputting the prompt into a language model including a neural network; The content generating device of claim 1 , further comprising: a processor configured to record the summary as the instruction.

3. accepting modifications to said summary from a user; The content generating device according to claim 2 , further comprising: if the summary is modified, the modified summary is recorded as the instruction.

4. In the above operation, a screw is turned into the article, 4. The content generation device according to claim 1, wherein the voice is acquired after the start of the task is estimated.

5. The content generation device according to claim 4 , wherein the start of the task is estimated using an image depicting the task.

6. The location where the screw is being turned is estimated; The content generating device according to claim 4 , wherein the instruction is associated with data of the estimated location and recorded.

7. A mixed reality device capable of displaying a virtual object overlaid on a real space, A mixed reality device that outputs the instructions recorded by the content generation apparatus according to any one of claims 1 to 3 when the task is performed.

8. The content generating device according to claim 1 ; A mixed reality device capable of displaying a virtual object superimposed on a real space, the mixed reality device estimates a location where the screw is being turned based on a result of hand tracking during the task of turning a screw into an object; The content generation device links the instruction to data of the estimated location and records it.

9. On the computer, Recognize the voice of the worker while working, generating instructions regarding the task based on the speech recognition result; The instruction is recorded in association with the work data. Content generation methods.

10. A program for causing the computer to execute the content generating method according to claim 9.

11. A storage medium storing the program according to claim 10.

Images