Processing device, learning device, processing system, processing method, program, and storage medium

Abstract

To provide a processing device, a learning device, a processing system, a processing method, program, and a storage medium capable of automatically acquiring data available for learning.SOLUTION: According to an embodiment, a processing device acquires images, coordinates and dialogue data communicated between a first device used by a first person performing work and a second device used by a second person. The processing device extracts a portion of a plurality of coordinates on the basis of the dialogue data. The processing device associates the extracted coordinates with the work.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a processing device, a learning device, a processing system, a processing method, a program, and a storage medium. [Background technology]

[0002] In some cases, a task is performed while one person is instructing another person. In order to reduce the number of people or labor required for instruction, it is preferable to automate at least a part of the instruction. In order to automate the instruction, it is necessary to accumulate data that can be used for learning for the instruction. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-51519 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a processing device, a learning device, a processing system, a processing method, a program, and a storage medium that are capable of automatically acquiring data that can be used for learning. [Means for solving the problem]

[0005] A processing device according to an embodiment acquires images, coordinates, and dialogue data communicated between a first device used by a first person performing a task and a second device used by a second person. The processing device extracts a portion of the coordinates based on the dialogue data. The processing device associates the extracted coordinates with the task. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing a processing system according to an embodiment. [Diagram 2]FIG. 2 is a schematic diagram illustrating a head mounted display. [Diagram 3] FIG. 3 is a schematic diagram illustrating a head mounted display. [Figure 4] FIG. 4 is a flowchart showing the processing by the processing system according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram showing an example of a dialogue during work. [Figure 6] FIG. 6 is a schematic diagram showing an example of a dialogue during work. [Figure 7] FIG. 7 is a schematic diagram showing an example of a dialogue during work. [Figure 8] FIG. 8 is a flowchart showing a specific example of a scenario. [Figure 9] FIG. 9 is a table illustrating the extracted data. [Figure 10] FIG. 10 is a flowchart showing a specific example of processing by the processing device according to the embodiment. [Figure 11] FIG. 11 is a table illustrating the learning data. [Figure 12] FIG. 12 is a flowchart showing the process performed by the learning device according to the embodiment. [Figure 13] 13(a) to 13(c) are schematic diagrams illustrating examples of learning data. [Figure 14] FIG. 14 is a flowchart showing the processing by the processing device according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram showing an example of a dialogue in another task. [Figure 16] FIG. 16 is a schematic diagram showing an example of a dialogue in another task. [Figure 17] FIG. 17 is a schematic diagram showing an example of a dialogue in another task. [Figure 18] FIG. 18 is a flowchart showing a specific example of a scenario. [Figure 19] FIG. 19 is a table illustrating the extracted data. [Figure 20]FIG. 20 is a flowchart showing a specific example of processing by the processing device according to the embodiment. [Figure 21] 21(a) to 21(d) are examples of learning data. [Figure 22] FIG. 22 is a flowchart showing the process performed by the learning device according to the embodiment. [Figure 23] FIG. 23 is a flowchart showing the processing by the processing device according to the embodiment. [Figure 24] FIG. 24 is a schematic diagram illustrating a pointing means in the embodiment. [Diagram 25] 25(a) to 25(d) are schematic diagrams illustrating the pointing means in the embodiment. [Figure 26] FIG. 26 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals and detailed explanations are omitted as appropriate.

[0008] When a beginner with little knowledge or experience works, it is desirable for an expert to be able to instruct the beginner. However, due to a decrease in the number of experts, economic globalization, the spread of infectious diseases, and other factors, it may be difficult for people to gather in the same place. In such cases, virtual reality (VR) technology and mixed reality (MR) technology using a head-mounted display (HMD) are useful. VR technology or MR technology allows one person's field of view to be shared with other people. The application of VR technology or MR technology allows people who are far away to communicate more smoothly. Experts can instruct beginners more efficiently and in a more understandable manner.

[0009] On the other hand, in order to reduce the number of people or labor required for the work, it is preferable to automate at least a part of the instruction for beginners. In order to automate the instruction, it is necessary to accumulate a lot of data that can be used for learning for that purpose. The invention according to the embodiment can be used to automatically acquire data.

[0010] FIG. 1 is a schematic diagram showing a processing system according to an embodiment. As shown in FIG. 1, a processing system 10 according to the embodiment includes a first device 1, a second device 2, a processing device 3, a storage device 4, and a learning device 5.

[0011] In the illustrated example, the first device 1 and the second device 2 are HMDs. The HMD includes a display device, an imaging device, an input device (microphone), a sensor, a processing circuit, etc. The first device 1 and the second device 2 communicate data with each other via the processing device 3.

[0012] The processing device 3 stores the data received from the first device 1 and the second device 2 in the storage device 4. The processing device 3 also generates data for learning using the data stored in the storage device 4. The learning device 5 executes learning using the generated data. The first device 1, the second device 2, the processing device 3, the storage device 4, and the learning device 5 are connected to each other via wireless communication or a network.

[0013] 2 and 3 are schematic diagrams illustrating a head mounted display. 2 can be used as the first device 1. The first device 1 is, for example, an MR device. The HMD 100 includes a band 102, a display 104, an image capture device 110, an image capture device 120, a microphone 130, a speaker 132, a battery 134, a sensor 136, and a control device 140.

[0014] The HMD 100 is worn on a person's head. The band 102 is flexible and can be attached to and detached from the person's head. The display 104 is attached to the band 102 and displays information. The display 104 is optically transparent, and a projection device (not shown) may project information onto the display 104.

[0015] The imaging device 110 captures an image of the scene in front of the display 104. For example, the imaging device 110 includes an image camera 111 and a depth camera 112. The image camera 111 captures a two-dimensional RGB image based on visible light reflected from an object. The depth camera 112 irradiates an object with infrared light and captures a depth image based on the infrared light reflected from the object.

[0016] The imaging device 120 captures an image of a person's eyes. For example, the imaging device 120 includes an image camera 121 and a depth camera 122. The image camera 111 captures an RGB image of the eye. The depth camera 112 captures a depth image of the eye. The microphone 130 receives a person's voice. The speaker 132 outputs sound toward the person. The battery 134 supplies power to each element of the HMD 100. The sensor 136 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration.

[0017] The control device 140 controls each element of the HMD 100. For example, the control device 140 controls the contents output to the display 104 and the speaker 132. The control device 140 detects the movement of the field of view based on the detection result by the sensor 136. The control device 140 calculates the position of the person's eyes, the direction of the line of sight, and the like based on the data obtained by the image camera 121 and the depth camera 122. The image camera 121 and the depth camera 122 repeatedly acquire data, and the data is repeatedly processed to track the line of sight of the person. That is, eye tracking is performed. The control device 140 adjusts the display position of information on the display 104 according to the calculated line of sight. The control device 140 also controls the focus of the image camera 111 according to the calculated line of sight.

[0018] 3 can be used as the second device 2. The second device 2 is, for example, a VR device. The HMD 150 includes a band 152, a display 154, an imaging device 170, a microphone 180, a speaker 182, a battery 184, a sensor 186, and a control device 190.

[0019] The HMD 150 is worn on a person's head. The band 152 is flexible and can be attached to and detached from the person's head. The display 154 is attached to the band 152 and displays information. The display 154 is optically non-transparent. A projection device (not shown) may project information onto the display 154.

[0020] The imaging device 170 captures an image of a person's eyes. For example, the imaging device 170 includes an image camera 171 and a depth camera 172. The image camera 171 captures an RGB image of the eye. The depth camera 172 captures a depth image of the eye. The microphone 180 receives a person's voice. The speaker 182 outputs sound toward the person. The battery 184 supplies power to each element of the HMD 150. The sensor 186 is a six-axis detection sensor.

[0021] The control device 190 controls each element of the HMD 150. For example, the control device 190 controls the contents output to the display 154 and the speaker 182. The control device 190 detects movement of the field of view based on the detection result by the sensor 186. The control device 190 calculates the position of the person's eyes, the direction of the line of sight, etc. based on the data obtained by the image camera 171 and the depth camera 172, and performs eye tracking. The control device 190 adjusts the display position of information on the display 154 according to the calculated line of sight.

[0022] The form of the HMD is not limited to the illustrated example, and may be any form. The form of the HMD may be selected from a single-lens goggle type, a two-lens eyeglass type, a hat type, a helmet type that completely covers the head, and the like. A beginner who actually performs the work preferably wears an MR device that allows the user to see the real situation. An expert preferably wears a VR device so that the user can be more immersed in the beginner's environment.

[0023] FIG. 4 is a flowchart showing the processing by the processing system according to the embodiment. First, data recording is started (step S1). For example, images (video) acquired by each imaging device in the HMD, conversation data, data acquired by devices used in the work, etc. are recorded. If a sensor or the like for monitoring the work is provided, recording of the sensor data is also started. The data is repeatedly stored in the storage device 4.

[0024] The processing device 3 accepts a selection of a worker (step S2). A worker is a person who actually performs work. The processing device 3 accepts a selection of a work site (step S3). The processing device 3 accepts a selection of a work object (step S4). The processing device 3 accepts a selection of a scenario (step S5). A scenario is a file that defines the procedure for processing by the processing device 3. For example, data (voice) indicating the worker, work site, work object, and scenario is input to a microphone of an HMD worn by the worker. The processing device 3 selects the worker, work site, work object, and scenario based on the input data. The worker, work site, work object, and scenario may also be automatically selected based on a work schedule, etc.

[0025] The processing device 3 accepts the selection of the instructor (step S6). An instructor is a person who trains the worker. The worker (first person) and the instructor (second person) each wear an HMD (first device and second device). For example, data indicating the instructor is input to the microphone of the HMD worn by the instructor. The processing device 3 selects the instructor based on the input data. An instructor may be assigned in advance to each worker, work site, work object, or scenario, and an instructor may be automatically selected according to the selected worker, work site, work object, or scenario.

[0026] Once the selections in steps S2 to S6 are complete, the worker begins work. Various data are repeatedly acquired during work. The processing device 3 stores the acquired data in the storage device 4 as history data (step S7). The processing device 3 extracts a portion of data from the stored history data (step S8). Hereinafter, the data extracted from the history data by the processing device 3 will be referred to as "extracted data."

[0027] For example, specific operations by the worker and the instructor are extracted. The operations may be points (coordinates) in an image, image identification results, etc. The extracted data is used as annotations. Annotations indicate notable areas of the work object, information about those areas, etc. The processing device 3 stores the extracted data in the storage device 4 (step S9).

[0028] The processing device 3 may output the extracted data to a user and accept corrections from the user. The user may be a worker, an instructor, or an administrator of the processing system 10. For example, the processing device 3 determines whether there are corrections to the extracted data (step S10). If there are corrections, the processing device 3 accepts the corrections (step S11). Instead of corrections, the extracted data may be deleted. For example, an inappropriate coordinate selected by the worker is corrected to an appropriate coordinate or deleted. If there are no corrections in step S10 or if a correction is accepted in step S11, the processing device 3 stores the extracted data in the storage device 4 (step S12).

[0029] Figures 5 to 7 are schematic diagrams showing examples of dialogue in work, Figure 8 is a flow chart showing a specific example of a scenario, and Figure 9 is a table showing an example of extracted data. Hereinafter, the embodiment will be described with reference to a specific example. Figs. 5 to 7 show an example of a dialogue when acquiring 3D data. The worker 210 and the instructor 220 dialogue using the first device 1 and the second device 2, respectively. The dialogue is performed using a microphone of the HMD. The dialogue may also be performed using an input device such as a keyboard. In the illustrated dialogue example, utterances 211 to 215 by the worker 210 are transmitted from the first device 1 to the second device 2. Utterances 221 to 227 by the instructor 220 are transmitted from the second device 2 to the first device 1. The contents of the utterances 211 to 215 and 221 to 227 are stored in the storage device 4 as text data.

[0030] The processing device 3 executes intent understanding of the utterance. Known techniques can be applied to intent understanding. In summary, the processing device 3 executes morphological analysis, syntactic analysis, and semantic analysis. In morphological analysis, a sentence is segmented into morphemes, which are the smallest units. In syntactic analysis, the parts of speech of the segmented morphemes are identified. In syntactic analysis, the structure of the sentence, such as phrase syntax, is analyzed based on grammatical rules, etc. In semantic analysis, a semantic syntax that expresses the meaning conveyed by the sentence is synthesized based on the meanings (concepts) of the words in the sentence and the semantic relationships between words, etc. In the dialogue example, the parts that are particularly related to data extraction are underlined.

[0031] The utterance 211 includes a device "CMC-GMS-001" indicating the work target, and "obtain 3D data" indicating the work content. The processing device 3 selects the work target and the work content based on the intention understanding of the utterance 211. The processing device 3 also refers to a predetermined database, obtains data on the site where the work target is installed, and determines the work site.

[0032] The processing device 3 selects a "3D data acquisition" scenario corresponding to the selected work target and work content from multiple scenarios stored in the storage device 4. The scenario defines the type of data to be recorded as history data, the order in which the data is recorded, etc. The "3D data acquisition" scenario is an example of a first scenario.

[0033] As shown in Fig. 8, the processing device 3 records the work target and work content based on the utterance 211 in the storage device 4 (steps S21 and S22). As shown in Fig. 9, in a database 300 to which data on the work target, work name, and work ID are linked, date and time 301, worker 302, instructor 303, scenario 304, and data 305 are recorded. Scenario 304 indicates the name of a process included in the selected scenario. Data 305 indicates the name or data value of data obtained in each scenario 304. For example, data is recorded in rows 311 and 312 of database 300 by steps S21 and S22.

[0034] Next, an image is captured by the imaging device of the HMD of the worker 210. The processing device 3 records the captured image in the storage device 4 (step S23). The image is captured by the imaging device before the scenario is determined, and recording of the image may start based on the result of understanding the intention of the utterance 212. The captured data (image) is recorded in row 313 of the database 300. The captured image is also transmitted to the HMD of the instructor 220. The instructor 220 can share the field of view of the worker 210 through the image displayed on the display of the HMD.

[0035] An image 401 shown in Fig. 5 shows a part of the captured image. The image 401 shows a device 400. In the task of "acquiring 3D data", an operator scans the device 400 with a 3D scanner. Point cloud data is obtained by scanning. The point cloud data is used to synthesize 3D data.

[0036] For example, many coordinates are pointed to by eye tracking or other pointing tools in each of the first device 1 and the second device 2. Among them, the worker 210 points to some coordinates 411 in the image 401. The worker 210 also asks the instructor 220 by speech 213 whether the coordinates 411 are appropriate. The processing device 3 determines that the coordinates 411 are scan coordinates by understanding the intention of the speech 213. As shown in Figs. 8 and 9, the processing device 3 records the pointed coordinates in a row 314 of the database 300 (step S24).

[0037] The instructor 220 conveys in speech 222 that the pointed coordinate 411 is appropriate. The instructor 220 also points to a part of coordinate 412 in the image 402, and conveys in speech 222 to the worker 210 that he wants the worker 210 to scan at the coordinate 412. The processing device 3 determines that the coordinate 412 is a scan coordinate by understanding the intention of the utterance 222. As shown in FIG. 9, the processing device 3 records the pointed coordinate 412 in row 315 of the database 300. Recording of the coordinate in the subsequent processing is also based on the processing device 3's understanding of the intention of the utterance, like the coordinates 411 and 412.

[0038] The processing device 3 determines whether the pointing is complete (step S25). In "acquiring 3D data", the device 400 is scanned at a plurality of coordinates. For example, when all coordinates to be scanned are pointed, the worker or instructor inputs to the processing device 3 that the pointing of all coordinates has been completed.

[0039] The worker 210 scans the device 400 at each of the coordinates 411 and 412. As shown in Fig. 8 and Fig. 9, the processing device 3 records the scanned data in a row 316 of the database 300 (step S26). The worker 210 informs the instructor 220 by speech 214 that the scan is complete.

[0040] The instructor 220 informs the worker 210 by speech 223 that 3D data will be synthesized based on the scanned data. 3D data 421 shown in Fig. 6 indicates 3D data of the device 400 based on the scanned data. As shown in Fig. 8 and Fig. 9, the processing device 3 records the synthesized 3D data in row 317 of the database 300 (step S27).

[0041] The instructor 220 checks the 3D data 421. The instructor 220 informs the worker 210 by speech 224 that the quality of the synthesized 3D data is not good. As shown in Fig. 8 and Fig. 9, the processing device 3 records the quality of the synthesized 3D data in the row 318 of the database 300 based on the intention understanding (step S28).

[0042] The processing device 3 judges whether the 3D data obtained is of good quality (step S29). If the 3D data is not of good quality, step S24 is executed again. For example, the processing device 3 judges that the 3D data is of poor quality based on the understanding of the intention of the utterance 224 by the instructor 220.

[0043] The instructor 220 points to a certain coordinate 413 in the image 403, and tells the operator 210 by speech 225 that he wants the operator 210 to scan the coordinate 413. The processing device 3 records the pointed coordinate in a row 319 of the database 300.

[0044] The worker 210 scans the device 400 at the pointed coordinates 413. The processing device 3 records the scanned data in a row 320 of the database 300. The worker 210 informs the instructor 220 by speech 215 that the scan is complete.

[0045] The instructor 220 communicates to the worker 210, by speech 226, his intention to synthesize the 3D data again. 3D data 422 shown in Fig. 7 shows the 3D data of the device 400 that has been further synthesized using newly scanned data. The processing device 3 records the synthesized 3D data in row 321 of the database 300.

[0046] The instructor 220 checks the 3D data 422. The instructor 220 informs the worker 210 by speech 227 that the quality of the synthesized 3D data is good. Based on the intention understanding, the processing device 3 records the quality of the synthesized 3D data in row 322 of the database 300. Since good 3D data has been obtained, the processing device 3 ends the scenario of "obtaining 3D data".

[0047] FIG. 10 is a flowchart showing a specific example of processing by the processing device according to the embodiment. The processing device 3 generates learning data using a portion of the history data stored in the database 300. Specifically, as shown in Fig. 10, the processing device 3 acquires the history data (step S31). The processing device 3 extracts a device that is the target of work from an image in a video included in the history data (step S32). For example, the processing device 3 extracts the device from a portion of an image acquired during work by an imaging device of the HMD. The processing device 3 calculates the coordinates of the extracted device.

[0048] The processing device 3 extracts the scan coordinates from the history data (step S33). The processing device 3 calculates the relative scan coordinates with respect to the device coordinates (step S34). The processing device 3 searches for the model of the device "CMC-GMS-001" that is the target of the work (step S35). The processing device 3 links the relative scan coordinates to the model, and stores the data in the storage device 4 as learning data (step S36). In this example, the learning data is linked to the model of the device, but the learning data may be linked directly to the device.

[0049] FIG. 11 is a table illustrating the learning data. 11 includes rows 314, 315, and 319 for scan coordinates extracted from database 300. Column 305a records the scan coordinates relative to the device coordinates. The relative coordinates are used as the training data.

[0050] FIG. 12 is a flowchart showing the process performed by the learning device according to the embodiment. The learning device 5 performs learning using the learning data generated by the processing device 3. As shown in FIG. 12, the learning device 5 acquires the type of device "CMC-GMS-001" (step S41). The learning device 5 acquires multiple relative scan coordinates associated with the type from the learning data stored by the processing device 3 (step S42). The learning device 5 clusters the multiple scan coordinates (step S43). The clustering is performed by unsupervised learning of machine learning. As a result of the clustering, the multiple scan coordinates are classified into multiple groups. In each group, a set of scan coordinates indicates a recommended scan area. The learning device 5 saves the recommended scan area obtained by the clustering (step S44).

[0051] 13(a) to 13(c) are schematic diagrams illustrating examples of learning data. FIG. 13(a) to FIG. 13(c) show the stored learning data. In FIG. 13(a) to FIG. 13(c), the horizontal axis indicates the relative X coordinate. The vertical axis indicates the relative Y coordinate. FIG. 13(a) shows multiple scan coordinates accumulated by multiple operations. FIG. 13(b) shows even more scan coordinates accumulated by more operations than those in FIG. 13(a). As shown in FIG. 13(a) and FIG. 13(b), learning data is accumulated by repeating the operations. The learning device 5 clusters the multiple scan coordinates shown in FIG. 13(b). As a result, for example, as shown in FIG. 13(c), the multiple scan coordinates are classified into groups G1 to G3. The groups G1 to G3 each indicate a recommended scanning area.

[0052] FIG. 14 is a flowchart showing the processing by the processing device according to the embodiment. After the learning device 5 has completed learning, the processing device 3 can use the learned data to instruct or assist in the work. After the learning is completed, the processing device 3 can be selected instead of the instructor in step S6 of the flowchart shown in FIG.

[0053] When guidance by the processing device 3 is selected, as shown in Fig. 14, the processing device 3 acquires a scenario corresponding to the device to be worked on and the work content (step S51). The processing device 3 starts recording the work history (step S52). The processing device 3 extracts the device from the image (step S53). When a scan is performed by the worker, the processing device 3 calculates the relative coordinates of the scan with respect to the device (step S54). The processing device 3 acquires the scan coordinates stored as learning data and the recommended scan area learned by the learning device 5 (step S55).

[0054] The processing device 3 calculates the center of gravity of a plurality of scan coordinates in each recommended scan area. For example, as shown in FIG. 13(c), centers of gravity C1 to C3 are calculated in each of groups G1 to G3. The processing device 3 displays the center of gravity of each recommended scan area as a recommended scan coordinate on the display of the HMD of the worker (step S56). The worker follows the guide by the processing device 3 to perform each scan at the recommended scan coordinate. When scanning is performed in each recommended scan area, the processing device 3 synthesizes 3D data (step S57). The processing device 3 judges whether the work is completed (step S58). For example, when good-quality 3D data is synthesized, the processing device 3 judges the work is completed.

[0055] During instruction by the processing device 3, history data shown in Fig. 9 is obtained. Data for learning may be extracted from the obtained history data. The processing device 3 generates learning data using the extracted data. The learning device 5 executes learning using the generated learning data.

[0056] Figures 15 to 17 are schematic diagrams showing examples of dialogue in another task, Figure 18 is a flow chart showing a specific example of a scenario, and Figure 19 is a table showing an example of extracted data. 15 to 17 show examples of dialogue during analysis of seaweed shown in an image. An operator 510 and an instructor 520 engage in dialogue using a first device 1 and a second device 2, respectively. In the illustrated dialogue examples, utterances 511 to 516 by the operator 510 are transmitted from the first device 1 to the second device 2. Utterances 521 to 527 by the instructor 220 are transmitted from the second device 2 to the first device 1. The contents of the utterances 511 to 516 and 521 to 527 are stored in the storage device 4 as text data.

[0057] The utterance 511 includes "seaweed sample CMC-GMS-005" indicating the work target, and "manual 005" indicating the work content. The processing device 3 selects the work target and the work content from the intention understanding of the utterance 511. The processing device 3 also selects a "seaweed analysis" scenario corresponding to the selected work target and work content from a plurality of scenarios stored in the storage device 4. The "seaweed analysis" scenario is another example of the first scenario.

[0058] As shown in Fig. 18, the processing device 3 records the work target and work content (scenario) based on the utterance 511 in the storage device 4 (steps S61 and S62). As shown in Fig. 19, a date and time 601, worker 602, instructor 603, scenario 604, and data 605 are recorded in a database 600 to which data on the work target, work name, and work ID are linked. The scenario 604 indicates the name of a process included in the selected scenario. The data 605 indicates the name or data value of data obtained in each scenario 604. For example, data is recorded in rows 611 and 612 of the database 600 by steps S61 and S62.

[0059] Next, an image is captured by the imaging device of the HMD of the worker 510. The processing device 3 records the captured image in the storage device 4 (step S63). The captured data (image) is recorded in row 613 of the database 600. The instructor 520 shares the field of view of the worker 510 through the image displayed on the display of the HMD.

[0060] Image 701 shown in Fig. 15 shows a portion of the captured image. Image 701 shows a plurality of seaweeds 720. In analyzing a seaweed sample, an operator photographs the seaweed and classifies the seaweed. Operator 510 points to coordinates 711a and 711b on a portion of image 701. A rectangle 711 is specified by the points at coordinates 711a and 711b.

[0061] As shown in Figures 18 and 19, the processing device 3 records the coordinates pointed by the worker 510 in row 614 of the database 600 (step S64). The worker 510 states, by speech 513, that he has pointed to coordinates 711a and 711b and describes the classification of the seaweed contained in the rectangle 711. The processing device 3 records the classification of the seaweed by the worker 510 in row 615 of the database 600 (step S64). The instructor 520 conveys, by speech 521, that the coordinates 711a and 711b are appropriate.

[0062] 16, the instructor 520 points to some coordinates 712a and 712b in the image 702. The instructor 520 tells the worker 510 by speech 522 that he wants the worker 510 to check the seaweed included in the rectangle 712 specified by the coordinates 712a and 712b. The processing device 3 records the pointed coordinates 712a and 712b in the row 616 of the database 600 (step S65).

[0063] The worker 510 states, by speech 513, the classification of the seaweed contained in the rectangle 712 pointed to by the instructor 520. The processing device 3 records the classification of the seaweed by the worker 510 in row 617 of the database 600 (step S65). The instructor 520 affirms the classification by the worker 510 by speech 523.

[0064] The instructor 520 tells the worker 510 by speech 524 that he wants the worker 510 to check the seaweed contained in the rectangle 713 specified by the coordinates 713a and 713b in the image 703. The processing device 3 records the pointed coordinates 713a and 713b in row 618 of the database 600 (step S65). The instructor 520 states the classification of the seaweed contained in the rectangle 713 by speech 525. The processing device 3 records the classification of the seaweed by the instructor 520 in row 619 of the database 600 (step S65).

[0065] The instructor 520 informs the worker 510 that the task is completed by speech 526. In step S66, the processing device 3 determines that the task of seaweed classification is completed, and ends the scenario.

[0066] FIG. 20 is a flowchart showing a specific example of processing by the processing device according to the embodiment. As shown in FIG. 20, the processing device 3 acquires history data (step S71). The processing device 3 extracts video and pointed coordinates from the history data (step S72). The processing device 3 cuts out an image of the timing when the coordinates were pointed from the video, and crops a rectangular area specified by the coordinates in the image (step S73). The processing device 3 annotates the cropped image by linking it to the classification result (step S74). The processing device 3 stores the annotated image in the storage device 4 as learning data related to the work target and work content (step S75).

[0067] 21(a) to 21(d) are examples of learning data. Figure 21(a) is an image 701a cropped from image 701 shown in Figure 15. Image 701a is annotated with "005A" indicating the seaweed classification. Similarly, Figures 21(b) and 21(c) are cropped images 702a and 703a, respectively, annotated with "005B" and "005C" indicating the classification.

[0068] When the worker 510 or the instructor 520 tries to point to a coordinate, if there is insufficient information to calculate the line of sight or if the intention cannot be properly understood, the coordinate is not recorded. As a result, as shown in Fig. 21(d), the classification is annotated for the entire image. Such inappropriate learning data is corrected by the user as in step S10 shown in Fig. 4, and the corrected learning data is saved.

[0069] FIG. 22 is a flowchart showing the process performed by the learning device according to the embodiment. The learning device 5 executes learning using the learning data extracted by the processing device 3. As shown in Fig. 22, the learning device 5 acquires a work object and a work content in order to identify a model to be learned (step S81). The learning device 5 acquires data associated with the work object and the work content from the learning data stored by the processing device 3 (step S82).

[0070] The learning device 5 trains a classification model using the acquired training data (step S83). The classification model outputs a classification result in response to an input of an image. For example, the classification model includes a neural network. Supervised learning is performed on the classification model. In supervised learning, an image is used as input data, and a classification is used as a label. The learning device 5 saves the trained classification model (step S84).

[0071] FIG. 23 is a flowchart showing the processing by the processing device according to the embodiment. After the learning by the learning device 5 is completed, the processing device 3 can instruct or assist the task using the learned classification model. After the learning is completed, the processing device 3 can be selected instead of the instructor in step S6 of the flowchart shown in FIG.

[0072] When guidance by the processing device 3 is selected, as shown in FIG. 23, the processing device 3 acquires a scenario corresponding to the device to be worked on and the work content (step S91). The processing device 3 starts recording the work history (step S92). The processing device 3 inputs the captured image into a classification model (step S93) and acquires a classification result by the classification model (step S94). The processing device 3 displays the coordinates with the highest degree of certainty, the classification, and the certainty on the HMD of the worker 510 as a guide for the work (step S95). If the classification is not appropriate, the processing device 3 accepts a correction of the classification (step S96). The processing device 3 repeats output of the classification result until it is determined in step S97 that the work is completed.

[0073] History data is also recorded during instruction by the processing device 3. Learning data may be generated from the obtained history data. The learning device 5 executes learning using the generated learning data. In this case, a new classification model may be generated, or an existing classification model may be re-learned.

[0074] FIG. 24 and FIG. 25(a) to FIG. 25(d) are schematic diagrams illustrating pointing means in the embodiment. As the pointing means, in addition to the above-mentioned eye tracking, a virtual pointing tool may be used. For example, the processing device 3 detects the hand of a person appearing in an image. That is, the processing device 3 executes hand tracking. A posture estimation model can be used for detecting the hand. The posture estimation model outputs the position of the skeleton of the human body according to the input of an image. The posture estimation model includes a neural network. It is preferable that the posture estimation model includes a convolutional neural network (CNN). As the posture estimation model, OpenPose, DarkPose, CenterNet, or the like can be used.

[0075] For example, as shown in Fig. 24, the user moves his / her hand in a shape as if holding a writing implement. The processing device 3 detects multiple joints 801 of the hand as a skeleton. When the processing device 3 determines that the positional relationship of the multiple joints 801 is the shape of the hand when holding a writing implement, it displays a virtual pointing tool 800. When the user moves his / her hand, the processing device 3 moves the pointing tool 800 in accordance with the hand movement. When the pointing tool 800 is displayed and the user taps a part of the image with the pointing tool 800, the user can point to that coordinate.

[0076] Also, the user can freely draw using the pointing tool 800. The processing device 3 displays a drawing area 821 ahead of the user's line of sight 820. The user can use the pointing tool 800 to draw virtual figures, characters, symbols, and the like in the drawing area 821.

[0077] It is also possible to attach a mark to a virtual object using a pointing tool 800. For example, as shown in FIG. 25(a), a product 830, which is the object of work, is scanned by a 3D scanner 840. By multiple scans, 3D data 850 of the product 830 is obtained as shown in FIG. 25(b). When the worker or instructor determines that the synthesized 3D data 850 is good, he or she attaches a virtual mark 851 to the 3D data 850 using the pointing tool 800 as shown in FIG. 25(c). As shown in FIG. 25(d), the 3D data 850 is saved together with the mark 851.

[0078] The user can understand from the mark 851 that the quality of the 3D data 850 has already been checked, that the quality of the 3D data 850 has been judged to be good, and so on. By displaying a tool that can draw as well as point, the user's convenience can be improved. In addition, by linking the data to the content drawn by the tool and saving it, the user can easily understand information about the data.

[0079] Advantages of the embodiment will be described. In guidance, it is effective for the work to be performed by showing specific points (coordinates) on an image. When the coordinates are specifically shown by the instructor, the worker can easily understand the points on which attention should be paid or the points on which work should be done. Therefore, if the coordinates can be shown to the worker without relying on an instructor, at least a part of the guidance can be automated.

[0080] While a worker is working or an instructor is providing guidance, many points (coordinates) are pointed out on the image. For example, a worker may ask a question about a specific point or check a part that is not directly related to the work. For this reason, of the many coordinates obtained during work, only a portion of the coordinates is directly related to the work. However, it is difficult to manually extract only those coordinates that are useful for automating guidance from the many obtained coordinates.

[0081] Regarding this issue, the processing device 30 according to the embodiment acquires from the storage device 4 the image, coordinates, and dialogue data communicated between the first device 1 and the second device 2. Then, the processing device 30 extracts some coordinates from the multiple coordinates communicated based on the dialogue data. By using the dialogue data, only coordinates directly related to the task can be automatically extracted. The extracted coordinates can be used for learning to automate instruction.

[0082] According to the embodiment, data that can be used for learning can be automatically acquired, and the burden of preparing data can be reduced.

[0083] For example, by performing intent understanding on the dialogue data and extracting coordinates according to a scenario corresponding to each task, it is possible to extract coordinates that are more directly related to the task with greater accuracy.

[0084] When the coordinates obtained by eye tracking of the first device 1 and the second device 2 are stored, more coordinates are obtained during the work. For example, a plurality of first coordinates transmitted from the first device 1 to the second device 2, and a plurality of second coordinates transmitted from the second device 2 to the first device 1 are obtained. According to the embodiment, it is possible to extract coordinates more directly related to the work from the vast amount of stored coordinates. For example, a portion of the plurality of first coordinates directly related to the work and a portion of the plurality of second coordinates directly related to the work are extracted.

[0085] The extracted coordinates can be used to generate learning data. As in the above example, pointing coordinates relative to the work object in the image are generated as learning data. Alternatively, an area corresponding to the pointing coordinates is cropped from the image. The classification result is associated with the cropped image and saved as learning data. By learning using the generated learning data, at least a part of the instruction can be automated thereafter using the learning results.

[0086] In the above example, an HMD is used as the first device 1 and the second device 2. This example is not limiting, and the first device 1 and the second device 2 may be other than an HMD. For example, a combination of a monitor, a camera, a microphone, a pointing device, and a speaker is used as each of the first device 1 and the second device. The worker and the instructor communicate using the microphone and the speaker. The worker captures an image of the work target with a camera. The captured image is displayed on the user interface of each monitor. The worker and the instructor point to the area to be worked on, etc., on the user interface using a pointing device.

[0087] In this case, it is also possible to obtain history data such as those shown in Figures 9 and 19. It is also possible to extract some data from the history data and generate learning data. Learning can be performed using the generated learning data.

[0088] However, for more efficient work and convenience for the worker and the instructor, it is desirable that the first device 1 and the second device 2 are HMDs. By using the HMD, the worker can perform various operations such as imaging and pointing while working. The instructor can share the worker's field of view more realistically.

[0089] FIG. 26 is a schematic diagram showing a hardware configuration. 26 is used as the processing device 3 and the learning device 5. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0090] The ROM 92 stores a program for controlling the operation of the computer. The ROM 92 stores programs necessary for the computer to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0091] The CPU 91 includes a processing circuit. The CPU 91 uses the RAM 93 as a working memory and executes a program stored in at least one of the ROM 92 and the storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes. The storage device 94 stores data necessary for executing the program and data obtained by executing the program.

[0092] The input interface (I / F) 95 connects the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0093] The output interface (I / F) 96 connects the computer 90 and the output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI (registered trademark)). The CPU 91 transmits data to the output device 96a via the output I / F 96, and causes the output device 96a to output information.

[0094] The communication interface (I / F) 97 connects the computer 90 to a server 97a external to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. Data in the storage device 4 may be stored in the server 97a.

[0095] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (audio input), and a touch pad. The output device 96a includes one or more selected from a monitor and a projector.

[0096] The functions of the processing device 3 may be realized by one computer 90, or may be realized by a plurality of computers 90 working together. The functions of the learning device 5 may be realized by one computer 90, or may be realized by a plurality of computers 90 working together.

[0097] The various data processing described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0098] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0099] According to the embodiments described above, a processing device, a learning device, a processing system, a processing method, a program, and a storage medium are provided that are capable of automatically acquiring data that can be used for learning.

[0100] Implementations may include the following features. (Feature 1) Acquire image, coordinate, and interaction data communicated between a first device used by a first person performing a task and a second device used by a second person; Extracting a portion of the plurality of coordinates based on the dialogue data; A processing device that links the extracted coordinates to the task. (Feature 2) selecting a first scenario corresponding to the task from among a plurality of scenarios having defined processing procedures; The processing device according to feature 1, extracting the portion of the plurality of coordinates based on an understanding of the intent of the dialogue data in accordance with the first scenario. (Feature 3) the plurality of coordinates include a plurality of first coordinates transmitted from the first device to the second device and a plurality of second coordinates transmitted from the second device to the first device; The processing device according to feature 1 or 2, wherein a portion of the plurality of first coordinates and a portion of the plurality of second coordinates are extracted based on the dialogue data. (Feature 4) the first device and the second device are each a head mounted display; The processing device according to any one of features 1 to 3, wherein the plurality of coordinates include coordinates pointed to by eye tracking. (Feature 5) The processing device according to any one of features 1 to 4, further comprising: generating learning data relating to the task using the extracted coordinates. (Feature 6) The processing device according to feature 5, further comprising: calculating coordinates of an object of the work appearing in the image; and generating, as the learning data, relative coordinates of the extracted coordinates with respect to the coordinates of the object. (Feature 7) 7. The processing device according to feature 5 or 6, wherein in generating the learning data, the image including the extracted coordinates and a classification of the image are associated with each other to generate the learning data. (Feature 8) A learning device that performs machine learning using the learning data generated by the processing device according to any one of features 5 to 7. (Feature 9) The learning device according to feature 8, wherein in the machine learning, learning of a clustering or classification model is performed. (Feature 10) A processing device that transmits a guide regarding the task to the first device using data learned by the learning device according to feature 8. (Feature 11) A first device worn by a first person performing a task; a second device worn by a second person; and a processing device that acquires image, coordinate, and dialogue data communicated between the first device and the second device, extracts a part of the coordinates based on the dialogue data, and generates learning data related to the task using the extracted coordinates; A learning device that performs machine learning using the learning data; A processing system comprising: (Feature 12) Acquire image, coordinate, and interaction data communicated between a first device used by a first person performing a task and a second device used by a second person; Extracting a portion of the plurality of coordinates based on the dialogue data; A processing method in which the extracted coordinates are linked to the task. (Feature 13) 13. A program for causing a computer to execute the processing method according to feature 12. (Feature 14) A storage medium storing the program according to feature 13.

[0101] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0102] 1: first device, 2: second device, 3: processing device, 4: storage device, 5: learning device, 10: processing system, 30: processing device, 90: computer, 91: CPU, 92: ROM, 93: RAM, 94: storage device, 95: input interface, 95a: input device, 96: output interface, 96a: output device, 97: communication interface, 97a: server, 98: system bus, 100: HMD, 102: band, 104: display, 110: imaging device, 111: image camera, 112: depth camera, 120: imaging device, 121: image camera, 122: depth camera, 130: microphone, 132: speaker, 134: battery, 136: sensor, 140: control device, 210: operator, 211-215: utterance, 220: instructor, 221-227: utterance, 300: database, 300a: learning data, 302: worker, 303: instructor, 304: scenario, 305: data, 305a: column, 311-322: row, 400: device, 401-403: image, 411-413: coordinate, 421, 422: 3D data, 510: worker, 511-516: utterance, 520: instructor, 521-527: utterance, 600: database, 602: worker, 603: instructor, 604: scenario, 605: data, 611-619: row, 701-703, 701a-703a: image, 711: Rectangle, 711a, 711b: Coordinates, 712: Rectangle, 712a, 712b: Coordinates, 713: Rectangle, 713a, 713b: Coordinates, 720: Seaweed, 800: Pointing tool, 801: Joint, 820: Line of sight, 821: Drawing area, 830: Product, 840: 3D scanner, 850: 3D data, 851: Mark, C1 to C3: Center of gravity, G1 to G3: Group

Claims

1. Acquiring image, coordinate, and interaction data communicated between a first device used by a first person performing a task and a second device used by a second person; Extracting a portion of the plurality of coordinates based on the dialogue data; A processing device that links the extracted coordinates to the task.

2. selecting a first scenario corresponding to the task from among a plurality of scenarios having defined processing procedures; The processing device according to claim 1 , further comprising: a processor configured to extract the part of the plurality of coordinates based on an understanding of an intent of the dialogue data in accordance with the first scenario.

3. the plurality of coordinates include a plurality of first coordinates transmitted from the first device to the second device and a plurality of second coordinates transmitted from the second device to the first device; The processing device according to claim 1 , wherein a portion of the plurality of first coordinates and a portion of the plurality of second coordinates are extracted based on the interaction data.

4. the first device and the second device are each a head mounted display; The processing device according to claim 1 , wherein the plurality of coordinates include coordinates pointed by eye tracking.

5. The processing device according to claim 1 , further comprising: a processor configured to generate learning data relating to the task using the extracted coordinates.

6. The processing device according to claim 5 , further comprising: a processor configured to calculate coordinates of an object of the work that appears in the image; and generate, as the learning data, relative coordinates of the extracted coordinates with respect to the coordinates of the object.

7. The processing device according to claim 5 , wherein, in generating the learning data, the image including the extracted coordinates is associated with a classification of the image to generate the learning data.

8. A learning device that performs machine learning using the learning data generated by the processing device according to any one of claims 5 to 7.

9. The learning device according to claim 8 , wherein the machine learning executes learning of a clustering or classification model.

10. A processing device that transmits a guide regarding the task to the first device using data learned by the learning device according to claim 8.

11. A first device worn by a first person performing a task; a second device worn by a second person; a processing device that acquires image, coordinate, and dialogue data communicated between the first device and the second device, extracts a part of the coordinates based on the dialogue data, and generates learning data related to the task using the extracted coordinates; A learning device that performs machine learning using the learning data; A processing system comprising:

12. Acquiring image, coordinate, and interaction data communicated between a first device used by a first person performing a task and a second device used by a second person; Extracting a portion of the plurality of coordinates based on the dialogue data; A processing method in which the extracted coordinates are linked to the task.

13. A program causing a computer to execute the processing method according to claim 12.

14. A storage medium storing the program according to claim 13.

Images