Learning data generation device, robot system, learning data generation method and learning data generation program

Abstract

The present invention provides a training data generation device that can generate training data without high labor costs and enormous working time. The training data generation device, which generates training data for use in machine learning, comprises a data acquisition unit, a data processing unit, and a data storage unit. The data acquisition unit captures an image of an existing area of ​​a plurality of workpieces. The data processing unit estimates at least area information, reflecting an area span of all or part of at least one workpiece, based on the captured image and generates training data containing the estimated area information. The data storage unit stores the generated training data and the image as training data.

Description

Area

[0001] The present disclosure relates to a learning data generation device, a robot system, a learning data generation method and a learning data generation program. State of the art

[0002] In recent years, machine learning has been deployed and used in practice across various fields. One technique of this kind is "supervised learning." Supervised learning is performed by preparing a large set of groups containing a feature value (a variable representing a feature of data: a key to prediction) and supervised data (labeled, correct data). This large set of groups is then provided to a computer (a machine learning device), and annotation (training with annotations) is cited as one of the methods used to generate the supervised data.

[0003] Annotation requires work to provide associated tags and metadata labels for large amounts of image, audio, video, text, and other data, and to generate the training data. Specifically, in image annotation, for example, area information about an object (workpiece) is learned by a person manually clicking on the image. When annotation is performed manually by a person, this method incurs high labor costs and is extremely time-consuming.

[0004] Traditionally, various suggestions have been made as techniques for easily generating the supervised data used for "supervised learning". List of quotations Patent literature [PTL 1] Unexamined Japanese patent publication (Kokai) No. 2022-118300 [PTL 2] Unexamined Japanese patent publication (Kokai) No. 2014-059729 Overview: Technical Task

[0005] As described above, since annotation is traditionally performed manually by a single person, this results in high labor costs and a significant amount of time. Furthermore, if the annotation is performed on, for example, the position, orientation, and shape of an object within a large amount of image data, it is necessary to minimize any variation in the results obtained by each individual annotator. Therefore, this requires the creation of detailed annotation rules, training, and selection of each individual performing the annotation, which further increases labor costs and extends the working time.

[0006] Therefore, it is desirable to provide a learning data generation device, a robot system, a learning data generation method, and a learning data generation program that can perform the annotation without high labor costs and enormous working time. Technical solution

[0007] One embodiment according to the present disclosure provides a training data generation device comprising a data acquisition unit, a data processing unit, and a data storage unit, and generating training data used for machine learning. The data acquisition unit captures an image of the presence area of ​​a plurality of workpieces; the data processing unit estimates at least area information, reflecting a range of an entirety or part of at least one of the workpieces, based on the captured image and generates training data containing the estimated area information; and the data storage unit stores the generated training data and the image as training data. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is a graphical representation that provides an example of an entire robot system to describe an example of a learning data generation device according to the present embodiment. [ Fig. 2] Fig. Figure 2 is a graphical function block representation to describe an example of the learning data generation device according to the present embodiment. [ Fig. 3] Fig. 3 is a flowchart to describe an example of processing in a first example of a learning data generation program according to the present embodiment. [ Fig. 4] Fig. Section 4 is a flowchart to describe an example of processing in a second example of the learning data generation program according to the present embodiment. [ Fig. 5] Fig. Section 5 is a flowchart to describe an example of processing in a third example for the learning data generation program according to the present embodiment. [ Fig. 6] Fig. Section 6 is a flowchart to describe an example of processing in a fourth example of the learning data generation program according to the present embodiment. [ Fig. 7] Fig. Figure 7 is a graphical representation to describe an example of a user's revision processing of training data in a training data generation method according to the present embodiment. [ Fig. 8] Fig. Figure 8 is a flowchart to describe an example of processing in a fifth example of the learning data generation program according to the present embodiment. [ Fig. 9] Fig. Figure 9 is a graphical representation of an example of a workpiece in the robot system in which an example of the learning data generation device according to the present embodiment is used. [ Fig. 10] Fig. Figure 10 is a graphical representation for describing a shape area of ​​a workpiece in an example of the learning data generation device according to the present embodiment. Description of the embodiments

[0008] Examples of a learning data generation device, a robot system, a learning data generation method, and a learning data generation program according to the present embodiment are described in detail below with reference to the accompanying drawings. The same or similar element is identified in each of the drawings by the same or similar reference numerals. Furthermore, an embodiment described below does not limit the technical scope of the invention described in the claims or the meaning of any term.

[0009] Fig. Figure 1 is a graphical representation that illustrates an example of the complete robot system to describe an example of the learning data generation device according to the present embodiment. As shown in Fig. As shown in Figure 1, a robot system 100 comprises a robot 1, a robot control unit 2, a learning data generation device 3, and a camera 4. The robot 1 includes a robot mechanism unit 10, an arm 11, and an end effector (hand area) 12.

[0010] It should be noted that in the Fig. In the robot system 100 shown in Figure 1, a machine learning device for performing machine learning (supervised learning) is integrated into the robot control unit 2 and is therefore not shown. However, if it is difficult to integrate the machine learning device into the robot control unit 2 due to high computational effort and large data volumes, the machine learning device can also be configured, for example, as a host computer, a general-purpose computer, or the like, located at a dedicated workstation near the robot control unit 2 or at a location isolated from the robot system 100.In addition, when learning is performed by feeding a large amount of data into a learning model, a general-purpose computing device or a general-purpose processor can be used; however, processing at a higher speed can be achieved using general-purpose computing on graphics processing units, a large PC cluster, and the like.

[0011] Robot 1, for example, is designed as a multi-axis robot, and the end effector 12 is located at one end of arm 11. Fig. The end effector 12 is an adsorption device (absorption hand); however, it is understood that the end effector 12 can be exchanged for various elements corresponding to a workpiece (object), work content, and the like used by the robot system 100. It should be noted that the robot mechanism unit 10 is used to cause the robot 1 to perform a predetermined movement based on a control command from the robot control unit 2.

[0012] The robot control unit 2 receives an output from the camera 4 and an output from the learning data generation device 3, generates a control command to cause the robot 1 to perform a predetermined movement, for example, based on a program, control data, and the like that have been stored in advance within a storage device, and outputs the control command to the robot mechanism unit 10. It should be noted that, as described above, the machine learning device that performs the supervised learning may not be integrated into the robot control unit 2 and may be located separately near the robot control unit 2 or at a location isolated from the robot system 100, for example, depending on the computational effort and data volume.

[0013] The learning data generation device 3 receives multiple images of a workpiece and generates training data (monitored data) containing area information about workpieces D1 to D9 (information about a shape area of ​​the workpieces) in each image. Furthermore, the learning data generation device 3 outputs an image associated with the generated training data to the robot control unit 2 (machine learning device) as training data.The learning data generation device 3 receives image data of workpieces D1 to D9, which were captured by camera 4. However, the image data provided to the learning data generation device 3 is not limited to an image captured by camera 4 in the robot system 100 and can, for example, be different types of image data, such as an image in which a workpiece is captured in advance and an image captured by another robot system. Furthermore, an image entered into the learning data generation device 3 is not limited to a two-dimensional image, and, as described in detail below, three-dimensional data (a three-dimensional image, three-dimensional point group data, and three-dimensional measurement data) can also be entered together.

[0014] Camera 4 is used to capture a two-dimensional image of the area containing the plurality of workpieces (for example, a plurality of cardboard boxes) D1 to D9, or a two-dimensional image and three-dimensional data (three-dimensional point group data). It includes two cameras 4a and 4b and a projector 4c. The projector 4c projects a predefined pattern onto an area containing the plurality of workpieces D1 to D9, and the two cameras 4a and 4b record the area containing the plurality of workpieces onto which the predefined pattern is projected by the projector 4c and measure a three-dimensional shape of the workpieces D1 to D9.In this way, camera 4 can be configured to measure a three-dimensional shape of the presence area of ​​the majority of workpieces D1 to D9 and to capture three-dimensional point group data; however, it can also be configured, for example, as a two-dimensional camera. It should be noted that camera 4 is in . Fig. 1 can also capture a two-dimensional image, in which the area of ​​presence of the majority of workpieces D1 to D9 is recorded, using an image taken by one of the two cameras 4a and 4b. In other words, camera 4 can be able to capture a suitable image, for example, corresponding to a workpiece on which robot 1 is performing work, a work content, and the like, or corresponding to an annotation of supervised learning.

[0015] Fig. Figure 2 is a graphical function block representation to describe an example of the learning data generation device according to the present embodiment. As in Fig. Figure 2 shows that the learning data generation device 3, which generates the training data used for machine learning, comprises a data acquisition unit 31, a data processing unit (arithmetic processing device) 32, a data storage unit 33, an acceptance unit 34, and a display unit 35. The data acquisition unit 31 acquires an image (a two-dimensional image or a two-dimensional image and three-dimensional point group data) of a presence area of ​​a plurality of workpieces (D1 to D9) and a trained model. The data processing unit 32 estimates at least area information, reflecting all or part of a range span of at least one of the workpieces (D0, D), based on the image acquired by the data acquisition unit 31.For the learned model, for example, a learning model generated by the other learning data generation device 3 or a learning model generated by the device's own learning data generation device 3 (previous learning model) can be used. With reference to... Fig. It should be noted in sections 3 to 8 that, as described in detail below, if no learned model is used according to any of the examples of the present embodiment, the data acquisition unit 31 does not need to acquire the learned model. Similarly, if no three-dimensional point group data is used according to any of the examples of the present embodiment, the data acquisition unit 31 does not need to acquire the three-dimensional point group data.

[0016] The area information about the workpiece (D0) includes, for example, outline area information, which reflects the outline of the workpiece; pick-up area information for adsorbing, suctioning, or gripping the workpiece; and / or local area information about the workpiece. It should be noted that the local area information about the workpiece (D0) includes, for example, an area that is a flat surface, a curved surface, or has a large surface area on the workpiece; an area on the workpiece that is not slippery; and / or an area with high density on the workpiece. The local area information about the workpiece is described below with reference to Fig. 9 described in detail.

[0017] The data storage unit 33 stores the training data and the image (for example, the two-dimensional image or the two-dimensional image and the three-dimensional point group data) generated by the data processing unit 32 as training data. The input unit 34 receives revision information based on the area information for at least one of the workpieces, which is entered, for example, by a worker (user). The data processing unit 32 then revises the training data based on the revision information received from the input unit 34 and outputs it. The display unit 35 displays the image and the training data and allows, for example, the user to perform further revisions to the training data.

[0018] When the data acquisition unit 31 acquires the learned model, the data processing unit 32 estimates the area information about the workpiece based on the image and the learned model and generates the training data. Furthermore, when the data acquisition unit 31 acquires the two-dimensional image of the presence area of ​​the majority of workpieces (D0 to D9, D), the data processing unit 32 estimates the area information based on the two-dimensional image acquired by the data acquisition unit 31 and generates the training data.When the data acquisition unit 31 acquires the two-dimensional image and the three-dimensional data (three-dimensional point group data) over the presence area of ​​the majority of workpieces (D0 to D9, D), the data processing unit 32 further estimates the area information based on the two-dimensional image(s) and three-dimensional point group data acquired by the data acquisition unit 31 and generates the training data.

[0019] The data processing unit 32 can estimate three-dimensional area information about the workpiece based on the processing result of an operation, such as an analysis of three-dimensional point group data (e.g., plane analysis, curvature analysis, blob analysis, coordinate system analysis, position analysis, depth analysis, scale analysis, feature analysis, environment point analysis, network analysis, and / or voxel analysis). It can also estimate area information about the workpiece in the two-dimensional image by comparing the three-dimensional point group data with the two-dimensional image and generate the training data. Furthermore, the data processing unit 32 can revise the training data based on the area information about the workpiece estimated from the analysis result of the three-dimensional point group data and output the training data.The data processing unit 32 can extract a feature by performing image processing based on the area information about at least one of the workpieces accepted by the acceptance unit 34 and the image, estimate the area information about the workpiece in the image by performing a comparison of the extracted feature and also generate the training data.

[0020] As described above, the data processing unit 32 can perform image processing based on the image, estimate the area information about the workpiece based on a processing result of the image processing and the image, and also generate the training data. Furthermore, the data processing unit 32 can estimate the area information about the workpiece, for example, based on a processing result of an operation such as image processing, pattern matching, plane matching, curvature matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis, and / or color information analysis.In addition, the data processing unit 32 can also revise the training data based on the area information about the workpiece, which is estimated based on the processing result of the image processing, and output the training data.

[0021] Before presenting a first to fifth example of a learning data generation program (learning data generation method) according to the present embodiment with reference to Fig. Sections 3 to 8 describe an example of the workpiece (object) D0, which is a work target, with reference to Fig. 9 is described and a formulation of a “shape area of ​​a workpiece” is given in the present description with reference to Fig. 10 described.

[0022] Fig. Figure 9 is a graphical representation of an example of a workpiece in the robot system, in which an example of the learning data generation device according to the present embodiment is used. In the robot system according to the present embodiment, the workpiece (object) D0, which is the work target, is, for example, not limited to the same raw material having a rectangular parallelepiped shape, as in the Fig. The cardboard boxes D1 to D9 shown in the diagram are shown, and various workpieces are conceivable. In other words, the one shown in the diagram is... Fig. In Figure 9, workpiece D0 is an air connection, and areas Da, Db, and Dd are formed from a plastic raw material (for example, PTFE: polytetrafluoroethylene), while area Dc is formed from a metal raw material (for example, brass or stainless steel). It is assumed that the metal forming area Dc has a higher specific density than the plastic forming areas Da, Db, and Dd, and that area Dc further surrounds areas Db to Dd with a hexagonal plane in such a way as to facilitate tightening with a tool such as a wrench.

[0023] Various local areas exist on a workpiece, such as a flat surface, a curved surface with a large area on the workpiece, a non-slip area, and a high-density (heavy) area. When, for example, a workpiece is picked up by an adsorption hand (12), the pick-up success rate changes according to the area of ​​the workpiece where adsorption is performed. Therefore, area information that defines the outline of a local area on a workpiece is preferably included as training data. This is particularly relevant if, for example, the workpiece is the area described in the diagram. Fig. Given that the air connection D0 shown in Figure 9 is involved, the success rate of removal is conceivably higher in a case where the area Dc, with a planar shape, is adsorbed by the adsorption hand 12 than in a case where the areas Da, Db, and Dd, with a curved shape, are adsorbed. Since removal can be performed at a position closer to the center of gravity of the entire workpiece D0, the success rate of removal is conceivably higher in a case where the area Dc is formed from the metal, which is a material with a higher density (and is heavier) than the areas Da, Db, and Dd, which are formed from the plastic with a lower density (and is lighter).In other words, the planar area Dc, formed from the metal of the air connection D0, which is the workpiece, is adsorbed by the adsorption hand 12, and in this way the workpiece D0 can be stably removed.

[0024] As in the reference to Fig. 6 and Fig. In the fourth example described in section 7, a user preferably refers to an image and training data displayed on the display unit 35 and revises the training data (learning data). In this case, the user can perform only part of the processing, thus requiring significantly less effort compared to a conventional case where area information about an object is trained by a person manually clicking on an image. In this way, the area information about the workpiece D0 preferably includes outline area information reflecting an outline of the workpiece D0, removal area information for adsorbing, suctioning, or gripping the workpiece D0, and / or local area information about the workpiece D0.

[0025] Next up is Fig. 10 a graphical representation for describing a shape area of ​​a workpiece in an example for the learning data generation device according to the present embodiment, wherein Fig. 10A represents a form area of ​​a workpiece D and Fig. 10B represents an area that also includes a background of workpiece D. In the present description, a formulation of a “shape area of ​​a workpiece” includes only the workpiece D, for example in a two-dimensional image P3 output by camera 4, and specifies an area (an area of ​​workpiece D itself) that does not include the background of workpiece D and is indicated by a reference sign A1, as in Fig. 10A is shown, and the wording does not specify an area that defines the workpiece D and the background, as in Fig. 10B is shown. In other words, a “shape area of ​​a workpiece” in this description refers to “area information that reflects a shape / outline of a workpiece”, and the shape / outline of the workpiece can be calculated using this information.

[0026] The first to fifth examples of the learning data generation program (the learning data generation method) according to the present embodiment are described below with reference to Fig. Sections 3 to 8 are described. It should be noted that the following description is based on a graphical function block representation of the function block with reference to Fig. The learning data generation device 3 described in section 2 is based on this, but it is understood that the learning data generation device 3 is not based on the device described in section 2. Fig. The representation shown in section 2 is limited.

[0027] Fig. Figure 3 is a flowchart describing an example of processing in the first example of the training data generation program (training data generation method) according to the present embodiment and represents a case in which the data acquisition unit 31 acquires a two-dimensional image and a trained model. As in Fig. As shown in Figure 3, at the beginning of an example for processing the learning data generation program in the first example, camera 4 captures a two-dimensional image of the presence area of ​​workpiece D in step ST11. In other words, as referenced in Figure 3, the camera 4 captures a two-dimensional image of the presence area of ​​workpiece D in step ST11. Fig. As described in 1, the camera 4 takes a two-dimensional image of an area where the workpieces (for example, the majority of cardboard boxes) D1 to D9 are present and outputs the two-dimensional image to the learning data generation device 3.

[0028] Next, processing proceeds to step ST12, and the data acquisition unit 31 acquires the two-dimensional image and a learned model and outputs the two-dimensional image and the learned model to the data processing unit 32. As referred to Fig. As described in section 2, the two-dimensional image received by the data acquisition unit 31 is a two-dimensional image that includes an area containing the workpiece captured by the camera 4. Furthermore, the learned model received by the data acquisition unit 31 can, for example, be a learned model previously created by a supplier of a robot system. Alternatively, as described above, the learned model can also be, for example, a learned model generated by the other learning data generation device 3 or a learned model generated by the learning data generation device 3 itself (previous learned model).

[0029] Furthermore, processing proceeds to step ST13, where data processing unit 32 estimates a workpiece area based on the two-dimensional image and the learned model from data acquisition unit 31. Subsequently, processing proceeds to step ST14, where data processing unit 32 generates training data (monitored data) based on estimation information about a shape area of ​​the workpiece D. In other words, data processing unit 32 estimates a shape area (area information) of the workpiece based on the two-dimensional image and the learned model and generates training data. As referenced in Fig. As described in section 10, the form area of ​​the workpiece D corresponds, for example, to the form area A1 in Fig. 10A herein.

[0030] The processing then proceeds to step ST15, and the data storage unit 33 stores the training data and the two-dimensional image as training data in the data storage unit 33, and the processing ends. Step ST16 involves processing the display of the two-dimensional image data by step ST12 and the training data by step ST14 on the display unit 35. A user can, for example, refer to the two-dimensional image and the training data displayed on the display unit 35 and also modify the training data. It is understood that no processing by the user is required; the two-dimensional image and the training data can simply be displayed on the display unit 35. It should be noted that the processing in step ST16 in the first example is similar to steps ST26, ST36, ST47, and ST55 in the examples described below.

[0031] In this way, according to the learning data generation program (the learning data generation procedure) in the present first example, the learning data (training data) can be easily generated without high labor costs and enormous working time.

[0032] Fig. Figure 4 is a flowchart describing an example of processing in the second example of the learning data generation program according to the present embodiment and represents a case in which the data acquisition unit 31 acquires a two-dimensional image and does not acquire a learned model. As in Fig. As shown in Figure 4, at the beginning of an example of the processing of the learning data generation program in the second example, camera 4 captures a two-dimensional image of the presence area of ​​workpiece D (D1 to D9) in step ST21. The processing then proceeds to step ST22, where data acquisition unit 31 captures the two-dimensional image and outputs it to data processing unit 32. As described in Figure 4, the following steps are performed: Fig. As described in 2, the two-dimensional image received by the data acquisition unit 31 is a two-dimensional image in which an area is recorded in which the workpiece recorded by the camera 4 is present.

[0033] Next, processing proceeds to step ST23, where data processing unit 32 estimates a shape area of ​​workpiece D by performing image processing on the two-dimensional image from data acquisition unit 31. Processing then proceeds to step ST24, where data processing unit 32 generates training data based on the estimation information about the shape area of ​​workpiece D. Finally, processing proceeds to step ST25, where data storage unit 33 stores the training data and the two-dimensional image as training data, and processing concludes. It should be noted that step ST26 is similar to step ST16 in the first example described above, and its description is omitted.

[0034] In this way, the learned model from the first example is not used in the training data generation program of the present second example, thus simplifying processing. However, since the learned model is not used, the accuracy of the training data may be slightly lower than in the first example. Therefore, the execution of the present second example is preferably determined based on the shape of a workpiece, which is a target, the content of a job, the time until a robot system actually works, or the like.

[0035] As a modification example for the learning data generation program (learning data generation method) according to the present embodiment, a user (worker) learns, for example, area information for only one workpiece (D0, D) (the shape area of ​​the workpiece) for a specific type of workpiece from a large number of workpieces (D0 to D9). Furthermore, a feature is extracted by performing image processing based on the learned area information and an image (a two-dimensional image). Subsequently, an area of ​​all (the majority of) the workpieces in the two-dimensional image can be estimated by performing a comparison processing of the extracted feature, and training data can also be generated.

[0036] Fig. Figure 5 is a flowchart describing an example of processing in the third example of the learning data generation program according to the present embodiment and represents a case in which the data acquisition unit 31 only acquires a two-dimensional image and does not acquire a learned model, the acceptance unit 34 accepts area information (shape area) about a workpiece, and the data processing unit 32 performs the image processing. As in Fig. As shown in Figure 5, at the beginning of an example of the processing of the learning data generation program in the third example, camera 4 captures a two-dimensional image of the presence area of ​​workpiece D in step ST31. The processing then proceeds to step ST32, where data acquisition unit 31 captures the two-dimensional image and outputs it to data processing unit 32.

[0037] Next, processing proceeds to step ST33, where the data processing unit 32 performs image processing on the two-dimensional image from the data acquisition unit 31 and the area information about the workpiece D, estimating a shape area of ​​all (the majority of) the workpieces D1 to D9 (D0, D) in the image. The image processing performed by the data processing unit 32 includes, for example, pattern matching, plane matching, curvature matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis, and / or color information analysis.

[0038] Furthermore, processing proceeds to step ST34, where data processing unit 32 generates training data based on estimation information about the form area of ​​the workpiece D. Processing then proceeds to step ST35, where data storage unit 33 stores the training data and the two-dimensional image as training data, and processing ends. It should be noted that the processing in step ST36 is similar to steps ST16 and ST26 described above and, as in steps ST45 to ST47, is described in the section on... Fig. As described in the fourth example in section 6, a user can refer to a two-dimensional image and training data displayed on display unit 35 and, furthermore, revise the training data. As described in section 6, Fig. As described in section 9, a revision of the training data can be carried out, for example, by the user based on a surface shape and a condition of the workpiece D, a raw material (material) that forms an area, or the like; however, for example, the determination of different areas, such as an area that is a flat surface, a curved surface that has a large surface area on the workpiece, an area that is not slippery, and an area with high density (that is heavy), can also be automated using a machine learning device and the like.

[0039] Fig. Figure 6 is a flowchart describing an example of processing in the fourth example of the training data generation program according to the present embodiment and represents a case in which the data acquisition unit 31 acquires a two-dimensional image and a trained model, and a user modifies the training data. As in Fig. As shown in Figure 6, at the beginning of an example of the processing of the learning data generation program in the fourth example, camera 4 captures a two-dimensional image of the presence area of ​​workpiece D in step ST41. The processing then proceeds to step ST42, where data acquisition unit 31 captures the two-dimensional image and a learned model and outputs the two-dimensional image and the learned model to data processing unit 32.

[0040] Next, processing proceeds to step ST43, where data processing unit 32 estimates a shape area of ​​workpiece D based on the two-dimensional image and the learned model from data acquisition unit 31. Processing then proceeds to step ST44. In step ST44, data processing unit 32 generates training data based on estimation information about the shape area of ​​workpiece D.

[0041] Furthermore, processing proceeds to step ST45, where a user (worker) revises the training data while viewing (referring to) display unit 35. Processing then proceeds to step ST46, where data storage unit 33 stores the training data and the two-dimensional image as training data, and processing subsequently ends. The processing in step ST47 corresponds to steps ST16, ST26, and ST36 described above. In this fourth example, as indicated in steps ST45 through ST47, the user can refer to the two-dimensional image and the training data displayed on display unit 35 and also revise the training data. As described in steps ST45 through ST47... Fig. As described in section 9, a revision of the training data can be carried out, for example, by the user based on a surface shape and a condition of the workpiece D, a raw material (material) that forms an area, or the like; however, for example, the determination of different areas, such as an area that is a flat surface, a curved surface that has a large surface area on the workpiece, an area that is not slippery, and an area with high density (that is heavy), can also be automated using a machine learning device and the like.

[0042] Fig. Figure 7 is a graphical representation describing an example of a user's revision processing of training data in the training data generation method according to the present embodiment. The following are presented: Fig. 7A, Fig. 7B and Fig. 7C in a top row represents a case where the training data is correct and user revision of the training data is unnecessary, and Fig. 7D, Fig. 7E and Fig. 7F in a lower row represents a case where the training data is incorrect and a revision of the training data by the user is necessary. Furthermore, Fig. 7A and Fig. 7D two-dimensional images, in which a plurality of workpieces are captured, are Fig. 7B and Fig. 7E Images in which an extraction result of an outline line by image processing is displayed over the two-dimensional image, and are Fig. 7C and Fig. 7F Images representing a final learning result (final learning data).

[0043] First, a presence area is recorded using camera 4 with respect to an area in which a plurality of workpieces (for example, a plurality of cardboard boxes) are arranged, and a plurality of two-dimensional images are recorded with respect to a presence area of ​​the plurality of workpieces, which are arranged in different arrangements by changing an arrangement of the workpieces ( Fig. 7A and Fig. 7D). Next, image processing is performed on each of the images, and a feature of the outline of each of the workpieces is extracted from each of the images ( Fig. 7B and Fig. 7E). For example, a black and white image can be generated by performing binary processing on a two-dimensional image as a method for extracting the outline, and a boundary between white and black can be found and obtained in the image. It should be noted that if the workpiece is, for example, a cardboard box, the color is almost uniform within a certain area of ​​the workpiece, and an outline of the workpiece can be obtained more accurately with a greater color difference between the workpiece area and a background area.

[0044] First, a case is considered in which an outline extracted by image processing is correct, for example, a case in which features of outlines of workpieces b11 to b14, as in Fig. 7B, as a result of performing image processing on recorded images of workpieces a11 to a14, as in Fig. 7A, extracted, described. The user views (referencing) the image as shown in Fig. 7B shows a two-dimensional image and training data about the workpiece, which are displayed on display unit 35. The user determines whether the training data (the two-dimensional image and the training data) displayed on display unit 35 and how it is displayed. Fig. 7B is shown, are correct, and if the user correctly determines, the final learning data (training data c11 to c14) will be displayed, as shown in Fig. 7C is displayed, captured without revising the learning data (training data), and the learning data is stored in the data storage unit 33.

[0045] In contrast, a case in which an outline extracted by image processing is incorrect, for example a case in which features of outlines of workpieces e11 to e14, as in Fig. 7E, as a result of performing image processing on captured images of workpieces d11 to d14, as in Fig. 7D, extracted, described. In other words, a case is described in which, as in Fig. 7E shows that outline lines on a left side and a lower side of workpiece e13 cannot be detected by the image processing, and an outline line between workpieces e12 and e14 cannot be detected by the image processing, and workpieces e12 and e14 are incorrectly identified as one workpiece when, as in Fig. 7D representation, for example, a color in an area of ​​workpiece d13 is very close to a background color in the image, and workpieces d12 and d14 are close together with an indistinct boundary between them. The user views the image as shown in Fig. 7E displays a two-dimensional image and training data about the workpiece, shown on display unit 35. The user then determines whether the training data (the two-dimensional image and the training data) displayed on display unit 35 and how it is displayed. Fig. 7E shown are correct.

[0046] If the user determines that the learning data, as in Fig. 7E shown are incorrect if, for example, the user determines that the presence of the workpiece is in e13 in Fig. If 7E is not recognized, the user corrects a presence area of ​​the workpiece and revises it by clicking and adding the missing outline of e13 with a mouse or similar tool. Furthermore, if the user determines that e12 and e14 are in Fig. If 7E does not refer to one workpiece but to two, the user performs a revision by defining two workpieces by editing and adding the missing boundary between e12 and e14 using the mouse or similar tools. Furthermore, if the user determines that an outline of a marking is located in an area of ​​e11 in Fig. 7E is extracted and the label surrounded by the outline is incorrectly identified as a cardboard box, the user performs a revision by deleting the outline surrounding the label. Furthermore, if the user determines that an outline is incorrectly extracted from a background area where no cardboard box is present, the user performs a revision by deleting any excess outline. In this way, final training data (training data f11 to f14) is captured, which has been revised by the user and is processed as described in Fig. 7F is shown, and the training data is stored in data storage unit 33. It should be noted that, for example, with reference to Fig. 9 described, a revision of the training data by the user is preferably used in the determination of areas of a raw material made of metal and a raw material made of plastic, which can be directly determined by the user (the person) who refers to an image (for example, a color image or a grayscale image) and the like.

[0047] This can lead to various problems, such as extracting an outline from a workpiece area if the color within that area is uneven, missing part of a workpiece outline, or failing to extract adjacent outlines when multiple workpieces are in close proximity. Furthermore, if the captured image has a low resolution or poor lighting conditions during capture, resulting in likely noise, an outline may be erroneously extracted from a background area where no workpiece is present.Therefore, such an extraction result is / are preferably displayed above an image on the display unit 35 and the extraction result is shown to the user, an incorrect outline is deleted / revised by a visual inspection by the user, a missing outline is added, training data that reflects a correct workpiece area is revised and training data is generated.

[0048] In other words, if training data containing an incorrect workpiece area is used, learning cannot be performed correctly, and it is difficult to create a learned model capable of accurately performing inference calculations on a specific area of ​​the workpiece. Conversely, by presenting training data and an image (training data) generated once to the user, and then revising the training data based on user input, the accuracy (reliability) of the training data can be improved. It should be noted that the processing described above is performed by the user; however, the user does not have to perform the training process from scratch and only adds revisions to an incorrect part. Thus, high labor costs and enormous work time are avoided.It should be noted that the user's modification of the training data, as described above, is not limited to the fourth example and can be widely used for the examples of the training data generation device, the robot system, the training data generation method, and the training data generation program according to the present embodiment.

[0049] Fig. Figure 8 is a flowchart describing an example of processing in the fifth example of the training data generation program according to the present embodiment. As in Fig. As shown in Figure 8, in step ST51, at the beginning of an example for processing the learning data generation program in the fifth example, camera 4 captures a two-dimensional image of the area where workpiece D is present, and a three-dimensional sensor also captures three-dimensional point group data (three-dimensional data) over the area where workpiece D is present. Furthermore, the processing proceeds to step ST52, and the data acquisition unit 31 estimates a shape area by performing image processing on the two-dimensional image.

[0050] The processing then proceeds to step ST53, where area information about workpiece D is corrected based on the three-dimensional point group data. To correct this, the three-dimensional point group data and a two-dimensional image of each of a plurality of workpieces are compared with a calculation result of a shape area in the two-dimensional image. Furthermore, any background area, obstacle area, area of ​​an adjacent workpiece, or similar that may have been erroneously included in the calculation result is excluded by means of a difference in a three-dimensional position included in the three-dimensional point group data, and corrected training data about the workpiece is acquired.

[0051] The processing then proceeds to step ST54, and the data holding unit 33 stores the training data and the two-dimensional image as training data, and the processing ends. It should be noted that when the area information about the workpiece D is corrected by the two-dimensional image based on the three-dimensional point group data, for example, a groove, a gap, a step, a three-dimensional planar surface, a three-dimensional curved surface, and the like are preferably extracted as features in the two-dimensional image. In this way, even if a workpiece and a background around the workpiece, a boundary between workpieces, or the like cannot be distinguished in a two-dimensional image, the reliability (accuracy) of the training data can be improved by comparison with three-dimensional point group data.It should be noted that in the present fifth example, step ST55 also corresponds to step ST47 in the section with reference to . Fig. The fourth example described in 6 corresponds to the processing in steps ST45 and ST46 in the fourth example, which can be added and carried out as described above.

[0052] The learning data generation program according to the present embodiment described above can be recorded and provided on a computer-readable medium for non-temporary recording or on a non-volatile semiconductor storage device, or it can be provided via wired or wireless data transmission. In this case, a computer-readable medium for non-temporary recording could, for example, be an optical disc such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM, a hard disk device, or the like. Furthermore, a PROM (Programmable Read Only Memory), flash memory (registered trademark), and the like are conceivable as non-volatile semiconductor storage devices.Furthermore, distribution from the server device can be provided via a wired or wireless WAN (Wide Area Network), LAN (Local Area Network), or via the Internet.

[0053] As described in detail above, according to the learning data generation device, the robot system, the learning data generation method and the learning data generation program, annotation can be performed without high labor costs and enormous working time in the present embodiment.

[0054] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. These embodiments include various additions and substitutions, without deviating from the core concept of the invention or from the idea and essence of the invention, which result from the content described in the claims and their equivalents. Modifications, partial deletions, and the like are possible. For example, in the embodiments described above, the sequence of each operation and the sequence of each process are presented as examples and are not limited to these. The same applies if numerical values ​​or equations are used in the description of the embodiments described above.

[0055] With regard to the embodiments and variants described above, the following descriptions are further disclosed. Annex 1

[0056] A learning data generation device (3) that generates learning data used for machine learning, wherein the learning data generation device includes: a data acquisition unit (31) that captures an image of a presence area of ​​a plurality of workpieces (D0 to D9, D); a data processing unit (32) that estimates at least area information reflecting an area span of an entirety or part of at least one of the workpieces (D0, D) based on the captured image and generates training data containing the estimated area information; and a data storage unit (33) that stores the generated training data and the image as training data. Appendix 2

[0057] The learning data generation device according to Annex 1, wherein Area information about the workpiece (D0, D) includes outline area information reflecting an outline of the workpiece (D0, D), pick-up area information for adsorbing, suction or gripping the workpiece (D0, D) and / or local area information about the workpiece (D0, D). Appendix 3

[0058] The learning data generation device according to Annex 2, wherein The local area information about the workpiece (D0, D) includes an area that is a flat surface, a curved surface, or has a large area on the workpiece (D0, D), an area on the workpiece (D0, D) that is not slippery, and / or an area with high density on the workpiece (D0, D). Appendix 4

[0059] The learning data generation device according to any one of Annexes 1 to 3, which further includes an acceptance unit (34) wherein the receiving unit (34) accepts revision information based on area information about at least one of the workpieces (D0, D), and the data processing unit (32) revises and outputs the training data based on the revision information received by the acceptance unit (34). Appendix 5

[0060] The learning data generation device according to any one of Annexes 1 to 4, wherein the data acquisition unit (31) further acquires a learned model, and The data processing unit (32) estimates area information about the workpiece (D0, D) based on the image and the learned model and generates the training data. Appendix 6

[0061] The learning data generation device according to any one of Annexes 1 to 5, wherein the data acquisition unit (31) captures a two-dimensional image of a presence area of ​​the majority of workpieces (D0 to D9, D), and the data processing unit (32) estimates the area information based on the two-dimensional image and generates the training data. Appendix 7

[0062] The learning data generation device according to any one of Annexes 1 to 5, wherein the data acquisition unit (31) acquires a two-dimensional image and three-dimensional data about a presence area of ​​the majority of workpieces (D0 to D9, D), and the data processing unit (32) estimates the area information based on the two-dimensional image and the three-dimensional data and generates the training data. Appendix 8

[0063] The learning data generation device according to Annex 7, wherein The data processing unit (32) performs an analysis of the three-dimensional data, estimates three-dimensional area information about the workpiece based on an analysis result, estimates area information about the workpiece in the two-dimensional image by comparing the three-dimensional data with the two-dimensional image, and generates the training data. Appendix 9

[0064] The learning data generation device according to Annex 8, wherein The data processing unit (32) estimates three-dimensional area information about the workpiece (D0, D) based on a processing result of an operation, as an analysis of the three-dimensional data, a plane analysis, a curvature analysis, a blob analysis, a coordinate system analysis, a position analysis, a depth analysis, a scale analysis, a feature analysis, a neighborhood point analysis, a network analysis and / or a voxel analysis. Appendix 10

[0065] The learning data generation device according to Annex 8 or 9, wherein the data processing unit (32) revises and outputs the training data based on area information about the workpiece (D0, D), which is estimated based on an analysis result of the three-dimensional data. Annex 11

[0066] The learning data generation device according to any one of Annexes 4 to 6, wherein The data processing unit (32) extracts a feature by performing image processing based on area information about at least one of the workpieces (D0, D) accepted by the receiving unit (34) and the image, estimates area information about the majority of workpieces (D0, D) in the image by performing a comparison processing of the extracted feature and generates the training data. Appendix 12

[0067] The learning data generation device according to any one of Annexes 1 to 11, wherein The data processing unit (32) performs image processing based on the image, estimates area information about the workpiece (D0, D) based on a processing result of the image processing and the image, and generates the training data. Appendix 13

[0068] The learning data generation device according to Annex 11 or 12, wherein The data processing unit (32) estimates area information about the workpiece (D0, D) based on a processing result of an operation, such as image processing, pattern matching, plane matching, curvature matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis and / or color information analysis. Appendix 14

[0069] The learning data generation device according to any one of Annexes 11 to 13, wherein the data processing unit (32) revises and outputs the training data based on area information about the workpiece (D0, D), which is estimated based on a processing result of the image processing. Appendix 15

[0070] The learning data generation device according to any one of Annexes 1 to 14, which further includes a display unit (35) the display unit (35) displays the image and the learning data. Appendix 16

[0071] The learning data generation device according to Annex 15, wherein The data processing unit (32) includes processing a reference to the image and the training data displayed on the display unit (35) and revising the training data. Appendix 17

[0072] A robot system (100) that includes: a robot (1) that performs a specified processing operation on the majority of workpieces (D0 to D9, D); a camera (4) that captures a presence area of ​​the majority of workpieces (D0 to D9, D) and outputs the image; the training data generation device (3) according to any one of Annexes 1 to 16, which receives the image from the camera (4) and outputs the training data; a machine learning device that receives the learning data from the learning data generation device (3) and performs machine learning; and a robot control device (2) that receives an output from the machine learning device and controls the robot (1). Appendix 18

[0073] A training data generation method for generating training data that can be used for machine learning, wherein the training data generation method includes: a data acquisition step to capture an image of a presence area of ​​a plurality of workpieces (D0 to D9, D); a data processing step for estimating at least area information reflecting a range of an entirety or part of at least one of the workpieces (D0, D), based on the captured image, and for generating training data containing the estimated area information; and a data storage step to save the generated training data and the image as training data. Annex 19

[0074] The learning data generation procedure according to Annex 18, which further includes: further capture of a learned model in the data acquisition step that was generated by performing prior learning; and, Estimating area information about the workpiece (D0, D) based on the image and the learned model, and generating the training data in the data processing step. Appendix 20

[0075] The learning data generation procedure according to Annex 18 or 19, which further includes: an acceptance step; Accepting area information about at least one of the workpieces (D0, D) in the acceptance step; and Extracting a feature by performing image processing based on the assumed area information and the image for the majority of workpieces (D0 to D9, D), estimating area information about the majority of workpieces (D0 to D9, D) in the image by performing a comparison processing of the extracted feature and generating the training data in the data processing step. Annex 21

[0076] The training data generation procedure according to any one of Annexes 18 to 20, which further includes, Performing image processing based on the image, estimating area information about the workpiece (D0, D) based on a processing result of the image processing and the image, and generating the training data in the data processing step. Annex 22

[0077] The training data generation procedure according to any one of Annexes 18 to 21, which further includes: further acquisition of a two-dimensional image and three-dimensional data over a presence area of ​​the majority of workpieces (D0 to D9, D) in the data acquisition step; and, Estimating the area information based on the two-dimensional image and the three-dimensional data, and generating the training data in the data processing step. Annex 23

[0078] The training data generation procedure according to any one of Annexes 18 to 22, which further includes: an acceptance step; Accepting revision information based on area information about at least one of the workpieces (D0, D) in the acceptance step; and, Revising the training data based on the assumed revision information in the data processing step. Annex 24

[0079] A learning data generation program that generates learning data used for machine learning, wherein the learning data generation program causes an arithmetic processing device to execute: a data acquisition step to capture an image of a presence area of ​​a plurality of workpieces (D0 to D9, D); a data processing step for estimating at least area information reflecting a range of an entirety or part of at least one of the workpieces (D0, D), based on the captured image, and for generating training data containing the estimated area information; and a data storage step to save the generated training data and the image as training data. List of reference symbols 1 robot 2 Robot control unit 3 Learning data generation device 4 cameras 10 robot mechanism units 11 Arm 12 End effector (hand area) 31 Data acquisition unit 32 Data processing unit 33 Data storage unit 34 Acceptance unit 35 Display unit 100 robot systems A1 Form area A2 area D, D1 to D9 Workpiece (object) QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2022-118300

[0004] JP 2014-059729

[0004]

Claims

[1] Learning data generation device that generates learning data used for machine learning, wherein the learning data generation device comprises: a data acquisition unit that captures an image of the presence area of ​​a plurality of workpieces; a data processing unit that estimates at least area information reflecting the range of an entirety or part of at least one of the workpieces, based on the captured image, and generates training data containing the estimated area information; and a data storage unit that stores the generated training data and the image as training data. [2] Learning data generation device according to claim 1, wherein area information about the workpiece includes outline area information reflecting an outline of the workpiece, removal area information for adsorbing, suction or gripping the workpiece and / or local area information about the workpiece. [3] Learning data generation device according to claim 2, wherein the local area information about the workpiece includes an area which is a flat surface, which is a curved surface or which has a large area on the workpiece, an area on the workpiece which is not slippery, and / or an area with a high density on the workpiece. [4] Learning data generation device according to any one of claims 1 to 3, further comprising an acceptance unit, wherein the receiving unit accepts revision information based on area information about at least one of the workpieces, and The data processing unit revises and outputs the training data based on the revision information accepted by the acceptance unit. [5] Learning data generation device according to any one of claims 1 to 4, wherein The data acquisition unit also captures a learned model, and The data processing unit estimates area information about the workpiece based on the image and the learned model and generates the training data. [6] Learning data generation device according to any one of claims 1 to 5, wherein The data acquisition unit captures a two-dimensional image of the area where the majority of workpieces are present, and The data processing unit estimates the area information based on the two-dimensional image and generates the training data. [7] Learning data generation device according to any one of claims 1 to 5, wherein The data acquisition unit captures a two-dimensional image and three-dimensional data about the presence area of ​​the majority of workpieces, and The data processing unit estimates the area information based on the two-dimensional image and the three-dimensional data and generates the training data. [8] Learning data generation device according to claim 7, wherein the data processing unit performs an analysis of the three-dimensional data, estimates three-dimensional area information about the workpiece based on an analysis result, estimates area information about the workpiece in the two-dimensional image by comparing the three-dimensional data with the two-dimensional image and generates the training data. [9] Learning data generation device according to claim 8, wherein the data processing unit estimates three-dimensional area information about the workpiece based on a processing result of an operation, as an analysis of the three-dimensional data, a plane analysis, a curvature analysis, a blob analysis, a coordinate system analysis, a position analysis, a depth analysis, a scale analysis, a feature analysis, a neighborhood point analysis, a network analysis and / or a voxel analysis. [10] Learning data generation device according to claim 8 or 9, wherein the data processing unit reworks and outputs the training data based on area information about the workpiece, which is estimated on the basis of an analysis result of the three-dimensional data. [11] Learning data generation device according to any one of claims 4 to 6, wherein the data processing unit extracts a feature by performing image processing based on area information about at least one of the workpieces accepted by the acceptance unit and the image, estimates area information about the plurality of workpieces in the image by performing comparison processing of the extracted feature and generates the training data. [12] Learning data generation device according to any one of claims 1 to 11, wherein the data processing unit performs image processing based on the image, estimates area information about the workpiece based on a processing result of the image processing and the image, and generates the training data. [13] Learning data generation device according to claim 11 or 12, wherein the data processing unit estimates area information about the workpiece based on a processing result of an operation such as image processing, pattern matching, plane matching, curvature matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis and / or color information analysis. [14] Learning data generation device according to any one of claims 11 to 13, wherein the data processing unit reworks and outputs the training data based on area information about the workpiece, which is estimated on the basis of a processing result of the image processing. [15] Learning data generation device according to any one of claims 1 to 14, further comprising a display unit, wherein The display unit shows the image and the learning data. [16] Learning data generation device according to claim 15, wherein the data processing unit includes processing a reference to the image and the training data displayed on the display unit and revising the training data. [17] Robot system that features: a robot that performs a predetermined processing operation on the majority of workpieces; a camera that captures the area where the majority of workpieces are present and outputs the image; the learning data generation device according to any one of claims 1 to 16, which receives the image from the camera and outputs the learning data; a machine learning device that receives the training data from the training data generation device and performs machine learning; and a robot control device that receives an output from the machine learning device and controls the robot. [18] Learning data generation method for generating learning data that can be used for machine learning, wherein the learning data generation method has: a data acquisition step to capture an image of the presence area of ​​a plurality of workpieces; a data processing step for estimating at least area information reflecting a range of an entirety or part of at least one of the workpieces, based on the captured image, and for generating training data containing the estimated area information; and a data storage step to save the generated training data and the image as training data. [19] Learning data generation method according to claim 18, further comprising: further capture of a learned model in the data acquisition step, which was generated by performing prior learning; and Estimating area information about the workpiece based on the image and the learned model, and generating the training data in the data processing step. [20] Learning data generation method according to claim 18 or 19, further comprising: an acceptance step; Accepting area information about at least one of the workpieces in the acceptance step; and Extracting a feature by performing image processing based on the assumed area information and the image for the majority of workpieces, estimating area information about the majority of workpieces in the image by performing a comparison processing of the extracted feature, and generating the training data in the data processing step. [21] Learning data generation method according to any one of claims 18 to 20, further comprising, Performing image processing based on the image, estimating area information about the workpiece based on a processing result of the image processing and the image, and generating the training data in the data processing step. [22] Learning data generation method according to any one of claims 18 to 21, further comprising: further acquisition of a two-dimensional image and three-dimensional data over a presence area of ​​the majority of workpieces in the data acquisition step; and Estimating the area information based on the two-dimensional image and the three-dimensional data, and generating the training data in the data processing step. [23] A learning data generation method according to any one of claims 18 to 22, further comprising: an acceptance step; Accepting revision information based on area information about at least one of the workpieces in the acceptance step; and Revising the training data based on the assumed revision information in the data processing step. [24] Learning data generation program that generates learning data used for machine learning, wherein the learning data generation program causes an arithmetic processing device to execute: a data acquisition step to capture an image of the presence area of ​​a plurality of workpieces; a data processing step for estimating at least area information reflecting a range of an entirety or part of at least one of the workpieces, based on the captured image, and for generating training data containing the estimated area information; and a data storage step to save the generated training data and the image as training data.

Images