Teacher data generation device and teacher data generation program

Abstract

To shorten time required for update of teacher data.SOLUTION: A control device 1 comprises: a label imparting unit 142 which imparts a label showing predetermined information to an object included in an image according to input operation of a user; a generation unit 143 which generates a set of an image and a label imparted to the image as teacher data; and a storage unit 12 which stores a second image Db on which detection by a detection model is performed, the detection being failed for an object, in association with a detection result by the detection model. The label imparting unit 142 imparts a label showing predetermined information to an object failed to be detected in the second image Db according to input operation of a user, on the other hand, substitutes the detection result for a label showing predetermined information on a successfully detected object. The generation unit 143 generates a set of the second image Db and the label imparted to the object failed to be detected and the detection result of the successfully detected object as the teacher data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The technology disclosed herein relates to a teacher data generation device and a teacher data generation program. [Background technology]

[0002] Conventionally, devices that generate training data used in machine learning have been known. For example, a device disclosed in Patent Document 1 performs labeling, i.e., annotation processing, on a large number of images that show an object to be detected. Then, training data in which the images and labels are associated with each other is generated. A learning model is generated by supervised machine learning using the training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-197978 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when an object detection process is performed on an image using the generated learning model, if an undetected object is found, the learning model needs to be re-trained and updated. Therefore, it is necessary to assign a label to the image containing the undetected object and generate new training data. However, if the learning model is updated after the device equipped with the learning model has been in operation, the device must be stopped, so it is desirable to generate new training data as quickly as possible.

[0005] The technology disclosed herein has been made in consideration of these points, and its purpose is to reduce the time required to generate training data for retraining an already trained detection model. [Means for solving the problem]

[0006] The technology disclosed herein is a training data generation device that generates training data for retraining a trained detection model that detects objects included in images and predetermined information about the objects. The training data generation device includes a label assignment unit that assigns labels indicating the predetermined information to objects included in images in response to a user's input operation, a generation unit that generates pairs of the images and the labels assigned to the images as training data, and a storage unit that stores images in which detection using the detection model has been performed and in which the detection of at least one object has failed, linked to the detection results using the detection model. The label assignment unit assigns the labels indicating the predetermined information to objects that have failed to be detected in the images stored in the storage unit in response to a user's input operation, while substituting the detection results for the labels indicating the predetermined information of objects that have been successfully detected. The generation unit generates training data from pairs of the images stored in the storage unit, the labels assigned to the objects that have failed to be detected, and the detection results of the objects that have been successfully detected.

[0007] Another technology disclosed herein is a training data generation program that causes a computer to realize a function of generating training data for retraining a trained detection model that detects objects included in images and predetermined information about the objects. The training data generation program causes a computer to realize a function of assigning labels indicating the predetermined information to objects included in images in response to a user's input operation, a function of generating pairs of the images and the labels assigned to the images as training data, a function of storing images in which detection using the detection model has been performed and in which the detection of at least one object has failed, in association with the detection results using the detection model, a function of assigning the labels indicating the predetermined information to objects in the images in the storage unit in response to a user's input operation, while substituting the labels indicating the predetermined information for objects that have been successfully detected with the detection results, and a function of generating pairs of the images in the storage unit, the labels assigned to the objects that have been unsuccessfully detected, and the detection results for the objects that have been successfully detected, as training data. [Effects of the Invention]

[0008] According to the above-described teacher data generation device, it is possible to reduce the time required to generate teacher data for retraining an already trained detection model.

[0009] According to the above-described training data generation program, it is possible to reduce the time required to generate training data for retraining an already trained detection model. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a robot system. [Figure 2] FIG. 2 is a plan view showing a bucket with loose objects placed therein. [Figure 3] FIG. 3 is a block diagram showing a schematic configuration of the control device and its peripheral devices. [Figure 4] FIG. 4 is a functional block diagram of the processing unit of the control device. [Figure 5] FIG. 5 is a flowchart showing the generation process of the processing unit of the control device. [Figure 6] FIG. 6 is a diagram showing an example of a state in which the first image is selected on the display unit. [Figure 7] FIG. 7 is a diagram showing an example of how a mark is added to an object in the first image on the display unit. [Figure 8] FIG. 8 is a diagram showing an example of a state in which marks are added to all objects in the first image on the display unit. [Figure 9] FIG. 9 is a diagram showing an example of the second image on the display unit. [Figure 10] FIG. 10 is a diagram showing an example of the second image on the display unit. [Figure 11] FIG. 11 is a diagram showing an example of the second image on the display unit. [Figure 12] FIG. 12 is a diagram showing an example of the second image on the display unit. [Figure 13] FIG. 13 is a diagram showing an example of the second image on the display unit. [Figure 14] FIG. 14 is a diagram showing an example of the second image on the display unit. [Figure 15] FIG. 15 is a diagram showing an example of the second image on the display unit. [Figure 16] FIG. 16 is a diagram showing an example of the second image on the display unit. DETAILED DESCRIPTION OF THE INVENTION

[0011] Exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0012] Fig. 1 is a diagram showing a schematic configuration of a robot system 100. Fig. 2 is a plan view showing a bucket B in which objects W are randomly placed.

[0013] The robot system 100 includes a control device 1, an imaging device 2, a robot control device 3, and a robot 4. The control device 1 is an example of a teacher data generation device. In the robot system 100, for example, the imaging device 2 captures an image of an object W placed loosely in a bucket B, and the control device 1 detects the object W from the image captured by the imaging device 2. The robot control device 3 causes the robot 4 to grasp the object W in the bucket B detected by the control device 1 and transport it to a predetermined location. In this example, the object W is a glass bottle.

[0014] The robot 4 has a base 41 and a robot arm 42 rotatably connected to the base 41. The robot arm 42 is, for example, a vertical articulated arm that grasps the object W. Specifically, the robot arm 42 is provided with a hand 43 for grasping the object W.

[0015] The robot control device 3 controls the robot 4. The robot control device 3 controls various operations of the robot arm 42 including the hand 43. The control device 1 and the robot control device 3 can communicate with each other via wire or wirelessly. The robot control device 3 determines the position of the object W in the bucket B based on the detection signal of the object W output from the control device 1. The robot control device 3 controls the robot arm 42 in accordance with the determined position of the object W to cause the robot arm 42 to grasp the object W.

[0016] The imaging device 2 is provided, for example, above the bucket B. The imaging device 2 is capable of communicating with the control device 1 via wire or wirelessly. The imaging device 2 captures an image of the bucket B, i.e., the object W inside the bucket B, and outputs the captured image to the control device 1. The imaging device 2 is, for example, a camera. Note that the imaging device 2 may be capable of communicating with the robot control device 3 in addition to the control device 1 via wire or wirelessly.

[0017] The control device 1 detects one or more objects W from the captured image output from the imaging device 2. More specifically, the control device 1 detects one or more objects W and predetermined information about the objects W from the captured image. The control device 1 outputs the predetermined information about the detected objects W to the robot control device 3. The robot control device 3 causes the robot arm 42 to grasp the target object W in the bucket B based on the predetermined information about the object W from the control device 1. In this example, the predetermined information about the object W is position information about the object W in the image and attribute information about the object W.

[0018] The control device 1 detects predetermined information about the object W from the image captured by the imaging device 2 using a detection model trained by machine learning. The control device 1 also has a function of generating training data for training the detection model. Specifically, the control device 1 assigns a label to a first image containing the object W to generate training data for training the pre-training detection model. The detection model trained using the first image in this way detects predetermined information about the object W in the image. At this time, if there is an object W that the detection model fails to detect, the image at that time is stored as a second image. The control device 1 then assigns a label to the stored second image to generate training data for retraining the trained detection model, i.e., for updating the trained detection model.

[0019] The label is also referred to as ground truth data or a tag. In this example, the label indicates predetermined information about the object W. That is, the label is the position of the object W in the image and the attribute of the object W. In this example, the attribute of the object W is the color of the object W.

[0020] The training data is data that is used to train a detection model. The training data is also referred to as learning data or training data. In this example, the training data is data that pairs a first image and a second image with labels assigned to the first image and the second image. In other words, the training data is data that associates the first image and the second image with the labels assigned to the first image and the second image.

[0021] In machine learning, learning is performed using multiple sets of training data. In this example, a collection of multiple sets of training data is referred to as a training data set, and each piece of data included in the training data set is referred to as training data. In other words, the term training data refers to the above-mentioned set, and the term training data set refers to a collection of sets.

[0022] The detection model is a learning model that uses machine learning. In this example, the detection model receives an image containing an object W as input and outputs predetermined information about the object W. Various known methods can be used for the machine learning itself, such as neural networks, reinforcement learning, or deep learning. For the machine learning, supervised learning or semi-supervised learning can be used.

[0023] 3 is a block diagram showing a schematic configuration of the control device 1 and its peripheral devices. The control device 1 includes an input unit 11, a storage unit 12, a display unit 13, and a processing unit .

[0024] The input unit 11 accepts an input operation from a user. The input unit 11 outputs an input signal corresponding to the input operation to the processing unit 14. The input unit 11 is, for example, a touch panel, a pointing device such as a mouse, or a keyboard.

[0025] The storage unit 12 is a computer-readable storage medium that stores various programs and various data. The storage unit 12 is formed of a magnetic disk such as a hard disk, an optical disk such as a CD-ROM or DVD, or a semiconductor memory.

[0026] Specifically, the storage unit 12 stores a first image Da, a second image Db, a teacher dataset DS, a generation program PG, and the like.

[0027] The first image Da is an image containing an object W. The first image Da is an image used to train a pre-learning detection model, and is a photographed image of the object W in the bucket B taken by the imaging device 2. The second image Db is an image on which detection has been performed using the trained detection model, and in which detection has failed for at least one object W. The trained detection model is, for example, a detection model trained on training data generated by the generation unit 143, described later, based on the first image Da. The image on which detection is performed using the detection model is a photographed image of the object W in the bucket B taken by the imaging device 2, but is a different image from the first image Da. The training dataset DS includes training data generated by the generation unit 143.

[0028] Here, failure of detection by the detection model for object W means that the detection model was unable to detect predetermined information about object W or was unable to properly detect the predetermined information about object W, and hereinafter, an object W that was unsuccessfully detected by the detection model will be referred to as an "object W that was unsuccessfully detected." Conversely, success of detection by the detection model for object W means that the detection model was able to properly detect the predetermined information about object W, and hereinafter, an object W that was unsuccessfully detected by the detection model will be referred to as an "object W that was successfully detected."

[0029] The storage unit 12 also stores the second image Db in association with the detection result by the detection model. The detection result by the detection model is predetermined information about the object W that is properly detected by the detection model, that is, predetermined information about the object W that is successfully detected.

[0030] The generation program PG is an example of a teacher data generation program. The generation program PG is a program for causing a computer, i.e., the processing unit 14, to realize various functions for generating teacher data. The generation program PG is read and executed by the processing unit 14.

[0031] The display unit 13 displays the first image Da and the second image Db. In this example, the display unit 13 displays a GUI (Graphical User Interface) screen, and displays the first image Da and the second image Db on the GUI screen. The display unit 13 is, for example, a liquid crystal display or an organic EL display.

[0032] 4 is a functional block diagram of the processing unit 14 of the control device 1. The processing unit 14 has various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and / or a DSP (Digital Signal Processor), and various semiconductor memories such as a RAM (Random Access Memory) and / or a ROM (Read Only Memory). The processing unit 14 reads and executes the generation program PG etc. from the storage unit 12.

[0033] Specifically, the processing unit 14 has a display control unit 141, a label assignment unit 142, a generation unit 143, a learning unit 144, and a detection unit 145 as functional blocks.

[0034] The display control unit 141 acquires a first image Da and a second image Db. Specifically, the display control unit 141 acquires the first image Da by reading out the first image Da from the storage unit 12. The display control unit 141 also acquires the second image Db by reading out the second image Db from the storage unit 12. The display control unit 141 causes the display unit 13 to display the first image Da and the second image Db acquired from the storage unit 12.

[0035] Furthermore, the display control unit 141 causes the display unit 13 to enlarge and display an image selected by the user's input operation from the plurality of first images Da and second images Db displayed on the display unit 13.

[0036] The labeling unit 142 assigns, to an object W included in the first image Da, a label indicating predetermined information of the object W, i.e., a label indicating position information and attribute information of the object W, in response to an input operation by the user. In other words, the labeling unit 142 performs annotation to assign a label to the first image Da.

[0037] The generation unit 143 generates a pair of the first image Da and the label assigned to the first image Da as training data. That is, training data that associates the first image Da with the label assigned to the first image Da is generated. The training data generated in this manner is stored in the storage unit 12 as a training data set DS.

[0038] The learning unit 144 generates a detection model. Specifically, the learning unit 144 reads training data from the storage unit 12 and generates a trained detection model by machine learning the training data. The trained detection model detects predetermined information of an object W included in an image. More specifically, when an image containing an object W is input, the detection model outputs predetermined information of the object W, i.e., the position of the object W and the attributes of the object W (in this example, the color). The trained detection model generated by the learning unit 144 is stored in, for example, the storage unit 12.

[0039] The detection unit 145 uses the trained detection model to detect an object W from within an image, and further detects the position and color of the object W. Specifically, the detection unit 145 inputs the image into the detection model read from the storage unit 12, and outputs the position and color of the object W. That is, the detection unit 145 performs detection processing using the detection model.

[0040] The detection unit 145 stores an image in which detection has been performed but in which the object W has been unsuccessfully detected, i.e., a second image Db, in the storage unit 12. That is, after the detection unit 145 performs the detection process, it stores the second image Db in the storage unit 12. At that time, the detection unit 145 associates the second image Db with the detection result based on the detection model and stores it in the storage unit 12. Note that the detection unit 145 may store not only the second image Db but also an image in which there is no object W in which detection has been unsuccessfully performed in the storage unit 12.

[0041] The labeling unit 142 assigns labels indicating predetermined information to objects W for which detection has failed in the second image Db in response to a user's input operation, while substituting the detection result by the detection model for labels indicating predetermined information for objects W for which detection has been successful. For objects W for which detection has been successful, the position and color of the objects W have been appropriately detected. Therefore, by using the successful detection result as a label, the process of assigning labels in response to a user's input operation can be omitted. In other words, the labeling process for the second image Db is performed only for objects W for which detection has failed, rather than for all objects W appearing in the second image Db. In this example, objects W for which detection has failed are undetected objects W or objects W that have been erroneously detected.

[0042] The generation unit 143 generates, as training data, a set of the second image Db, the label attached to the object W for which detection failed, and the detection result of the object W for which detection was successful. In this way, the generation unit 144 generates training data for retraining the already-trained detection model.

[0043] More specifically, the labeling unit 142 assigns a label to an object W included in the first image Da and the second image Db displayed on the display unit 13 by assigning a mark indicating the predetermined information to the object W through a user's input operation. Note that with respect to the second image Db, a mark is assigned to an object W included in the second image Db for which detection has failed through a user's input operation. Specifically, the labeling unit 142 assigns a mark to the object W in accordance with the user's input operation, thereby assigning the position of the mark as the position of the object W. The position of the mark is set as a position where the robot arm 42 performs a predetermined process on the object W based on the position of the mark. Specifically, the position of the mark is set as a position where the robot arm 42 grasps the object W. That is, in this example, the predetermined process performed by the robot arm 42 on the object W is a process of grasping the object W. Furthermore, the labeling unit 142 assigns attribute information of the object W by assigning a mark to the object W in a color that is distinguished according to the attribute information of the object W. In this example, the labeling unit 142 assigns color information about the bottle by attaching a mark to the bottle in a color that is differentiated according to the color of the bottle.

[0044] Furthermore, when the display control unit 141 displays the second image Db on the display unit 13, it attaches a mark indicating the detection result by the detection model to the successfully detected object W included in the second image Db. That is, the display control unit 141 attaches a mark similar to the label attachment process by the label attachment unit 142, that is, a mark indicating predetermined information of the successfully detected object W in the second image Db. Therefore, the detection result by the detection model linked to the second image Db can be used in place of the label attachment process.

[0045] Next, an overview of the processing in the control device 1 will be described. In the control device 1, the generation unit 143 assigns a label to the first image Da and generates training data for training a pre-learning detection model. Hereinafter, this training data generation processing will be referred to as the first generation processing. Subsequently, the learning unit 144 generates a detection model through machine learning using the training data generated in the first generation processing. Next, the detection unit 145 performs a detection processing on the image output from the imaging device 2 using the detection model, and associates the second image Db that has undergone the detection processing with the detection result of the detection model and stores it in the storage unit 12. The generation unit 143 assigns a label to the second image Db and generates training data for updating the trained detection model so that the trained detection model can be retrained. Hereinafter, this training data generation processing will be referred to as the second generation processing. The learning unit 144 updates the detection model using the training data generated in the second generation processing.

[0046] Next, the details of the first and second generation processes of the teacher data described above will be explained with reference to Fig. 5. Fig. 5 is a flowchart showing the generation process of the teacher data generation device. This flowchart is common to the first and second generation processes.

[0047] FIG. 6 is a diagram showing an example of a state in which a first image Da is selected on the display unit 13. First, in step S1, the first image Da is acquired. Specifically, the display control unit 141 reads out the first image Da from the storage unit 12. At this time, the number of first images Da to be read out can be set. As shown in FIG. 6, the multiple first images Da acquired by the display control unit 141 are displayed as a thumbnail list 131 on the display unit 13. Note that in this generation process, the display control unit 141 causes the display unit 13 to display a GUI screen.

[0048] In the following step S2, a first image Da is selected from the thumbnail list 131. Specifically, the user selects a first image Da to which a label is to be assigned from the thumbnail list 131. More specifically, the user selects the target first image Da with a mouse pointer P. In this example, the first image Da at the top of the image list is selected with the pointer P (see FIG. 6). The display control unit 141 enlarges and displays the first image Da selected by the user in the center of the display unit 13. Note that in FIG. 6 and subsequent figures, hatched bottles indicate brown bottles or broken pieces.

[0049] FIG. 7 is a diagram showing an example of how a mark is added to the object W in the first image Da on the display unit 13. In the following step S3, the labeling unit 142 adds a label to the enlarged and displayed first image Da. As shown in FIG. 7, the user performs an input operation of designating each object W (bottle) in the first image Da with the pointer P, thereby adding a label to the object W. More specifically, when the user designates a predetermined position on the object W with the pointer P, a mark M is added to the designated position. The predetermined position is designated visually as a position that is thought to be easy for the robot arm 42 to grasp. As a result, a label indicating the position of the object W is added to the first image Da.

[0050] Furthermore, the marks M are color-coded according to the color of the object W. In this example, as shown in the class designation section 132 of the display unit 13, the marks M are color-coded according to whether the bottle is a brown bottle or a transparent bottle. Furthermore, in this example, the marks M are color-coded according to the state of the object W. That is, the marks M are color-coded according to whether they are broken pieces of a bottle or not. The color of the marks M is designated by the user specifying the color and state (i.e., whether they are broken pieces or not) with the pointer P in the class designation section 132. In this way, by designating the color of the marks M attached to the object W, a label indicating the color and state of the object W is assigned to the first image Da. Note that in FIG. 7 and subsequent figures, for convenience of explanation, the shape of the marks M is changed according to the color and state of the object W.

[0051] When the mark M is thus attached to the object W, the position information and class information (i.e., color and state) of the object W are displayed as a label attachment list 136 on the left side of the display unit 13. In the label attachment list 136, position information 137 and class information 138 of each object W to which the mark M is attached are displayed. FIG. 8 is a diagram showing an example of a state in which marks have been attached to all objects W in the first image Da on the display unit 13. As shown in FIG. 8, when the mark M has been attached to all objects W in the first image Da, the attachment of labels to this first image Da is completed. That is, the user presses a save button 139 on the display unit 13 with the pointer P. This confirms the position information and class information of all objects W in the first image Da.

[0052] In step S4, training data is generated. That is, the generation unit 143 generates a pair of a first image Da and a label assigned to the first image Da as training data. For example, if the first image Da includes multiple objects W, a pair of one first image Da and multiple labels is generated as training data. In the subsequent step S5, the generation unit 143 stores the generated training data in the storage unit 12. This completes the first training data generation process. The processes in steps S1 to S5 above are performed for each first image Da in the thumbnail list 131.

[0053] The learning unit 144 generates a trained detection model using the teacher data thus generated. Then, for example, when the robot system 100 is operated, the detection unit 145 performs detection processing on the image captured by the imaging device 2 using the trained detection model. At this time, the second image Db is associated with the detection result by the detection model and accumulated in the storage unit 12.

[0054] Next, the second generation process will be described based on the flowchart of Fig. 5. Figs. 9 to 15 are diagrams showing an example of the second image Db on the display unit 13. Note that Figs. 9 to 15 only show the second image Db displayed on the GUI screen of the display unit 13.

[0055] First, in step S1, the display control unit 141 acquires second images Db. Specifically, the display control unit 141 reads out second images Db from the storage unit 12. At this time, the display control unit 141 can set the number of second images Db to be read out. Similar to the first images Da, the acquired multiple second images Db are displayed as a thumbnail list 131 on the display unit 13. In this second generation process, as in the first generation process, a GUI screen is displayed on the display unit 13.

[0056] In the subsequent step S2, the second image Db is enlarged and displayed. Specifically, the user selects the target second image Db from the thumbnail list 131. The display control unit 141 enlarges and displays the selected second image Db in the center of the display unit 13. At this time, the display control unit 141 attaches a mark M indicating the detection result by the detection model associated with the selected second image Db to the successfully detected object W in the second image Db.

[0057] In the following step S3, a label may be assigned. For example, a case will be described where the second image Db selected in step S2 is the image shown in Fig. 9. The second image Db in Fig. 9 is an image containing an undetected object Wa (i.e., an object W whose detection was unsuccessful) that was not detected by the detection process by the detection unit 145. It is assumed that no erroneously detected object exists. In other words, a mark M is assigned by the display control unit 141 to the detected object W (i.e., an object W whose detection was successful), and no mark M is assigned to the undetected object Wa.

[0058] In the case of the second image Db of FIG. 9, as shown in FIG. 10, the user labels only the undetected objects Wa by attaching a mark M. In this case, similar to the labeling in the generation operation described above, the user attaches the mark M to a predetermined position on the object Wa and specifies the color of the mark M according to the color and state of the object Wa. In this way, for detected objects W, the labeling process is omitted by using the mark M (i.e., the detection result) attached to the object W. Therefore, the time required for the labeling process is shortened. In this way, as shown in FIG. 11, when the labeling process for all undetected objects Wa is completed, step S3 ends. Note that if the attribute of the undetected object Wa is a new attribute, the user presses the class addition button 133 displayed on the display unit 13 with the pointer P to add new attribute information in the class specification unit 132.

[0059] Next, a case will be described in which the second image Db selected in step S2 is the image shown in Fig. 12. The second image Db in Fig. 12 is an image in which an object Wb (i.e., an object W that has failed to be detected) is present that has been erroneously detected by the detection process by the detection unit 145. It is assumed that no undetected object exists.

[0060] In this example, the class information (more specifically, color information) of the object Wb is erroneously detected. In this case, the user designates the erroneous mark M attached to the erroneously detected object Wb with the pointer P and presses the class change button 135 displayed on the display unit 13. After pressing the class change button 135, the user designates the correct color in the class designation unit 132, thereby changing the color of the erroneous mark M (see FIG. 13). In this way, step S3 ends. In this case, too, the labeling process is omitted for the detected object W (i.e., the object W that was successfully detected) by using the mark M (i.e., the detection result) attached to that object W.

[0061] Next, a case will be described in which the second image Db selected in step S2 is the image shown in Fig. 14. The second image Db in Fig. 14 is an image in which an object Wc (i.e., an object W whose detection has failed) is erroneously detected by the detection process by the detection unit 145. It is assumed that no undetected object exists.

[0062] In this example, the position information of the object Wc is erroneously detected. That is, the position of the mark M attached to the object Wc is significantly deviated from the predetermined position. In this case, the user designates the mark M attached to the erroneously detected object Wc with the pointer P and moves it to the predetermined position (see FIG. 15). As a result, the mark M attached to the erroneously detected object Wc is changed to the correct position, and step S3 ends. In this case as well, the labeling process is omitted for the detected object W (i.e., the object W successfully detected) by using the mark M attached to that object W (i.e., the detection result).

[0063] Next, a case will be described where the second image Db selected in step S2 is the image shown in Fig. 16. The second image Db in Fig. 16 is an image in which a mark Ma is added to a position where no object W is present by the detection process performed by the detection unit 145.

[0064] In this example, a mark Ma is attached to the wall of bucket B. In this case, the user designates mark Ma with pointer P and presses delete button 134 on display unit 13. This causes mark Ma to be deleted from second image Db, and step S3 ends. In this case as well, for a detected object W (i.e., an object W that has been successfully detected), the labeling process is omitted by using mark M (i.e., the detection result) attached to the object W.

[0065] When step S3 is completed in this manner, the process proceeds to step S4. In step S4, training data is generated. That is, the generation unit 143 generates a set of the second image Db, the label assigned to the object W for which detection failed, and the detection result of the object W for which detection was successful, as training data. When training data is generated based on the second image Db, the time required for the label assignment process to the second image Db is shortened as described above, and therefore the time required for generating training data is shortened. In the following step S5, the generation unit 143 stores the generated training data in the memory unit 12. This completes the second generation process. The processes of steps S1 to S5 described above are performed for each second image Db in the thumbnail list 131.

[0066] As described above, the control device 1 generates training data for retraining a trained detection model that detects an object W included in an image and predetermined information about the object W. The control device 1 (trainer data generation device) includes a label assignment unit 142 that assigns labels indicating predetermined information to an object W included in an image in response to a user's input operation; a generation unit 143 that generates pairs of the image and the labels assigned to the image as training data; and a storage unit 12 that stores a second image Db in which detection using the detection model has been performed and in which detection of at least one object W has failed, in association with the detection result using the detection model. The label assignment unit 142 assigns labels indicating predetermined information to the object W for which detection has failed in response to a user's input operation in the second image Db stored in the storage unit 12, while substituting the detection result using the detection model for the label indicating the predetermined information for the object W for which detection has been successful. The generation unit 143 generates, as training data, pairs of the second image Db stored in the storage unit 12, the labels assigned to the object W for which detection has failed, and the detection result for the object W for which detection has been successful.

[0067] Furthermore, the generation program PG (teacher data generation program) causes the computer to realize a function of generating teacher data for retraining a trained detection model that detects an object W included in an image and predetermined information about the object W. The generation program PG causes the computer to realize a function of assigning a label indicating predetermined information to an object W included in an image in response to a user's input operation, a function of generating a pair of the image and the label assigned to the image as teacher data, a function of storing a second image Db in which detection using the detection model has been performed but in which detection of at least one object W has failed, in association with the detection result using the detection model, a function of assigning a label indicating predetermined information to the object W for which detection has failed in response to a user's input operation in the stored second image Db, while substituting the label indicating the predetermined information for the object W for which detection has been successful with the detection result using the detection model, and a function of generating a pair of the second image Db, the label assigned to the object W for which detection has failed, and the detection result for the object W for which detection has been successful, as teacher data.

[0068] According to these configurations, predetermined information is appropriately detected for objects W whose detection was successful. Therefore, the labeling process can be substituted by using the successful detection results. In other words, the labeling process for the second image Db is performed only for objects W whose detection failed, rather than for all objects W. This omits the labeling process for objects W whose detection was unsuccessful, thereby reducing the workload of the labeling process for the second image Db. Therefore, the time required for the labeling process for the second image Db can be shortened. The generation unit 143 then generates, as training data, a set of the second image Db, the labels assigned to the objects W whose detection was unsuccessful, and the detection results of the objects W whose detection was successful. In this way, when generating training data based on the second image Db, the time required for the labeling process for the second image Db is shortened. Therefore, the time required for generating training data for retraining a trained detection model can be shortened.

[0069] The control device 1 further includes a display unit 13 that displays the second image Db. The labeling unit 142 assigns a label to an object W that has failed to be detected and is included in the second image Db displayed on the display unit 13, by attaching a mark indicating predetermined information to the object W through an input operation by the user.

[0070] According to this configuration, a label is assigned by attaching a mark M to the object W in the second image Db displayed on the display unit 13, so the labeling process can be performed while visually recognizing the object. Furthermore, by changing the shape and color of the mark M, it becomes easy to identify the label corresponding to the object W. These factors improve the ease of the labeling process.

[0071] The control device 1 further includes a display control unit 141 that causes the display unit 13 to display the second image Db of the memory unit 12. When the display control unit 141 causes the display unit 13 to display the second image Db of the memory unit 12, the display control unit 141 attaches a mark indicating the detection result to the object W that was successfully detected and is included in the second image Db of the memory unit 12.

[0072] According to this configuration, a mark M indicating the detection result is attached to the object W in the second image Db that has been successfully detected, making it easier to use the detection result. Therefore, the labeling process can be easily substituted with the detection result, and the labeling process can be easily omitted.

[0073] In the control device 1, the predetermined information of the object W is position information of the object W. The labeling unit 142 assigns a label to the object W using the position of the mark M added to the object W by the user's input operation as position information of the object.

[0074] According to this configuration, a label indicating the position information of the object W is given by adding the mark M to the displayed object W, which makes the labeling process easier.

[0075] The position of the mark M is set as a position where the robot arm 42 performs a predetermined process on the object W based on the position of the mark M.

[0076] According to this configuration, it is possible to easily grasp the position related to the processing by the robot arm 42 from the position of the mark M. Therefore, it becomes easy to control the processing operation of the robot arm 42 with respect to the object W.

[0077] The position of the mark M is set as the position where the robot arm 42 grasps the object W.

[0078] According to this configuration, the gripping position of the robot arm 42 can be easily determined from the position of the mark M. This makes it easier to control the gripping operation of the robot arm 42 with respect to the object W.

[0079] In the control device 1, the predetermined information of the object W is attribute information of the object W. The labeling unit 142 assigns the attribute information of the object W by attaching a mark M of a color that is distinguished according to the attribute information of the object W to the object W.

[0080] According to this configuration, the marks M are colored differently depending on the attribute information of the objects W, thereby improving the visibility of the attribute information of each object W. This improves the ease of labeling processing.

[0081] In addition, in the control device 1, an object W that has failed to be detected is an object W that has not been detected or an object W that has been erroneously detected.

[0082] According to this configuration, since non-detection and erroneous detection are typical forms of detection failure, it is possible to effectively reduce the labeling process for the second image Db.

[0083] Other Embodiments As described above, the above embodiment has been described as an example of the technology disclosed in this application. However, the technology of the present disclosure is not limited to this and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiment can be combined to create new embodiments. Furthermore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately determining that these non-essential components are essential.

[0084] For example, although the color of the mark M is changed in accordance with the attribute of the object W, the shape of the mark M may be changed instead of the color.

[0085] Furthermore, the object W for which detection has failed may be limited to undetected objects W, or may be limited to erroneously detected objects W.

[0086] Furthermore, the predetermined information of the object W may be either position information or attribute information of the object W. Furthermore, the attribute information of the object W is not limited to the color of the object W, but may also be the shape or size of the object W.

[0087] Furthermore, the control device 1 (i.e., the teacher data generation device) of the above embodiment is configured to have not only the function of generating teacher data, but also a learning function by the learning unit 144 and a detection function by the detection unit 145. However, the technology disclosed herein is not limited to this, and at least one of the learning unit 144 and the detection unit 145 may be provided independently. In other words, the teacher data generation device may be a device that omits both the learning unit 144 and the detection unit 145, or a device that omits the learning unit 144 or the detection unit 145.

[0088] The control device 1 may also be a device that omits the first generation process among the processes for generating training data, i.e., a device that performs only the second generation process. In other words, the control device 1 may be a device that purchases a trained detection model from an external source and installs it.

[0089] Furthermore, the predetermined process performed by the robot arm 42 on the object W may be painting, welding, screwing, or the like on the object W, in addition to gripping the object W.

[0090] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor. [Explanation of symbols]

[0091] 1. Control device (teaching data generation device) 12 Storage section 13 Display section 42 Robot Arm 141 Display control unit 142 Label assignment unit 143 Generation part Da 1st image Db 2nd image PG generation program (teaching data generation program) W object M mark

Claims

1. A training data generation device for generating training data for retraining a trained detection model that detects an object included in an image and predetermined information about the object, comprising: a label assignment unit that assigns a label indicating the predetermined information to an object included in an image in response to an input operation by a user; a generation unit that generates a pair of the image and the label assigned to the image as training data; a storage unit that stores an image in which detection using the detection model has been performed and in which the detection has failed for at least one object, in association with the detection result using the detection model; a display unit that displays the image in the storage unit, the labeling unit assigns the label indicating the predetermined information to an object for which detection has failed in the image stored in the storage unit in response to an input operation by a user, while substituting the detection result for the label indicating the predetermined information of an object for which detection has succeeded; the generation unit generates, as training data, a set of the image in the storage unit, the label attached to the object for which detection has failed, and the detection result of the object for which detection has succeeded; The label assignment unit is a teacher data generation device that assigns the label by adding a mark indicating the specified information to the object that has failed to be detected and is included in the image of the memory unit displayed on the display unit through a user input operation.

2. 2. The teacher data generation device according to claim 1, a display control unit that causes the image in the storage unit to be displayed on the display unit; The display control unit is a teacher data generation device that, when displaying the image from the memory unit on the display unit, attaches the mark indicating the detection result to the successfully detected object included in the image from the memory unit.

3. 3. The teacher data generation device according to claim 1, the predetermined information is position information of an object in the image in the storage unit, The label assignment unit is a teacher data generation device that assigns the label to an object using the position of the mark assigned to the object by a user's input operation as position information of the object.

4. 4. The teacher data generation device according to claim 3, A teacher data generation device in which the position of the mark is set as a position where a robot arm performs a predetermined processing on the object based on the position of the mark.

5. 5. The teacher data generation device according to claim 4, A teacher data generation device in which the position of the mark is set as the position where the robot arm grasps the object.

6. 6. The teacher data generation device according to claim 4, A teacher data generation device capable of communicating with a robot control device that controls the robot arm.

7. 3. The teacher data generation device according to claim 1, the predetermined information is attribute information of an object, The label assignment unit is a teacher data generation device that assigns the label using the color of the mark as attribute information of the object when the mark, which is of a color distinguished according to the attribute information of the object, is assigned to the object by a user's input operation.

8. 8. The teacher data generation device according to claim 1, A training data generation device in which the object that has failed to be detected is an undetected object or an erroneously detected object.

9. A training data generation program that causes a computer to realize a function of generating training data for retraining a trained detection model that detects an object included in an image and predetermined information about the object, a function of assigning a label indicating the predetermined information to an object included in an image in response to an input operation by a user; a function of generating a pair of the image and the label assigned to the image as training data; a function of storing an image in which detection using the detection model has been performed and in which the detection has failed for at least one object, in association with the detection result using the detection model; a function of assigning the label indicating the predetermined information to an object for which detection has failed in the stored image in response to an input operation by a user, while substituting the detection result for the label indicating the predetermined information of an object for which detection has succeeded; a function of generating a set of the stored image, the label attached to the object for which detection has failed, and the detection result of the object for which detection has succeeded as training data; A training data generation program that causes a computer to realize a function of displaying the stored image and assigning the label to the object that failed to be detected in the image by adding a mark indicating the specified information to the object through a user input operation.

Images