Learning device, learning method, and program

Abstract

To perform more efficient learning without confusing detection processing of an object and a combination with recognition processing of a relationship.SOLUTION: In a learning device according to an embodiment, an object combination query creation unit creates an object combination query from data in which noise is added to at least one of an object class, region information, and a combination of a plurality of objects for teacher data. The relationship query creation unit creates a relationship query from data in which noise is added to at least the relationship class for the teacher data. The update unit updates the weight of the detection model such that the object class, the region information, and the combination of the plurality of objects of the training data before the noise is applied are reconfigured based on the output of the detection unit and the training data, and updates the weight of the recognition model such that the relationship class of the training data before the noise is applied is reconfigured based on the output of the recognition unit and the training data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a learning device, a learning method, and a program. [Background technology]

[0002] A technology called relationship detection, which recognizes objects and relationships between objects in an image, has been known for some time. For example, in an image containing a person riding a horse, the objects "person" and "horse" and their combination are detected, and the relationship of "riding" is recognized for the detected combination. In other words, relationship detection detects and recognizes the series of relationships, including the object, of "person" "riding" a "horse." One example of a method for relationship detection is to detect objects and their combinations, and then recognize the relationship between the objects included in the detected combinations using the features of those objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-103740 [Patent Document 2] Japanese Patent Publication No. 2024-003483 [Non-patent literature]

[0004] [Non-Patent Document 1] Focusing on what to decode and what to train: Efficient Training with HOI Split Decoders and Specific Target Guided Denoising, [Retrieved June 5, 2024], Internet〈URL:https: / / arxiv.org / abs / 2307.02291〉 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional techniques, it has been difficult to perform more efficient learning without confusing the process of detecting objects and combinations with the process of recognizing relationships. [Means for solving the problem]

[0006] A learning device according to an embodiment includes an image input unit, a calculation unit, a training data input unit, an object combination query creation unit, a detection unit, a relationship query creation unit, a recognition unit, and an update unit. The image input unit accepts an input of an image. The calculation unit calculates, from the image, feature quantities indicating characteristics of the image. The training data input unit accepts input of training data that teaches correct answers for object classes indicating objects included in the image, area information identifying the objects, combinations of multiple objects, and relationship classes indicating relationships between the combinations. The object combination query creation unit creates an object combination query used to detect multiple objects and the combinations from data in which noise is added to at least one of the object classes, the area information, and the multiple object combinations. The detection unit detects the multiple objects and the combinations from the image based on the feature quantities and the object combination query by a detection process using a detection model. The relationship query creation unit creates a relationship query used to recognize the relationship from data in which noise is added to at least the relationship classes. The recognition unit recognizes the relationship between the plurality of objects included in the image by recognition processing using a recognition model based on the feature amount and the relationship query. The update unit removes the noise from the object combination query created from the data to which the noise has been added based on the output of the detection unit and the training data, and updates weights of the detection model so that the object classes, the region information, and combinations of the plurality of objects in the training data before the noise has been added are reconstructed, and removes the noise from the relationship query created from the data to which the noise has been added based on the output of the recognition unit and the training data, and updates weights of the recognition model so that the relationship classes in the training data before the noise has been added are reconstructed. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram showing an example of the functional configuration of the learning device according to the first embodiment. [Figure 2] FIG. 4 is a diagram showing an example of processing by an update unit according to the first embodiment. [Figure 3] FIG. 4 is a diagram showing an example of a method for adding noise to a class of training data according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a method for adding noise to rectangles in training data according to the first embodiment. [Figure 5] 4 is a flowchart showing an example of a learning method according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of the functional configuration of a learning device according to a second embodiment. [Figure 7A] FIG. 10 is a diagram showing an example of processing by an update unit according to the second embodiment. [Figure 7B] FIG. 10 is a diagram showing an example of processing by an update unit according to the second embodiment. [Figure 8] FIG. 10 is a diagram showing an example of the functional configuration of a learning device according to a third embodiment. [Figure 9A] FIG. 11 is a diagram showing an example of processing by an update unit according to the third embodiment. [Figure 9B] FIG. 11 is a diagram showing an example of processing by an update unit according to the third embodiment. [Figure 10] FIG. 10 is a diagram showing an example of the functional configuration of a learning device according to a fourth embodiment. [Figure 11] 10 is a flowchart showing an example of a learning method according to the fourth embodiment. [Figure 12] FIG. 13 is a diagram showing an example of the functional configuration of a learning device according to a fifth embodiment. [Figure 13] 13 is a flowchart showing an example of a learning method according to the fifth embodiment. [Figure 14] 13 is a flowchart showing an example of a learning method in a modified example of the fifth embodiment. [Figure 15] FIG. 1 is a diagram showing an example of the hardware configuration of a learning device according to the first to fifth embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of a learning device, a learning method, and a program will be described in detail with reference to the accompanying drawings.

[0009] (First embodiment) First, an example of the functional configuration of the learning device of the first embodiment will be described.

[0010] [Example of functional configuration] FIG. 1 is a diagram showing an example of the functional configuration of a learning device 100 according to the first embodiment. The learning device 100 according to the first embodiment detects objects and combinations based on input images and recognizes relationships between those combinations. The learning device 100 may be provided in a device that outputs images, or may be provided in a device separate from the device that outputs images.

[0011] As shown in FIG. 1, the learning device 100 of the first embodiment includes an image input unit 101, a calculation unit 102, a training data input unit 103, an object combination query creation unit 104, a detection unit 105, a relationship query creation unit 106, a recognition unit 107, an update unit 108, and a memory unit 200.

[0012] The image input unit 101 receives an image output from an imaging device such as an RGB camera. Alternatively, the image input unit 101 may be the device itself that outputs an image, such as an RGB camera. In the example of FIG. 1, the image input unit 101 reads an image from the storage unit 200 and inputs the image to the calculation unit 102. The image input unit 101 may have a function to perform preprocessing on the image. For example, the preprocessing includes changing the size of the image, rotating the image, and standardizing the image.

[0013] The calculation unit 102 calculates feature amounts based on the image input by the image input unit 101. The feature amount calculation method may be, for example, a method of calculating the feature amounts of an image by performing convolution calculations, such as with a CNN (Convolutional Neural Network). Alternatively, for example, the feature amount calculation method may be a method of dividing an image into patch units, calculating self-attention, and extracting the feature amounts of the image, such as with a ViT (Vision Transformer).

[0014] The teacher data input unit 103 receives teacher data that teaches correct answers about objects and combinations for the image input by the image input unit 101 and the relationships between those combinations. Specifically, the teacher data input unit 103 receives input of teacher data that teaches correct answers about object classes that indicate objects included in the image, area information that identifies the objects, combinations of multiple objects, and relationship classes that indicate the relationships between the combinations.

[0015] The training data may be training data that teaches each combination and relationship, such as object A-object B-relationship, for example. Furthermore, for example, the training data may be training data that teaches a sentence including an object and a relationship, and a rectangle that identifies the object in the sentence. For example, a sentence including an object and a relationship is, "A person is riding a bicycle."

[0016] 1, the training data input unit 103 inputs training data read from the storage unit 200 to the object combination query creation unit 104, the relationship query creation unit 106, and the update unit 108. The training data input unit 103 may have a function of performing preprocessing on the training data. For example, the preprocessing includes normalizing the rectangles of the objects in the training data and converting the format of the training data.

[0017] The object combination query creation unit 104 creates an object combination query to be used for detecting objects and combinations based on the annotation data including noise of at least one of the object rectangles and classes of the training data input by the training data input unit 103.

[0018] For example, when detecting object A, object B, and a combination of objects A and B, the object combination query creation unit 104 converts the class and rectangle of object A to which noise has been added based on the added data, and the class and rectangle of object B to which noise has been added based on the added data, into feature amounts. Then, the object combination query creation unit 104 creates an object combination query by concatenating the feature amounts of object A and the feature amounts of object B. Details of the noise addition method and the query creation method will be described later.

[0019] The detection unit 105 detects objects and combinations based on the object combination query created by the object combination query creation unit 104.

[0020] For example, the detection unit 105 receives an object combination query created from data in which noise has been added to training data by the object combination query creation unit 104, and outputs a result so as to reconstruct the objects and combinations of the training data. The detection unit 105 removes noise from the input data in which noise has been added to training data, and the update unit 108 performs learning (weight update) so that the objects and combinations of the original training data are reconstructed.

[0021] The relationship query creation unit 106 creates a relationship query to be used for relationship recognition based on data in which noise has been added to at least the relationship class of the training data input by the training data input unit 103. For example, when a relationship is taught for a combination of object A and object B, the relationship query creation unit 106 converts the class and rectangle of object A, the class and rectangle of object B, and the relationship class into feature quantities, and creates a relationship query by linking these feature quantities. Details of the method of adding noise and the method of creating a query will be described later.

[0022] The recognition unit 107 recognizes relationships based on the relationship query created by the relationship query creation unit 106. For example, the recognition unit 107 receives a relationship query created from data in which noise has been added to the training data of the relationship query creation unit 106, and outputs a result so as to reconstruct the relationships of the training data. The recognition unit 107 removes noise from the input data in which noise has been added to the training data, and the update unit 108 performs learning (weight update) so that the relationships of the original training data are reconstructed.

[0023] The update unit 108 updates the weights based on the teacher data from the teacher data input unit 103 and the outputs of the detection unit 105 and the recognition unit 107. For example, the update unit 108 updates the weights so that the detection results of the objects and combinations in the detection unit 105 reconstruct the objects and combinations in the teacher data, and updates the weights so that the results of the relationship recognition in the recognition unit 107 reconstruct the relationships in the teacher data.

[0024] That is, the update unit 108 removes noise from the input data to which noise has been added to the training data, and performs training on the detection unit 105 and the recognition unit 107 so that the original training data is reconstructed. This accelerates the convergence of training and achieves highly accurate detection.

[0025] A learning dataset is stored in the storage unit 200. The image input unit 101 reads images for learning from the storage unit 200. The image input unit 101 reads training data for learning from the storage unit 200. For example, images and training data of a batch size obtained by dividing the images and training data included in the learning dataset into batches are input sequentially to the image input unit 101 and the training data input unit 103.

[0026] [Example of weight update method] 2 is a diagram showing an example of processing by the update unit 108 in the first embodiment. The update unit 108 updates the weights based on the training data input from the training data input unit 103 and the outputs from the detection unit 105 and the recognition unit 107. An example of the weight update processing for the recognition result of relationship recognition will be described with reference to FIG. 2.

[0027] 2 is a table showing the training data for one image and the recognition results of the recognition unit 107. In the example of FIG. 2, the training data for one image includes two objects and combinations, and instructions (1) and (2) indicating their relationships, which instruct the objects (1) and (2) and their relationships. From this training data, it can be confirmed that person A, person B, and a horse are present in the image, and that person A is riding the horse and person B is pulling the same horse.

[0028] For each of objects (1) and (2), the x1y1x2y2 in parentheses represents rectangle information (an example of area information) that indicates the position and size of the object. For example, x1y1 represents the coordinates of the top left vertex of the rectangle, and x2y2 represents the coordinates of the bottom right vertex of the rectangle.

[0029] A relationship query is created from data in which noise is added to each instruction data. The recognition unit 107 outputs a relationship recognition result in accordance with each relationship query. When updating the weights, the recognition unit 107 learns by updating the weights so as to increase the predicted probability of a relationship that matches the instruction that was the basis for creating the relationship query and the recognition result.

[0030] The detection unit 105 also performs a weight update process similar to that of the recognition unit 107. In the case of the detection unit 105, the weights are updated so that the predicted probability of the object class and the value of the rectangle match, thereby training the detection unit 105.

[0031] The learning by the update unit 108 shown in Figure 2 can be called learning to reconstruct teacher data, as it inputs a query created from data in which noise has been added to teacher data, removes the noise from the input query, and learns so that the teacher data and the output match.

[0032] [Example of how to add noise] Next, an example of a noise addition method in the object combination query creation unit 104 and the relationship query creation unit 106 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a diagram showing an example of a noise addition method for classes of training data in the first embodiment. Fig. 3 shows a table in which object classes and relationship classes are extracted for training data for one image, and a table showing the results of adding noise to the classes. In the example of Fig. 3, noise is added to object (2) and relationship classes. Noise is added by, for example, replacing them with random classes selected by random numbers.

[0033] Fig. 4 is a diagram showing an example of a method for adding noise to rectangles in training data according to the first embodiment. Fig. 4 shows object rectangles for one piece of training data and the result of adding noise to the rectangles. In the example of Fig. 4, noise is added to the rectangle of a horse, which corresponds to object (2) in Fig. 3, and the noise is added by changing the position and size of the rectangle.

[0034] The object combination query creation unit 104 creates an object combination query from data in which noise is added to at least object rectangles or classes. The relationship query creation unit 106 creates a relationship query from data in which noise is added to at least relationship classes.

[0035] [Example of how to create a query] Next, an example of a query created by the object combination query creation unit 104 and the relationship query creation unit 106 and a method for creating the query will be described. The object combination query creation unit 104 and the relationship query creation unit 106 convert the training data to which noise has been added into features as queries. The training data includes classes of objects and classes of relationships between rectangles, and by converting some or all of this information into features, it becomes possible to learn how to reconstruct the training data.

[0036] Therefore, for example, object and relationship classes are converted into feature quantities using the embedding representation of the classes. Furthermore, for example, object rectangles are converted into feature quantities using a fully connected layer and a feed-forward network (FFN). By performing calculations such as concatenation or averaging of these feature quantities, feature quantities containing information on the object classes and rectangles and some or all of the relationship classes are created.

[0037] For example, the detection unit 105 detects objects and combinations and learns to reconstruct teacher data for the objects and combinations, so that the classes of the objects included in the combinations and the features of the connected rectangles are input to the detection unit 105.

[0038] Furthermore, for example, the recognition unit 107 recognizes the relationships between combinations of objects and learns to reconstruct teacher data about the relationships, so that features linking the classes and rectangles of the objects included in the combinations with the classes of the relationships are input to the recognition unit 107.

[0039] [Example of learning method] 5 is a flowchart showing an example of the learning method according to the first embodiment. First, the image input unit 101 accepts an input of an image (step S1). Next, the calculation unit 102 calculates a feature quantity indicating the characteristics of the image from the image input in step S1 (step S2).

[0040] Next, the teacher data input unit 103 accepts input of teacher data that teaches correct answers regarding object classes that indicate objects contained in the image, area information (e.g., rectangles) that identify the objects, combinations of multiple objects, and relationship classes that indicate the relationships between the combinations (step S3).

[0041] Next, the object combination query creation unit 104 and the relationship query creation unit 106 add noise to the training data (step S4) and create a query from the training data with the noise added (step S5). Specifically, the object combination query creation unit 104 creates an object combination query used to detect multiple objects and combinations from data with noise added to at least one of the object class, area information, and combination of multiple objects. The relationship query creation unit 106 creates a relationship query used to recognize a relationship from data with noise added to at least the relationship class.

[0042] Next, the detection unit 105 and the recognition unit 107 detect object combinations and recognize relationships (step S6). Specifically, the detection unit 105 detects multiple objects and combinations from the image input in step S1 by a detection process using a detection model, based on the feature amounts calculated in step S2 and the object combination query created in step S5. Furthermore, the recognition unit 107 recognizes the relationships between multiple objects included in the image input in step S1 by a recognition process using a recognition model, based on the feature amounts calculated in step S2 and the relationship query created in step S5.

[0043] Next, based on the output of the detection unit 105 and the training data, the update unit 108 removes noise from the object combination query created from the data to which noise has been added, and updates the weights of the detection model so that the object classes, area information, and combinations of multiple objects of the training data before the noise was added are reconstructed, and based on the output of the recognition unit 107 and the training data, the update unit 108 removes noise from the relationship query created from the data to which noise has been added, and updates the weights of the recognition model so that the relationship classes of the training data before the noise was added are reconstructed (step S7).

[0044] As described above, the learning device 100 of the first embodiment enables more efficient learning without confusing the object and combination detection process with the relationship recognition process. Specifically, in the first embodiment, an object combination query is created from data in which noise is added to the object rectangles or object classes of the training data, and a relationship query is created from data in which noise is added to at least the relationship class. Then, weights are updated so that the outputs of the detection unit 105 and the recognition unit 107 match the training data, and reconstruction learning is performed in the detection unit 105 and the recognition unit 107, respectively. By performing learning in which the detection unit 105 and the recognition unit 107 focus on their respective tasks (detection processing and recognition processing), the convergence of learning can be accelerated and each task can be performed with higher accuracy.

[0045] In the past, when learning reconstruction, objects and combinations were detected in the first stage, and the features of those combinations were input to the relationship recognition in the second stage to recognize the relationships, which posed a problem in that the tasks of detecting objects and combinations and the relationship recognition could be confused. This confusion of tasks could result in the relationship reconstruction being performed simultaneously when detecting objects and combinations in the first stage, making it impossible to learn the relationship reconstruction in the relationship recognition in the second stage.

[0046] On the other hand, in the first embodiment described above, queries are created by the object combination query creation unit 104 and the relationship query creation unit 106, and the detection unit 105 and the recognition unit 107 can learn to reconstruct training data, respectively. This allows learning to be focused on each task, accelerates convergence, and enables learning with high accuracy.

[0047] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described. In the second embodiment, a configuration will be described in which incorrect answer teaching data for incorrect answers to the detection of objects and combinations and the recognition of relationships is added, and learning is performed to suppress the detection and recognition of the incorrect answer teaching data.

[0048] [Example of functional configuration] Fig. 6 is a diagram showing an example of the functional configuration of a learning device 100-2 according to the second embodiment. As shown in Fig. 6, the learning device 100-2 according to the second embodiment includes an image input unit 101, a calculation unit 102, a teacher data input unit 103, an object combination query creation unit 104, a detection unit 105, a relationship query creation unit 106, a recognition unit 107, an update unit 108, an incorrect answer teaching data input unit 109, and a storage unit 200.

[0049] In the second embodiment, an incorrect answer teaching data input unit 109 is added to the configuration of the first embodiment in Fig. 1. The incorrect answer teaching data is data that teaches incorrect answers regarding object classes, area information, combinations of multiple objects, and relationship classes. The incorrect answer teaching data is used to detect inappropriate objects and combinations and to suppress recognition of inappropriate relationships.

[0050] In the second embodiment, learning for suppressing detection and recognition using incorrect teaching data is added, and therefore the processing of the object combination query creation unit 104, the relationship query creation unit 106, and the update unit 108 differs from that of the first embodiment. The components other than those described above are basically the same as those of the first embodiment in FIG. 1 that processes teaching data.

[0051] The processing of the object combination query creation unit 104 and the relationship query creation unit 106 in the second embodiment will be described. In the first embodiment, the object combination query creation unit 104 and the relationship query creation unit 106 create queries from data in which noise is added to the training data in order to learn how to reconstruct the training data. In the second embodiment of FIG. 6, the method of creating queries from the training data and incorrect teaching data is basically the same as in the first embodiment.

[0052] Note that noise does not need to be added to the incorrect solution teaching data, but adding noise may increase the variety of the incorrect solution teaching data. Specifically, the object combination query creation unit 104 may add noise to at least one of the object class, area information, and combination of multiple objects included in the incorrect solution teaching data, and create an object combination query based on the incorrect solution teaching data to which noise has been added. Furthermore, the relationship query creation unit 106 may add noise to at least the relationship class included in the incorrect solution teaching data, and create a relationship query based on the incorrect solution teaching data to which noise has been added.

[0053] [Example of weight update method] 7A and 7B are diagrams showing an example of processing by the update unit 108 in the second embodiment. In Figs. 7A and 7B, processing for updating the weight for the recognition result of the recognition unit 107 will be described.

[0054] 7A is a table showing training data for one image and the recognition results of the relationships. As in the first embodiment, the weights of the training data are updated so that the predicted probability of classes with matching relationships between the instruction that was the basis for creating the relationship query and the recognition results using that relationship query becomes higher.

[0055] 7B is a table showing incorrect teaching data for one image and the relationship recognition results. For the incorrect teaching data, the weights are updated so that the predicted probability of a class with a matching relationship between the teaching that was the basis for creating the relationship query and the recognition result using that relationship query becomes lower. As a result, the weights are updated so as not to recognize relationships regarding the incorrect teaching data, and learning can be performed to suppress the recognition of incorrect relationships.

[0056] Similarly, in the case of the detection unit 105, learning is performed so as to update the weights for the incorrect teaching data and the detection results of the objects and combinations so that the predicted probability of the incorrect object class becomes lower.

[0057] As described above, in the second embodiment, the incorrect answer data input unit 109 accepts input of incorrect answer data that teaches incorrect answers for the object class, area information, combination of multiple objects, and relationship class. The object combination query creation unit 104 creates an object combination query based on the incorrect answer data. The relationship query creation unit 106 creates a relationship query based on the incorrect answer data. Then, the update unit 108 updates the weights of the detection model so that the prediction probability of predicting an incorrect answer is lowered from the output of the detection unit 105 in response to the object combination query based on the incorrect answer data, and updates the weights of the recognition model so that the prediction probability of predicting an incorrect answer is lowered from the output of the recognition unit 107 in response to the relationship query based on the incorrect answer data.

[0058] According to the second embodiment, by using incorrect answer teaching data to update the weights so that the prediction probability of outputting an incorrect answer becomes lower, learning can be performed to suppress detection and recognition of incorrect answers.

[0059] (Third embodiment) Next, a third embodiment will be described. In the description of the third embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described. In the third embodiment, first incorrect answer teaching data for incorrect answers that suppress the detection of objects and combinations, and second incorrect answer teaching data for incorrect answers that suppress the recognition of relationships are added. Then, learning is performed to suppress the detection and recognition of the first and second incorrect answer teaching data. The second incorrect answer teaching data that suppresses the recognition of relationships is, for example, incorrect answer data in which the objects and combinations are correct but only the relationships are incorrect.

[0060] [Example of functional configuration] 8 is a diagram illustrating an example of the functional configuration of a learning device 100-3 according to the third embodiment. As shown in FIG. 8, the learning device 100-3 according to the third embodiment includes an image input unit 101, a calculation unit 102, a teacher data input unit 103, an object combination query creation unit 104, a detection unit 105, a relationship query creation unit 106, a recognition unit 107, an update unit 108, first and second incorrect teaching data input units 110 and 111, and a storage unit 200.

[0061] In the third embodiment, first and second incorrect answer teaching data input units 110 and 111 are added to the configuration of the first embodiment in Fig. 1. The first incorrect answer teaching data is data in which at least one incorrect answer is taught regarding an object class, area information, and a combination of a plurality of objects. The second incorrect answer teaching data is data in which an incorrect answer is taught regarding a relationship.

[0062] In the third embodiment, learning for suppressing detection and recognition using the first and second incorrect teaching data is added, and therefore the processing of the object combination query creation unit 104, the relationship query creation unit 106, and the update unit 108 differs from that of the first embodiment. The components other than those described above are basically the same as those of the first embodiment in FIG. 1 that processes teaching data.

[0063] The processing of the object combination query creation unit 104 and the relationship query creation unit 106 of the third embodiment will be described. In the first embodiment, the object combination query creation unit 104 and the relationship query creation unit 106 create queries from data in which noise is added to the training data in order to learn the reconstruction of the training data. In the third embodiment of FIG. 8, the method of creating queries from the training data and the first and second incorrect teaching data is basically the same.

[0064] Note that noise may not be added to the first and second incorrect answer teaching data, but adding noise may increase the variety of the first and second incorrect answer teaching data. Specifically, the object combination query creation unit 104 may add noise to at least one of the object class, area information, and combination of multiple objects included in the first incorrect answer teaching data, and create an object combination query based on the first incorrect answer teaching data to which noise has been added. Furthermore, the relationship query creation unit 106 may add noise to at least the relationship class included in the second incorrect answer teaching data, and create a relationship query based on the second incorrect answer teaching data to which noise has been added.

[0065] By adding noise to the first and second incorrect teaching data, the variation in rectangles, classes, etc. can be increased, thereby making it possible to further enhance the effect of suppressing the detection of objects and combinations, as well as the recognition of relationships, for the first and second incorrect teaching data.

[0066] [Example of weight update method] 9A and 9B are diagrams showing an example of processing by the update unit 108 in the third embodiment. In Figs. 9A and 9B, processing for updating the weight for the recognition result of the recognition unit 107 will be described.

[0067] 9A is a table showing training data for one image and the recognition results of the relationships. As in the first embodiment, the weights of the training data are updated so that the predicted probability of classes with matching relationships between the instruction that was the basis for creating the relationship query and the recognition results using that relationship query becomes higher.

[0068] 9B is a table showing the second incorrect answer data for one image and the relationship recognition result. An example of the process of the update unit 108 using the second incorrect answer data will be described with reference to FIG.

[0069] The second incorrect answer teaching data in FIG. 9B is taught the same person A and horse as teaching (1) in the training data in FIG. 9A, and the objects and combinations are correct. The teaching data is for the relationship "pull," which is different from teaching (1) in the training data in FIG. 9A, and the relationship is incorrect. For the second incorrect answer teaching data, the weights are updated so that the predicted probability of classes with matching relationships between the teachings that were used to create the relationship query and the recognition results using that relationship query becomes lower.

[0070] In this way, for example, if there is data in which the object and combination are correct but only the relationship is incorrect, this is input as second incorrect teaching data to the second incorrect teaching data input unit 111, thereby learning only about the error in relationship recognition and learning to suppress that error.

[0071] Similarly, in the case of the detection unit 105, learning is performed so as to update the weights for the first incorrect teaching data and the detection results of the objects and combinations so that the predicted probability of the incorrect object class becomes lower.

[0072] As described above, in the third embodiment, the first incorrect answer data input unit 110 accepts input of first incorrect answer data that teaches at least one incorrect answer regarding an object class, region information, and a combination of multiple objects. The second incorrect answer data input unit 111 accepts input of second incorrect answer data that teaches an incorrect answer regarding a relationship class. The object combination query creation unit 104 creates an object combination query based on the first incorrect answer data. The relationship query creation unit 106 creates a relationship query based on the second incorrect answer data. Then, the update unit 108 updates the weights of the detection model so that the prediction probability of predicting an incorrect answer is lower among the outputs of the detection unit 105 corresponding to the object combination query based on the first incorrect answer data, and updates the weights of the recognition model so that the prediction probability of predicting an incorrect answer is lower among the outputs of the recognition unit 107 corresponding to the relationship query based on the second incorrect answer data.

[0073] According to the third embodiment, learning can be performed by further modifying the learning content based on the first and second incorrect answer teaching data described above. For example, if there is data in which the object and combination are correct but only the relationship is incorrect, this data is input as second incorrect answer teaching data to the second incorrect answer teaching data input unit 111, whereby only the relationship recognition error can be learned and the error can be suppressed.

[0074] Furthermore, according to the third embodiment, queries can be created for the first and second incorrect teaching data, and the queries can be learned by the detection unit 105 and the recognition unit 107, respectively. This allows learning to be performed in which, in the case of data in which an object is incorrectly detected, the detection unit 105 suppresses detection of the target object and combination, and in the case of data in which an object is correctly detected but the relationship is incorrectly recognized, the recognition unit 107 suppresses relationship recognition.

[0075] (Fourth embodiment) Next, a fourth embodiment will be described. In the description of the fourth embodiment, the same description as in the first embodiment will be omitted, and only differences from the first embodiment will be described. In the fourth embodiment, a user interface is added that allows the user to check and change the learning data to which noise has been added.

[0076] [Example of functional configuration] 10 is a diagram illustrating an example of the functional configuration of a learning device 100-4 according to the fourth embodiment. As illustrated in FIG. 10, the learning device 100-4 according to the fourth embodiment includes a learning unit 121, a display control unit 122, and an input control unit 123.

[0077] The learning unit 121 of the fourth embodiment has the functions of the image input unit 101, calculation unit 102, teacher data input unit 103, object combination query creation unit 104, detection unit 105, relationship query creation unit 106, recognition unit 107, update unit 108 and memory unit 200 of the first embodiment.

[0078] Display control unit 122 displays display information including teacher data, incorrect answer teaching data, and learning data with noise added on a display device such as a monitor. The learning data includes the teacher data and incorrect answer teaching data described above. For example, the user can refer to the display information to check whether data different from the expected data has been input to learning device 100-4.

[0079] The input control unit 123 receives operation inputs for setting or changing parameters of noise added to training data via input devices such as a keyboard and a mouse. For example, if a user determines that training data to which noise has been added is inappropriate training data, the user can set or change the parameters of the noise via the input control unit 123.

[0080] The display control unit 122 and the input control unit 123 may be one of the functions of the learning unit 121, or may be one of the functions of the object combination query creation unit 104 and the relationship query creation unit 106.

[0081] [Example of learning method] 11 is a flowchart showing an example of a learning method according to the fourth embodiment. First, the learning unit 121 acquires learning data including the above-mentioned training data (step S11), and creates data by adding noise to the training data (step S12).

[0082] Next, the display control unit 122 displays the data to which noise has been added in step S12 on the display device (step S13), and allows the user to determine whether unexpected data has been created by adding noise.

[0083] Here, the display control unit 122 may display the teacher data together with the data to be displayed to the user, and if incorrect teaching data exists, may also display the incorrect teaching data together. Furthermore, unexpected data is, for example, data in which noise is added to the rectangle of an object, resulting in a rectangle with a position or size that is significantly different from that of the object in the teaching data. If the added noise is too large, the difficulty of learning may increase significantly, and the learning may not converge.

[0084] Next, if the input control unit 123 receives an operation to change the parameters for setting noise (step S14, Yes), the learning unit 121 changes the settings of the noise to be added to the training data in accordance with the change operation from the user (step S15).

[0085] If the operation to change the parameters for setting noise has not been received (No in step S14), the process proceeds to step S 16. The processes in steps S16 to S18 are the same as the processes in steps S5 to S7 in the first embodiment, and therefore will not be described.

[0086] As described above, according to the fourth embodiment, the user can change the parameters for generating noise while checking the data to which noise has been added. This allows the user to more easily adjust the noise parameters, thereby preventing unexpected data from being learned.

[0087] (Fifth embodiment) Next, a fifth embodiment will be described. In the description of the fifth embodiment, the same description as in the fourth embodiment will be omitted, and only differences from the fourth embodiment will be described. In the fifth embodiment, a configuration will be described in which an instruction is given to an inference result that includes an error in an object, an object combination, or a relationship from an inference result obtained using a detection model and a recognition model, and the inference result can be registered as incorrect solution instruction data.

[0088] [Example of functional configuration] Fig. 12 is a diagram illustrating an example of the functional configuration of a learning device 100-5 according to the fifth embodiment. As shown in Fig. 12, the learning device 100-5 according to the fifth embodiment includes a learning unit 121, a display control unit 122, an input control unit 123, an inference unit 124, a learning dataset storage unit 200-1, and a parameter storage unit 200-2.

[0089] The learning unit 121 of the fifth embodiment has the functions of, for example, the image input unit 101, calculation unit 102, training data input unit 103, object combination query creation unit 104, relationship query creation unit 106, and update unit 108 of the first embodiment.

[0090] The inference unit 124 of the fifth embodiment has the functions of, for example, the image input unit 101, the calculation unit 102, the detection unit 105, and the recognition unit 107 of the first embodiment. The training data set storage unit 200-1 stores a training data set. The parameter storage unit 200-2 stores parameters used by the learning unit 121 and the inference unit 124.

[0091] 12 is shown as an example for convenience of explanation, and the functional configuration may be modified as appropriate. For example, the learning unit 121, the inference unit 124, the training data set storage unit 200-1, and the parameter storage unit 200-2 may be implemented as a single processing unit.

[0092] [Example of learning method] 13 is a flowchart showing an example of a learning method in the fifth embodiment. First, the inference unit 124 reads out parameters of the above-mentioned detection model and recognition model from the parameter storage unit 200-2, and executes inference processing based on the parameters on the image to be processed (step S31).

[0093] Next, the display control unit 122 displays the inference result obtained by the inference process of step S31 on the display device (step S32). By the process of step S32, the user can check whether or not there is an incorrect answer in the inference result in the detection of objects and combinations or in the recognition of relationships.

[0094] If an incorrect inference result exists (Yes at step S33), the input control unit 123 receives an input for creating incorrect teaching data based on the incorrect inference result, and the incorrect teaching data is created (step S34).

[0095] The incorrect answer teaching data created in step S34 is registered (stored) in the learning dataset of the learning dataset storage unit 200-1 together with the image used in the inference process in step S31 (step S35). Note that if the same image has already been registered in the learning dataset, only the incorrect answer teaching data may be registered.

[0096] Next, the learning unit 121 performs learning using the learning data set in which the incorrect teaching data is registered, thereby updating the parameters in the parameter storage unit 200-2 (step S36).

[0097] Note that learning and parameter updating are not essential, and the process may end after step S35. Also, if there is no incorrect inference result (step S33, No), the process ends immediately.

[0098] As described above, according to the fifth embodiment, for an inference result that includes an error in an object, a combination of objects, or a relationship, incorrect answer teaching data indicating that the inference result is incorrect can be registered. This allows the incorrect answer teaching data for the incorrect inference result to be collected, and learning that suppresses detection and recognition errors can be performed using a learning data set including the incorrect answer teaching data.

[0099] In the fifth embodiment, for example, a user can have the learning device 100-5 collect and learn incorrect teaching data while checking inference results at an actual site, etc., and can accurately adapt to the on-site data. Conventionally, there has been no learning method that focuses on incorrect inference results during inference at an actual site, etc., and it has not been possible to suppress incorrect answers during on-site operation, etc.

[0100] (Modification of the fifth embodiment) Next, a modified example of the fifth embodiment will be described. In the description of the modified example, the same explanation as in the fifth embodiment will be omitted, and only differences from the fifth embodiment will be described. In the modified example, an example of a learning method different from the flowchart of FIG. 13 of the fifth embodiment will be described.

[0101] [Example of learning method] Fig. 14 is a flowchart showing an example of a learning method in a modified example of the fifth embodiment. Explanation of steps S41 and S42 is omitted here, as they are the same as steps S31 and S32 in Fig. 13. By the processing of step S42, the user can check whether or not there are any incorrect answers in the inference results, in the detection of objects and combinations, or in the recognition of relationships.

[0102] In a modified example, the incorrect answer teaching data is created separately as first incorrect answer teaching data that teaches incorrect answers regarding the detection of objects and combinations, and second incorrect answer teaching data that teaches incorrect answers regarding the recognition of relationships.

[0103] If an incorrect inference result exists for the detection of objects and combinations (step S43, Yes), the input control unit 123 receives input to create first incorrect teaching data based on the incorrect inference result for the detection of objects and combinations, and the first incorrect teaching data is created (step S44).

[0104] The first incorrect answer teaching data created in step S44 is registered (stored) in the learning dataset of the learning dataset storage unit 200-1 together with the image used in the inference process in step S41 (step S45). Note that if the same image has already been registered in the learning dataset, only the first incorrect answer teaching data may be registered.

[0105] If there is no incorrect inference result for the detection of objects and combinations (No at step S43), or after the processing at step S45 is executed, the processing proceeds to the processing at step S46.

[0106] If an incorrect inference result exists regarding the relationship recognition (step S46, Yes), when the input control unit 123 accepts input to create second incorrect teaching data based on the incorrect inference result regarding the relationship recognition, the second incorrect teaching data is created (step S47).

[0107] The second incorrect answer teaching data created in step S47 is registered (stored) in the learning dataset of the learning dataset storage unit 200-1 together with the image used in the inference process in step S41 (step S48). Note that if the same image has already been registered in the learning dataset, only the second incorrect answer teaching data may be registered.

[0108] If there is no incorrect inference result for relationship recognition (No at step S46), or after the process at step S48 is executed, the process proceeds to step S49.

[0109] The learning unit 121 determines whether the learning data set has been updated, that is, whether the first or second incorrect solution teaching data has been registered (step S49). If the learning data set has not been updated (step S49, No), the process ends.

[0110] If the learning data set has been updated (step S49, Yes), the learning unit 121 updates the parameters in the parameter memory unit 200-2 by performing learning using the learning data set in which at least one of the first and second incorrect answer teaching data is registered (step S50).

[0111] Note that learning and parameter updating are not essential, and the process may end after step S46 or S48.

[0112] In this way, according to the modified example, it is possible to collect and learn first incorrect teaching data that teaches that at least one of the object or combination is incorrect from the inference results, and to collect and learn second incorrect teaching data that teaches that the relationship is incorrect.

[0113] Compared to the fifth embodiment described above, for example, when only the relationship recognition result is incorrect and the object detection result is correct, learning can be performed to suppress only the relationship recognition as second incorrect teaching data.

[0114] Finally, an example of the hardware configuration of the learning devices 100 to 100-5 according to the first to fifth embodiments will be described.

[0115] [Example of hardware configuration] 15 is a diagram showing an example of the hardware configuration of the learning devices 100 to 100-5 of the first to fifth embodiments. The learning devices 100 to 100-5 of the first to fifth embodiments include a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0116] Note that learning devices 100 to 100-5 may not be equipped with some of the above configurations. For example, if learning devices 100 to 100-5 can use the input function and display function of an external device, learning devices 100 to 100-5 may not be equipped with display device 204 and input device 205.

[0117] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a ROM and a RAM. The auxiliary storage device 203 is a HDD, a memory card, or the like.

[0118] Display device 204 is, for example, a liquid crystal display. Input device 205 is an interface for operating learning devices 100-100-5. Display device 204 and input device 205 may be realized by a touch panel or the like having display and input functions. Communication device 206 is an interface for communicating with other devices.

[0119] For example, the programs executed by the learning devices 100 to 100-5 are provided as computer program products in the form of installable or executable files recorded on computer-readable storage media such as memory cards, hard disks, CD-RWs, CD-ROMs, CD-Rs, DVD-RAMs, and DVD-Rs.

[0120] Furthermore, for example, the programs executed by the learning devices 100 to 100-5 may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network.

[0121] Furthermore, for example, the programs executed by the learning devices 100-100-5 may be provided via a network such as the Internet without being downloaded. Specifically, the learning process may be performed by a so-called ASP (Application Service Provider) type service that realizes processing functions by issuing execution instructions and obtaining results from a server computer without transferring the programs.

[0122] Furthermore, for example, the programs for the learning devices 100 to 100-5 may be provided in a state where they are pre-installed in a ROM or the like.

[0123] The programs executed by the learning devices 100-100-5 have a modular configuration that includes functions that can be realized by the programs among the functional configurations described above. As for the actual hardware, the processor 201 reads the programs from a storage medium and executes them, and the above-mentioned functional blocks are loaded onto the main storage device 202. In other words, the above-mentioned functional blocks are generated on the main storage device 202.

[0124] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) rather than by software.

[0125] Furthermore, each function may be realized using a plurality of processors 201, and in this case, each processor 201 may realize one of the functions, or may realize two or more of the functions.

[0126] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0127] 100 Learning Device 101 Image input unit 102 Calculation section 103 Teacher data input section 104 Object combination query creation unit 105 Detector 106 Relationship Query Creation Unit 107 Recognition part 108 Update Department 109 Incorrect answer data input section 110 First incorrect answer teaching data input unit 111 Second incorrect answer instruction data input section 121 Learning Department 122 Display control unit 123 Input control section 124 Reasoning part 200 Storage section 200-1 Learning dataset storage unit 200-2 Parameter storage section 201 processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input Device 206 Communication Equipment 210 Bus

Claims

1. an image input unit that accepts input of an image; a calculation unit that calculates, from the image, a feature amount that indicates a feature of the image; a training data input unit that receives input of training data that teaches correct answers regarding object classes that indicate objects included in the image, area information that identifies the objects, combinations of a plurality of the objects, and relationship classes that indicate relationships between the combinations; an object combination query creation unit that creates an object combination query to be used for detecting the plurality of objects and the combinations from data in which noise is added to at least one of the object class, the region information, and the plurality of object combinations; a detection unit that detects the plurality of objects and the combinations from the image based on the feature amounts and the object combination query by a detection process using a detection model; a relationship query creation unit that creates a relationship query to be used for recognizing the relationship from at least data in which noise is added to the relationship class; a recognition unit that recognizes a relationship between the plurality of objects included in the image based on the feature amount and the relationship query by performing a recognition process using a recognition model; an update unit that removes the noise from the object combination query created from the data to which the noise has been added, based on the output of the detection unit and the training data, and updates weights of the detection model so that the object class, the region information, and the combination of the plurality of objects in the training data before the noise has been added are reconstructed, and that removes the noise from the relationship query created from the data to which the noise has been added, based on the output of the recognition unit and the training data, and updates weights of the recognition model so that the relationship class of the training data before the noise has been added is reconstructed; A learning device comprising:

2. an incorrect answer data input unit that receives input of incorrect answer teaching data that teaches an incorrect answer regarding the object class, the area information, the combination of the plurality of objects, and the relationship class; the object combination query creation unit creates an object combination query based on the incorrect teaching data, the relational query creation unit creates a relational query based on the incorrect answer teaching data, the update unit updates weights of the detection model so as to lower a prediction probability of predicting an incorrect answer from among outputs of the detection unit in response to an object combination query based on the incorrect answer teaching data, and updates weights of the recognition model so as to lower a prediction probability of predicting an incorrect answer from among outputs of the recognition unit in response to a relationship query based on the incorrect answer teaching data. The learning device according to claim 1 .

3. the object combination query creation unit adds noise to at least one of the object class, the area information, and the plurality of object combinations included in the incorrect solution teaching data, and creates an object combination query based on the incorrect solution teaching data to which the noise has been added; the relational query creation unit adds noise to at least the relational class included in the incorrect solution teaching data, and creates a relational query based on the incorrect solution teaching data to which the noise has been added; The learning device according to claim 2 .

4. a first incorrect answer data input unit that receives input of first incorrect answer data in which at least one incorrect answer is taught regarding the object class, the area information, and a combination of the plurality of objects; a second incorrect answer data input unit that receives input of second incorrect answer data in which an incorrect answer about the relational class is taught; the object combination query creation unit creates an object combination query based on the first incorrect teaching data, the relational query creation unit creates a relational query based on the second incorrect answer teaching data; the update unit updates weights of the detection model so as to lower a prediction probability of predicting an incorrect answer from an output of the detection unit in response to an object combination query based on the first incorrect answer teaching data, and updates weights of the recognition model so as to lower a prediction probability of predicting an incorrect answer from an output of the recognition unit in response to a relationship query based on the second incorrect answer teaching data. The learning device according to claim 1 .

5. the object combination query creation unit adds noise to at least one of the object class, the region information, and the plurality of object combinations included in the first incorrect answer teaching data, and creates an object combination query based on the first incorrect answer teaching data to which the noise has been added; the relational query creation unit adds noise to at least the relational class included in the second incorrect answer teaching data, and creates a relational query based on the second incorrect answer teaching data to which the noise has been added; The learning device according to claim 4 .

6. a display control unit that displays data in which noise has been added to at least one of the object class, the area information, the combination of the plurality of objects, and the relationship class on a display device; an input control unit that receives input of parameters for changing the noise settings; The learning device according to claim 1 , comprising:

7. the display control unit further displays the plurality of objects and the combinations detected by the detection unit, as well as an inference result including a relationship between the plurality of objects recognized by the recognition unit; The input control unit further receives an input for creating incorrect solution teaching data based on the inference result. The learning device according to claim 6.

8. the display control unit further displays the plurality of objects and the combinations detected by the detection unit, as well as an inference result including a relationship between the plurality of objects recognized by the recognition unit; The input control unit further receives at least one of an input for creating first incorrect answer teaching data based on the plurality of objects included in the inference result and the combination, and an input for creating second incorrect answer teaching data based on a relationship between the plurality of objects included in the inference result. The learning device according to claim 6.

9. A step in which a learning device receives an input of an image; a step of the learning device calculating, from the image, a feature amount indicating a feature of the image; a step in which the learning device receives input of training data that teaches correct answers regarding object classes that indicate objects included in the image, area information that identifies the objects, combinations of a plurality of the objects, and relationship classes that indicate relationships between the combinations; The learning device creates an object combination query to be used for detecting the plurality of objects and the combinations from data in which noise is added to at least one of the object class, the region information, and the plurality of object combinations; detecting, by the learning device, the plurality of objects and the combinations from the image based on the feature amounts and the object combination query by a detection process using a detection model; The learning device creates a relationship query to be used for recognizing the relationship from at least data in which noise is added to the relationship class; a step in which the learning device recognizes a relationship between the plurality of objects included in the image based on the feature amount and the relationship query by a recognition process using a recognition model; the learning device removes the noise from the object combination query created from the data to which the noise has been added, based on an output of the detecting step and the training data, and updates weights of the detection model so that the object class, the region information, and the combination of the plurality of objects in the training data before the noise has been added are reconstructed; and removes the noise from the relationship query created from the data to which the noise has been added, based on an output of the recognizing step and the training data, and updates weights of the recognition model so that the relationship class in the training data before the noise has been added is reconstructed; Learning methods including.

10. Computer, an image input unit that accepts input of an image; a calculation unit that calculates, from the image, a feature amount that indicates a feature of the image; a training data input unit that receives input of training data that teaches correct answers regarding object classes that indicate objects included in the image, area information that identifies the objects, combinations of a plurality of the objects, and relationship classes that indicate relationships between the combinations; an object combination query creation unit that creates an object combination query to be used for detecting the plurality of objects and the combinations from data in which noise is added to at least one of the object class, the region information, and the plurality of object combinations; a detection unit that detects the plurality of objects and the combinations from the image based on the feature amounts and the object combination query by a detection process using a detection model; a relationship query creation unit that creates a relationship query to be used for recognizing the relationship from at least data in which noise is added to the relationship class; a recognition unit that recognizes a relationship between the plurality of objects included in the image based on the feature amount and the relationship query by performing a recognition process using a recognition model; an update unit that removes the noise from the object combination query created from the data to which the noise has been added, based on the output of the detection unit and the training data, and updates weights of the detection model so that the object class, the region information, and the combination of the plurality of objects in the training data before the noise has been added are reconstructed, and that removes the noise from the relationship query created from the data to which the noise has been added, based on the output of the recognition unit and the training data, and updates weights of the recognition model so that the relationship class of the training data before the noise has been added is reconstructed; A program to function as a

Images