Recognition device, recognition method, and program

Abstract

Provided is a recognition device comprising: a recognition model training unit that trains a recognition model so that the recognition model receives, as input data, an image consisting of an object and a background and outputs a set of data including the type, position, and orientation of the object; a recognition unit that uses the recognition model to obtain, from the image consisting of the object and the background, a set of recognition result data, which is the result of recognizing at least the type, position, and orientation of the object; a generation unit that uses a generation model to generate, from the image consisting of the object and the background, a pair of another set of data that includes the type, position, and orientation of the object and another image that includes the object and the background as arranged on the basis of the other set of data; and a generation model training unit that trains the generation model on the basis of the value of a loss function that indicates the deviation between another set of recognition result data, which is the result of recognizing the other image using the recognition unit, and the other set of data.

Description

Recognition device, recognition method, and program

[0001] The present invention relates to a recognition device, a recognition method, and a program.

[0002] When performing object recognition using an AI (Artificial Intelligence) model, it is necessary to secure a large amount of annotation data (training data based on bounding boxes) for learning. However, creating annotation data using real data requires a great deal of effort and results in increased costs.

[0003] In particular, at work sites (factories, warehouses, construction sites, etc.), it is often difficult to obtain or create large amounts of training data, making the above problem more pronounced.

[0004] To address this issue, there is an artificial data generation technique that generates artificial training data by combining background data and data of the object to be detected (images, 3D models, etc.), such as a Generative Adversarial Network (GAN).

[0005] Patent Literature 1 discloses an image generation method that can synthesize a background image with an image of an object to be detected and generate training data for learning a neural network, etc. Specifically, it is possible to obtain an image of the object as an image to be detected from an image (target image) consisting of the object and background, obtain information indicating the position, shape, size, etc. of the object from image metadata, and use a generative model to generate another image in which an object with a different position, shape, size, etc. is placed on the background image.

[0006] By comparing the generated image with the target image and once the similarity between the two reaches a predetermined standard, it is possible to further fine-tune the parameters of the generative model so that the conditions for the currently generated image can be applied to the generation of other images. This makes it possible to generate multiple images that serve as training data from a single target image.

[0007] JP 2019-159630 A

[0008] The disclosures of the above prior art documents are incorporated herein by reference. The following analysis has been carried out by the present inventors.

[0009] In addition to the above, data generation AI technologies have also been developed that optimize the distribution of artificial training data generated by a data generation network using a desired objective function, such as the loss function of a recognition model. For example, there are data generation techniques that train a generative model so as to maximize the loss function of the recognition model during training.

[0010] While the data generation AI described above generates data useful for learning, the learning tends to be unstable, and the generated data may be biased. For example, in object recognition, if the objective function is designed to learn to maximize the loss function, images may be generated in which objects are placed in impossible positions, or images may be generated with combinations that are more difficult to recognize than necessary.

[0011] In this regard, learning methods such as curriculum learning have been proposed, which allow for the generation of highly accurate models by controlling the distribution of the training data, for example by first learning the easy parts of the training data and gradually increasing the difficulty of the training.

[0012] However, while curriculum learning can produce a certain degree of effectiveness, the appropriate curriculum differs for each task (domain), and therefore the curriculum must be set somewhat by trial and error for each task, which poses the problem of difficulty in stably controlling the distribution of training data for generating highly accurate models.

[0013] In light of the above-mentioned problems, an object of the present invention is to provide a recognition device that uses data generation AI technology for generating recognition models for object recognition, which is capable of generating data useful for learning in an unbiased, stable, and uniform manner, and that uses this data to train a recognition model, thereby achieving outstanding recognition accuracy.

[0014] According to a first aspect of the present invention or disclosure, there is provided a recognition device having: a recognition model training unit that uses an image consisting of an object and a background as input data and trains a recognition model so that data including the type, position, and orientation of the object becomes output data; a recognition unit that uses the recognition model to obtain recognition result data that is a result of recognizing at least the type, position, and orientation of the object from the image consisting of the object and the background; a generation unit that uses a generative model to generate, from the image consisting of the object and the background, a set of another data including the type, position, and orientation of the object and another image that includes the object and the background arranged based on the other data; and a generative model training unit that trains the generative model based on the other recognition result data that is a result of recognizing the other image by the recognition unit and the value of a loss function that indicates deviation from the other data.

[0015] According to a second aspect of the present invention or disclosure, there is provided a recognition method for causing a computer to execute the following steps, the recognition method comprising: a step of training a recognition model using an image consisting of an object and a background as input data, so that data including the type, position, and orientation of the object becomes output data; a step of using the recognition model to obtain recognition result data that is a result of recognizing at least the type, position, and orientation of the object from the image consisting of the object and the background; a step of generating, using a generative model, a set of another data including the type, position, and orientation of the object from the image consisting of the object and the background, and another image that includes the object and the background arranged based on the another data; and a step of training the generative model based on the another recognition result data that is a result of recognizing the another image and the value of a loss function that indicates deviation from the another data.

[0016] According to a third aspect of the present invention or disclosure, there is provided a program for causing a computer to execute the following processes: a process of training a recognition model using an image consisting of an object and a background as input data, so that data including the type, position, and orientation of the object becomes output data; a process of using the recognition model to obtain recognition result data that is a result of recognizing at least the type, position, and orientation of the object from the image consisting of the object and the background; a process of generating, using a generative model, a set of another data including the type, position, and orientation of the object from the image consisting of the object and the background, and another image that includes the object and background arranged based on the other data; and a process of training the generative model based on the other recognition result data that is a result of recognizing the other image and the value of a loss function that indicates the deviation from the other data.

[0017] According to each aspect of the present invention and disclosure, the present invention provides a recognition device, a recognition method, and a program that can generate data useful for learning in an unbiased, stable, and uniform manner in a data generation AI technology for generating a recognition model for object recognition, and that can contribute to achieving outstanding recognition accuracy by using this data to train a recognition model.

[0018] FIG. 1 is a block diagram showing an example of the configuration of a recognition device according to an embodiment. FIG. 2 is a schematic diagram showing an overview of the processing of the recognition device according to the first embodiment. FIG. 3 is a block diagram showing an example of the configuration of the recognition device according to the first embodiment. FIG. 4 is a schematic diagram showing a series of operations of the recognition device according to the first embodiment. FIG. 5 is a schematic diagram showing a series of operations of the recognition device according to the first embodiment. FIG. 6 is a flowchart for explaining an example of the operation of the recognition device according to the first embodiment. FIG. 7 is a block diagram showing an example of the hardware configuration of the recognition device according to the first embodiment. FIG. 8 is a schematic diagram showing the configuration of a generation unit of the recognition device according to the second embodiment. FIG. 9 is a schematic diagram showing the configuration of a generation unit of the recognition device according to the second embodiment.

[0019] [Overview of Processing of One Embodiment] First, an overview of processing of one embodiment will be described. Note that the drawing reference symbols attached to this overview are attached to each element for convenience as an example to facilitate understanding, and the description of this overview is not intended to be limiting in any way. Furthermore, connection lines between blocks in each figure include both bidirectional and unidirectional. Unidirectional arrows are used to schematically indicate the flow of the main signal (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc. shown in this disclosure, input ports and output ports exist at the input and output ends of each connection line. The same applies to input / output interfaces.

[0020] [Configuration of an embodiment] Next, the configuration of a recognition device according to an embodiment will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of a recognition device according to an embodiment. As shown in this diagram, a recognition device 10 according to an embodiment includes a recognition model training unit 11, a recognition unit 12, a generation unit 13, and a generation model training unit 14.

[0021] The recognition model training unit 11 receives an image consisting of an object and a background as input data and trains a recognition model so that data including the type, position, and orientation of the object becomes output data. The recognition unit 12 uses the recognition model to obtain recognition result data that is a result of recognizing at least the type, position, and orientation of the object from the image consisting of the object and the background. The generation unit 13 uses a generative model to generate, from the image consisting of the object and the background, a set of other data including the type, position, and orientation of the object and another image including the object and the background arranged based on the other data. The generative model training unit 14 trains a generative model based on the value of a loss function that indicates the deviation between the other recognition result data, which is a result of recognition of the other image by the recognition unit, and the other data.

[0022] According to one embodiment of the recognition device 10, a recognition model is first trained using image data consisting of an object and a background and its annotation data. That is, an image consisting of an object and a background is input as input data, and the recognition model is trained to output at least the type, position, and orientation of the object. Next, using the image data consisting of the object and a background as input, a generative model outputs data on another image consisting of the object and a background, as well as data on the type, position, and orientation of another object placed in the other image, which serves as its annotation data. The recognition model recognizes the other image and obtains recognition result data. The value of the loss function is calculated using this recognition result data and the data on the type, position, and orientation of the other object, which serves as annotation data.

[0023] In this way, the recognition device of one embodiment generates training data using a generative model that generates training data, and even if there is a small amount of training data with given annotations, it is possible to improve recognition accuracy by training the recognition model using the generated training data. Furthermore, by calculating the value of a loss function using recognition result data, it is possible to control the learning of the generative model based on the value of the loss function and generate training data with annotations that are useful for learning the recognition model.

[0024] Specific embodiments will be described in more detail below with reference to the drawings. Note that the same components in each embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0025] [First Embodiment] Figure 2 is a schematic diagram showing an overview of the processing of a recognition device according to a first embodiment. As shown in this diagram, the recognition device 10 of this embodiment comprises a database that stores background / object data, an artificial data generation AI, and a recognition AI. The artificial data generation AI generates artificial data in which an object is placed in the background from the background / object data. The recognition AI recognizes the artificial data and obtains an inference result. The inference result is fed back to the recognition AI and the artificial data generation AI based on the value of a loss function.

[0026] [Configuration of the First Embodiment] Next, the configuration of the recognition device 10 according to this embodiment will be described with reference to the drawings. Fig. 3 is a block diagram showing an example of the configuration of the recognition device 10 according to this embodiment. As shown in this diagram, the recognition device 10 according to the first embodiment has a recognition model training unit 11, a recognition unit 12, a generation unit 13, a generative model training unit 14, an initial training data holding unit 15, and a reference value holding unit 16.

[0027] The recognition model training unit 11 uses an image consisting of an object and a background as input data and trains a recognition model so that data including the object's type, position, and orientation is output data. An "object" is something that is the target of recognition in object recognition. For example, in an autonomous driving system, this includes vehicles, people, signs, and traffic lights traveling on a road. A "background" provides a space in which an object is placed, and an image of an object placed in the background is the target of recognition for the recognition model. The "type" of an object is the category to which the object to be recognized belongs, and examples include "automobile," "person," and "artificial object" such as signs and traffic lights. The "position and orientation of an object" are coordinates or position vectors that indicate the location of an object within a background, such as a position vector relative to the center of gravity of a bounding box representing the object, and a certain range within space specified by data such as the height, width, depth, and rotation angle of the bounding box.

[0028] A "recognition model" refers to a set of operations for converting predetermined input data to obtain output data. In this embodiment, "recognition" means detecting an object contained in input image data and obtaining the type, position, and orientation to identify the object as output data.

[0029] The "recognition model" may be realized by a neural network model consisting of activation functions and weighting parameters that represent the strength of the connections between the activation functions and the background. In this embodiment, "learning" the recognition model means adjusting the parameters so that, for a specific input image consisting of an object and a background, data indicating the type, position, and orientation of a specific object, as indicated by annotation data, is output.

[0030] The training may be performed by a gradient method using a loss function, which indicates the sum of errors between the output of the recognition model for the input image and the desired output, the annotation data, as an objective function, i.e., by calculating a combination of parameters that minimizes the loss function.

[0031] The recognition unit 12 acquires recognition result data, which is a result of recognizing at least the type, position, and orientation of an object from an image consisting of the object and a background using a recognition model. Specifically, the recognition unit 12 predicts the type, position, and orientation of a specific object using a bounding box for data acquired from a device such as an optical camera or LiDAR (Light Detection and Ranging).

[0032] The generation unit 13 generates, from an image consisting of an object and a background, a set of other data including the type, position, and orientation of the object, and another image including the object and background arranged based on the other data, using a generative model. In other words, it is possible to artificially generate a set of an image of a background and an object, and annotation data of the object, which can be used for training a recognition model.

[0033] The generative model training unit 14 trains the generative model based on the value of a loss function that indicates the deviation between another recognition result data, which is the result of the recognition unit 12 recognizing another image generated by the generation unit 13, and another (annotation) data generated by the generation unit 13. The training may be performed in a direction that maximizes or minimizes the value of the loss function; however, training in a direction that maximizes the value of the loss function increases the generation of images with impossible object arrangements or arrangements that are difficult to recognize, which are the problems described above. Therefore, in the recognition device 10 of this embodiment, training is preferably performed in a direction that minimizes the value of the loss function.

[0034] The initial training data storage unit 15 stores initial training data consisting of a given image of an object and a background, and data including the object's type, position, and orientation. The "given image" is not an image generated by the generation unit 13, but initial training data based on actual measurements, etc. Initially, the recognition AI and generation AI are trained using this initial training data. That is, the recognition model training unit 11 trains a recognition model using the initial training data, the recognition unit 12 recognizes the image of the initial training data using the recognition model and obtains recognition result data, and the generation model training unit 14 trains a generation model in a direction that minimizes the value of a loss function that indicates the deviation between the recognition result data and data including the object's type, position, and orientation included in the initial training data.

[0035] The reference value storage unit 16 stores a reference value of the loss function. The “reference value” is a target value of the loss function, and is a value that controls the difficulty of recognition by the recognition unit 12 of the artificial training data (other data) generated by the generation unit 13.

[0036] When this reference value is set, the recognition unit 12 of the recognition device 10 obtains the value of a loss function that indicates the deviation between the recognition result data, which is the result of recognizing the image generated by the generative model, and data including the type, position, and orientation of the object in the image generated by the generative model, and the generative model training unit 14 trains the generative model so that the value of the loss function approaches the reference value.

[0037] The reference value may be a fixed value or may be determined based on the value of the loss function. For example, if the value of the loss function is x, the reference value S(x) may be a constant multiple of the most recent value of the loss function, such as S(x) = 1.05 × x.

[0038] [Explanation of Operation] Figures 4(a) to 4(c) are schematic diagrams showing a series of operations of the recognition device 10 of this embodiment. Referring to Figure 4(a), first, 1. Train the recognition AI (recognition model) using limited initial training data (R0). The recognition AI is trained until the loss function L is minimized. Next, using the trained recognition AI (R0), train the generation AI (generative model) so that the value of the recognition AI's loss function is minimized (G0). Referring to Figure 4(b), next, 3. Record the value of the loss function of R0 in G0 (L(R0)). Next, 4. Set a reference value (S(L(R0))) for the loss value of the generation AI based on L(R0). Referring to Figure 4(c), next, 5. Train the generation AI so that the loss approaches the reference value (G1). Finally, 6. Train the recognition AI using data generated by G1 (R1).

[0039] In the recognition device 10 of this embodiment, it is possible to train a recognition AI with higher recognition accuracy by repeating steps 3 to 6. For example, it is possible to repeat step 3 until the value of the loss function reaches a predetermined error range.

[0040] 5 is a flowchart illustrating an example of the operation of the recognition device 10 of this embodiment. As shown in this figure, initial training data is first acquired (step S101). Next, a recognition model is trained using the initial training data (step S102). Next, a generative model is trained in a direction that minimizes the loss function of the recognition model (step S103).

[0041] Next, the reference value of the stored loss function is acquired (step S104). After that, the value of the loss function is acquired (step S105). Specifically, the value of the loss function is acquired, which indicates the discrepancy between the recognition result data relating to the result of recognizing, using the recognition model, an image consisting of an object generated by the generative model and a background, and the data including the type, position, and orientation of the object generated by the generative model.

[0042] Next, it is determined whether the acquired value of the loss function is within a predetermined error range with respect to the reference value. If the value of the loss function is within the predetermined error range with respect to the reference value (step S106, Y), the process ends. If it is not within the predetermined error range (step S106, N), the generative model is trained so that the value of the loss function approaches the reference value (step S107), and the recognition model is trained using the data generated by the generative model (step S108). After that, the process returns to acquiring the reference value of the loss function (step S104).

[0043] [Hardware Configuration] The recognition device 10 of this embodiment can be executed by an information processing device (computer) and has the configuration exemplified in Fig. 6. The recognition device 10 includes a CPU (Central Processing Unit) 61, a memory 62, an input / output interface 63, and a NIC (Network Interface Card) 64 as a communication means, which are interconnected by an internal bus 65.

[0044] However, the configuration shown in Fig. 6 is not intended to limit the hardware configuration of the recognition device. The recognition device 10 may include hardware not shown, and may not include the input / output interface 63 as necessary. Furthermore, the number of CPUs and other components included in these devices is not intended to be limited to the example shown in Fig. 6. For example, the recognition device 10 may include multiple CPUs.

[0045] The memory 62 is a RAM (Random Access Memory), a ROM (Read Only Memory), or an auxiliary storage device (such as a hard disk).

[0046] The input / output interface 63 is a means for interfacing with a display device and an input device (not shown). The display device is, for example, a liquid crystal display. The input device is, for example, a camera or a sensor that receives an image consisting of an object and a background, and a device that receives user operations such as a keyboard or a mouse.

[0047] The functions of the recognition device 10 are realized by a group of programs (processing modules) stored in the memory 62, such as a recognition model learning program, a generative model learning program, a recognition program, a generation program, and a reference value determination program, as well as a group of data used by each program, such as initial training data, reference value data, and other parameters. The processing modules are realized, for example, by the CPU 61 executing each program stored in the memory 62. The programs can be updated by downloading them via a network or using a storage medium storing the programs. Furthermore, the processing modules may be realized by semiconductor chips. That is, it is sufficient to have some means for executing the functions performed by the processing modules using hardware and / or software.

[0048] [Hardware Operation] In the recognition device 10, the recognition model learning program is called from memory 62 and executed by the CPU 61. The program reads initial training data stored in memory 62 and executes recognition model learning. Next, the recognition program is called from memory 62 and executed by the CPU 61. The program reads initial training data and outputs recognition result data. Next, the generative model learning unit is called from memory 62 and executed by the CPU 61. The program uses a loss function that calculates the distance between a vector indicating the initial training data (annotations) and a vector indicating the recognition result data, and executes learning in the direction that minimizes the loss function. For example, parameters that make the derivative of the loss function zero are calculated and stored in memory 62 together with the value of the loss function at that time.

[0049] Next, a reference value determination program is called from memory 62 and put into execution by CPU 61. This program calls the reference value stored in memory 62, performs a comparison operation with the calculated loss function value, and determines whether it is within the error range of the reference value. If it is within the error range of the reference value, the processing ends. If it is not within the error range of the reference value, the generative model training program is called from memory 62 and put into execution by CPU 61. This program trains the generative model so that the value of the loss function approaches the reference value.

[0050] Next, the generation program is called from memory 62 and executed by CPU 61. This program receives an image consisting of an object and a background as input data, and outputs training data consisting of an image consisting of another object and background, and its annotation data (at least the type, position, and orientation of the object). Next, the recognition model learning program is again executed by CPU 61, and the recognition model is learned using the training data generated by the generation program as input.

[0051] When learning is complete, the reference value of the loss function is obtained again, the value of the loss function is acquired, and a reference value judgment program is used to judge whether the value of the loss function is within a predetermined error range with respect to the reference value, and the learning and generation processes are repeated until it is within the error range.

[0052] [Explanation of Effects] According to the recognition device 10 of this embodiment, by generating data for learning and training a recognition model using the data, it is possible to proceed with learning while generating training data, even if initial training data is difficult to obtain and the amount of data is small. In this case, it is possible to proceed with learning so that the value of the loss function approaches a predetermined reference value. Rather than simply maximizing or minimizing the value of the loss function, it is possible to generate data at a level useful for learning using a uniform method. By training the recognition model in this way, it is possible to provide a recognition device that achieves excellent recognition accuracy.

[0053] Second Embodiment In this embodiment, a description will be given particularly of the configuration of the generation unit 13 of the recognition device 10. Specifically, the recognition device 10 is a generation unit having a network that generates the type, position, and orientation of an object, and a network that generates an image made up of the object and a background, wherein the network that generates the type, position, and orientation of an object receives an image made up of the object and the background as input and outputs the position and orientation of the object generated by random variables sampled based on a predetermined distribution, and the network that generates the image made up of the object and the background generates an image in which the object is placed in the background based on the position and orientation of the object.

[0054] 7 and 8 are schematic diagrams showing the configuration of the generation unit 13 of the recognition device 10 according to the second embodiment. Fig. 7 shows a network that generates the type, position, and orientation of an object from the data generated by the generation unit 13. Fig. 8 shows a network that generates an image in which an object is placed in the background.

[0055] In Figure 7, first, a bird's eye view image is input from a device such as a camera. The input data is encoded by an encoder to obtain latent variable values. These values ​​are further decoded to generate a location map. The location map indicates the distribution of object location candidates. Based on these location maps, object coordinates are generated by sampling.

[0056] Sampling of the object's position is performed by generating random numbers (noise) with a predetermined distribution. Each coordinate is provided as a random variable, and a random term is added to each sample to form a distribution. Sampling is also performed on the orientation (rotation).

[0057] In Figure 8, the object (Object Data) is converted and positioned based on the type, position, and orientation vectors of the object generated in Figure 7. Specifically, the original Object Data is converted by multiplying the position vector and orientation vector. Finally, it is combined with the background. The generated image data may be an image in which a bounding box is placed in the background, as shown in the figure.

[0058] 7, the generative network can generate vectors representing the type, position, and orientation of the objects, as well as an occlusion mask that represents the overlap of multiple objects. This vector data is reflected in the image data by being multiplied by the object data in FIG. 8. That is, the generator 13 can calculate the area that disappears due to the overlap of multiple objects, and generate an image in which the object is placed in a background by projecting that area onto the image of the object.

[0059] [Explanation of Effects] In the recognition device of this embodiment, background data is used as input information and combined with object data generated using a 3D model or the like, the positions and orientations of the objects to be combined are generated, and specific learning data can be generated based on the generated positions and orientations.

[0060] Some or all of the above-described embodiments can also be described as in the following supplementary notes. However, the following supplementary notes are merely examples of the present invention, and the present invention is not limited to such cases. [Supplementary Note 1] As in the recognition device according to the first aspect described above. [Supplementary Note 2] Preferably, the recognition device of Supplementary Note 1 further includes an initial training data storage unit that stores initial training data consisting of a set of given images consisting of an object and a background, and data including the type, position, and orientation of the object, wherein the recognition model training unit trains a recognition model using the initial training data, the recognition unit recognizes the image of the initial training data using the recognition model to obtain recognition result data, and the generative model training unit trains a generative model in a direction that minimizes the value of a loss function that indicates the deviation between the recognition result data and the data including the type, position, and orientation of the object included in the initial training data. [Supplementary Note 3] The recognition device preferably according to Supplementary Note 1 or 2, further comprising a reference value holding unit that holds a reference value for the value of the loss function, wherein the recognition unit acquires the value of the loss function that indicates the deviation between recognition result data that is the result of recognizing an image generated by the generative model and data including the type, position, and orientation of an object in the image generated by the generative model, and the generative model training unit trains the generative model so that the value of the loss function approaches the reference value. [Supplementary Note 4] The reference value is determined based on the value of the loss function, preferably according to the recognition device of Supplementary Note 3. [Supplementary Note 5] The recognition model training unit further uses an image including an object and a background generated by the generative model as input data, and trains the recognition model so that data including the type, position, and orientation of the object related to the image generated by the generative model becomes output data, preferably according to the recognition device of Supplementary Note 3 or 4. [Supplementary Note 6] The generation unit has a network that generates the type, position, and orientation of an object, and a network that generates an image consisting of an object and a background, the network that generates the type, position, and orientation of an object takes the image consisting of an object and a background as input and outputs the position and orientation of the object generated by random variables sampled based on a predetermined distribution, and the network that generates the image consisting of an object and a background generates an image in which the object is placed in the background based on the position and orientation of the object, preferably a recognition device as described in any of Supplementary Notes 1 to 5.[Supplementary Note 7] The generation unit calculates an area that disappears due to overlap of multiple objects, and generates an image in which the object, obtained by projecting the area onto an image of the object, is placed in a background, preferably the recognition device of Supplementary Note 6. [Supplementary Note 8] As with the recognition device according to the second aspect described above. [Supplementary Note 9] The recognition method preferably of Supplementary Note 8 further includes the steps of acquiring initial training data consisting of a set of a given image consisting of an object and a background, and data including the type, position, and orientation of the object, training a recognition model using the initial training data, recognizing the image of the initial training data using the recognition model to obtain recognition result data, and training a generative model in a direction that minimizes the value of a loss function that indicates the discrepancy between the recognition result data and the data including the type, position, and orientation of the object included in the initial training data. [Supplementary Note 10] The recognition method of Supplementary Note 9 preferably further comprises: (a) acquiring a reference value for the loss function; (b) acquiring a value of the loss function indicating a discrepancy between recognition result data relating to the result of recognizing an image consisting of an object and a background generated by a generative model and data including the type, position, and orientation of the object generated by the generative model; (c) training the generative model so that the value of the loss function approaches the reference value; and (d) training the recognition model so that an image including the object and background generated by the generative model is used as input data and data including the type, position, and orientation of the object related to the image generated by the generative model is output data. [Supplementary Note 11] The recognition method of Supplementary Note 10 preferably repeats steps (a) to (d) until the value of the loss function reaches a predetermined error range with respect to the reference value. [Supplementary Note 12] As in the program according to the third aspect described above.

[0061] The disclosures of the above-cited patent documents and other documents are incorporated herein by reference. Modifications and adjustments of the embodiments are possible within the scope of the entire disclosure of the present invention (including the claims), and further based on the basic technical concepts thereof. Furthermore, various combinations and selections (including partial deletions) of various disclosed elements (including elements of each claim, each embodiment, each element of each embodiment, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concepts, including the claims. In particular, with regard to the numerical ranges described herein, any numerical value or subrange included within the range should be construed as being specifically described, even if not otherwise specified.

[0062] 10: Recognition device 11: Recognition model learning unit 12: Recognition unit 13: Generation unit 14: Generation model learning unit 15: Initial training data storage unit 16: Reference value storage unit 61: CPU 62: Memory 63: Input / output interface 64: NIC 65: Internal bus

Claims

1. a recognition model learning unit that uses an image consisting of an object and a background as input data and learns a recognition model so that data including the type, position, and orientation of the object becomes output data; a recognition unit that acquires recognition result data that is a result of recognizing at least the type, position, and orientation of the object from an image consisting of the object and a background using the recognition model; a generation unit that generates, from an image consisting of an object and a background, a set of other data including the type, position, and orientation of the object, and another image including the object and background arranged based on the other data, using a generative model; a generative model learning unit that learns the generative model based on another recognition result data, which is a result of recognizing the other image by the recognition unit, and a value of a loss function that indicates a deviation between the other data and the other recognition result data; A recognition device having:

2. an initial training data storage unit that stores initial training data consisting of a set of a given image consisting of an object and a background, and data including the type, position, and orientation of the object; and the recognition model learning unit learns the recognition model using the initial training data; the recognition unit recognizes the images in the initial training data using the recognition model to obtain the recognition result data; the generative model learning unit learns the generative model in a direction to minimize a value of a loss function indicating a deviation between the recognition result data and data including the type, position, and orientation of the object included in the initial training data. The recognition device according to claim 1 .

3. a reference value storage unit that stores a reference value of the loss function; and the recognition unit acquires a value of a loss function indicating a deviation between the recognition result data, which is a result of recognizing an image generated by the generative model, and data including a type, position, and orientation of an object in the image generated by the generative model; the generative model training unit trains the generative model so that the value of the loss function approaches the reference value; The recognition device according to claim 1 .

4. The reference value is determined based on the value of the loss function. The recognition device according to claim 3 .

5. the recognition model learning unit further uses an image including an object and a background generated by the generative model as input data, and learns the recognition model so that data including the type, position, and orientation of the object related to the image generated by the generative model becomes output data. The recognition device according to claim 3 .

6. the generation unit includes a network that generates a type, position, and orientation of an object, and a network that generates an image consisting of the object and a background; The network for generating the object type, position, and orientation receives an image of the object and a background as input, and outputs the object's position and orientation generated by random variables sampled based on a predetermined distribution; The network for generating an image consisting of the object and the background generates an image in which the object is placed in the background based on the position and orientation of the object. The recognition device according to any one of claims 1 to 5.

7. The recognition device according to claim 6 , wherein the generation unit calculates an area that disappears due to overlap of a plurality of the objects, and generates an image in which the object is placed in a background by projecting the area onto an image of the object.

8. 1. A recognition method that causes a computer to perform the following steps: A step of learning a recognition model using an image consisting of an object and a background as input data so that data including the type, position, and orientation of the object becomes output data; a step of acquiring recognition result data, which is a result of recognizing at least the type, position, and orientation of the object from an image consisting of the object and a background using the recognition model; generating, from an image consisting of an object and a background, a set of other data including the type, position, and orientation of the object, and another image including the object and background arranged based on the other data, using a generative model; training the generative model based on a value of a loss function indicating a deviation between another recognition result data, which is a result of recognizing the other image, and the other data; A recognition method having the following.

9. obtaining initial training data consisting of a set of given images of an object and a background, and data including the type, position, and orientation of the object; training the recognition model using the initial training data; recognizing the images of the initial training data using the recognition model to obtain the recognition result data; training the generative model in a direction that minimizes a value of a loss function that indicates a discrepancy between the recognition result data and data including the type, position, and orientation of the object included in the initial training data; The method of claim 8 further comprising:

10. A process of training a recognition model using an image of an object and a background as input data so that data including the type, position, and orientation of the object is output data; a process of acquiring recognition result data, which is a result of recognizing at least the type, position, and orientation of the object from an image consisting of the object and a background using the recognition model; a process of generating, from an image consisting of an object and a background, a set of other data including the type, position, and orientation of the object, and another image including the object and background arranged based on the other data, using a generative model; a process of training the generative model based on a value of a loss function indicating a deviation between another recognition result data, which is a result of recognizing the other image, and the other data; A program that causes a computer to execute the following.