Object detection model learning device and object detection model learning method

Abstract

To provide an object detection model training device capable of improving training accuracy of an object detection model.SOLUTION: In an object detection model training device, a training data generating unit 100 generates training data by using computer graphics to synthesize images of a detection target object, and generates a first teaching signal related to the detection target object, and a second teaching signal corresponding to a state of the detection target object during imaging. An object detection model comprises a first inferring unit 161 for outputting a first inferred result relating to the detection target object, and a second inferring unit 162 for outputting a second inferred result relating to the state of the detection target object during imaging. The first inferring unit 161 obtains the first inferred result using an input image and the second inferred result. An object detection training unit 150 trains the second inferring unit 162 by providing the training data to the second inferring unit 162, and trains the first inferring unit 161 by providing the training data and the second inferred result to the first inferring unit 161.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an object detection model learning device, an object detection device, and an object detection model learning method.

Background Art

[0002] Conventionally, when learning object detection by image recognition, computer graphics (CG) has been used to generate learning data. For example, Patent Document 1 describes, "To obtain a learned model used when performing object detection processing, posture detection processing, etc., a learning data generation system that can acquire a large amount of learning data required during learning processing in a short time is provided." and "The learning data generation system acquires a background image obtained by imaging a three-dimensional space. Also, it acquires CG object generation data, which is data for computer graphics processing including at least one of the shape and texture of an object. A CG object image is generated based on the acquired CG object generation data. A rendering image obtained by synthesizing the CG object image with the background image so that the CG object is arranged at a predetermined position in the three-dimensional space is acquired as a learning image."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the conventional technology, learning accuracy equivalent to that when using a real image obtained by actually imaging an object to be detected as learning data can be achieved. However, the conventional technology does not achieve higher learning accuracy than learning with real images.

[0005] An object of the present invention is to improve the learning accuracy of an object detection model. [Means for solving the problem]

[0006] To achieve the above objective, one representative object detection model learning device of the present invention comprises: an object detection learning unit that learns an object detection model that detects objects from an input image; and a learning data generation unit that generates learning data used for the learning; the learning data generation unit generates the learning data by synthesizing images of the object to be detected using computer graphics; generates a first training signal relating to the object to be detected with respect to the learning data; generates a second training signal relating to the state of the object to be detected at the time of imaging with respect to the learning data; and the object detection model comprises: a first inference unit that outputs a first inference result relating to the object to be detected with respect to the input image; and an inference unit that outputs a first inference result relating to the object to be detected with respect to the input image. The object detection learning unit comprises: a second inference unit that outputs a second inference result regarding the state of the image; the first inference unit obtains the first inference result using the input image and the second inference result; the object detection learning unit comprises: a second error calculation unit that obtains the difference between the second inference result obtained by providing the learning data to the second inference unit and the second teacher signal as a second error; a first error calculation unit that obtains the difference between the first inference result obtained by providing the learning data and the second inference result to the first inference unit and the first teacher signal as a first error; and an inference parameter update unit that updates the parameters of the second inference unit based on the second error and updates the parameters of the first inference unit based on the first error. Furthermore, one representative object detection device of the present invention comprises a first inference unit that outputs a first inference result regarding an object to be detected based on an input image, and a second inference unit that outputs a second inference result regarding the state of the object to be detected at the time of imaging based on the input image, wherein the first inference unit obtains the first inference result using the input image and the second inference result. An object detection learning method for training an object detection model that detects objects from input images, comprising the steps of: generating training data by synthesizing images of objects to be detected using computer graphics; generating a first teacher signal relating to the objects to be detected with respect to the training data; generating a second teacher signal relating to the state of the objects to be detected at the time of imaging with respect to the training data; providing the training data to a second inference unit of the object detection model to obtain a second inference result relating to the state of the objects to be detected at the time of imaging; providing the training data and the second inference result to a first inference unit of the object detection model to obtain a first inference result relating to the objects to be detected; determining the difference between the first inference result and the first teacher signal as a first error; determining the difference between the second inference result and the second teacher signal as a second error; updating the parameters of the second inference unit based on the second error; and updating the parameters of the first inference unit based on the first error. [Effects of the Invention]

[0007] According to the present invention, the learning accuracy of object detection models can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a diagram showing the configuration of an object detection model learning device. [Figure 2] Figure 2 is a diagram showing the configuration of the object detection device. [Figure 3] Figure 3 is an explanatory diagram of the constraints on the CG synthesis parameters. [Figure 4] Figure 4 shows a specific example of the first teacher signal selection condition. [Figure 5] Figure 5 shows a specific example of the second teacher signal selection condition. [Figure 6] Figure 6 is a flowchart showing the learning process steps. [Figure 7]Figure 7 is a flowchart showing the object detection processing procedure. [Figure 8] Figure 8 shows a specific example of a training signal. [Figure 9] Figure 9 shows a specific example of a computer-generated image. [Figure 10] Figure 10 is an explanatory diagram of the inference results. [Figure 11] Figure 11 is a diagram showing the configuration of an object detection model learning device according to a modified example. [Figure 12] Figure 12 is a diagram showing the configuration of an object detection device according to a modified example. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Note that the embodiments described below are not intended to limit the invention as defined in the claims, and not all elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention. Furthermore, elements that are essential to the structure of the invention but are well-known may be omitted from the illustration and description.

[0010] In the following explanation, the term "xxx table" may be used to describe information that yields output for a given input. This information can be data with any structure. Therefore, "xxx table" can also be referred to as "xxx information."

[0011] Furthermore, in the following explanation, the structure of each table is just an example; one table may be divided into two or more tables, or all or part of two or more tables may constitute a single table.

[0012] Furthermore, in the following explanation, the process may be described using "program" as the subject. Since a program is executed by the processor unit and performs defined processes using the memory unit and / or interface unit as appropriate, the subject of the process may be the processor unit (or a device such as a controller that has that processor unit).

[0013] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-transitory) recording medium. Also, in the following description, two or more programs may be implemented as one program, or one program may be implemented as two or more programs.

[0014] Also, the "processor unit" is one or more processors. A processor is typically a microprocessor such as a CPU (Central Processing Unit), but may also be another type of processor such as a GPU (Graphics Processing Unit). Also, the processor may be single-core or multi-core. Also, the processor may be a processor in a broad sense such as a hardware circuit (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

[0015] Also, in the following description, when describing elements of the same type without distinction, reference signs (or common signs among the reference signs) may be used, and when describing elements of the same type separately, the identification numbers (or reference signs) of the elements may be used. Also, the number of each element shown in each figure is an example and is not limited to the illustration.

Example

[0016] FIG. 1 is a configuration diagram of an object detection model learning device. The object detection model learning device shown in FIG. 1 includes an object detection learning unit 150 and a learning data generation unit 100. The object detection learning unit 150 performs learning of an object detection model that detects an object from an input image. The learning data generation unit 100 generates learning data used by the object detection learning unit 150 for learning.

[0017] The training data generation unit 100 generates training data by synthesizing images of the object to be detected using computer graphics. The training data generation unit 100 also generates a first training signal related to the object to be detected and a second training signal corresponding to the state of the object at the time of imaging using the training data.

[0018] The object detection learning unit 150 includes an object detection unit 160, a first error calculation unit 170, a second error calculation unit 171, and an inference parameter update unit 180. The object detection unit 160 is a processing unit corresponding to the object detection model and has a first inference unit 161 and a second inference unit 162.

[0019] The second inference unit 162 outputs a second inference result regarding the state of the object to be detected at the time of imaging, based on the input image. The first inference unit 161 outputs a first inference result regarding the object to be detected, based on the input image. At this time, the first inference unit 161 uses the input image and the second inference result to obtain the first inference result. For example, if the second inference unit 162 outputs the camera position and lighting direction as inference results, the first inference unit 161 considers the camera position and lighting direction to infer the presence or absence of the object to be detected.

[0020] The first inference unit 161 and the second inference unit 162 can utilize known object detection techniques. For example, SSD (Single Shot MultiBox Detector) and YOLO (You only look once: Unified, real-time object detection). These technologies utilize neural networks and, upon inputting an image, output numerical information indicating the type of object to be detected and its range within the image. Since the output numerical information is not limited to these types, it can be used not only by the first inference unit 161 but also by the second inference unit 162.

[0021] The second error calculation unit 171 calculates the second error by determining the difference between the second inference result obtained by providing the training data as an input image to the second inference unit 162 and the second training signal. The first error calculation unit 170 provides the training data and the second inference result to the first inference unit 161 and calculates the difference between the first inference result and the first teacher signal as the first error.

[0022] The inference parameter update unit 180 updates the parameters of the second inference unit 162 based on the second error and updates the parameters of the first inference unit 161 based on the first error, thereby training the object detection unit 160.

[0023] The learning data generation unit 100 includes a condition input unit 110, a CG synthesis parameter constraint condition storage unit 120, a CG synthesis parameter generation unit 121, a learning CG generation unit 122, a teacher signal selection condition storage unit 130, a teacher signal selection unit 131, a first teacher signal generation unit 132, and a second teacher signal generation unit 133.

[0024] The condition input unit 110 is an interface that accepts input of various conditions related to learning. These conditions related to learning include CG synthesis parameter constraints, which are constraints on the parameters of computer graphics, and teacher signal selection conditions, which are conditions for selecting a teacher signal. The teacher signal selection conditions also include a selection condition for a first teacher signal and a selection condition for a second teacher signal.

[0025] The CG synthesis parameter constraint storage unit 120 stores the input CG synthesis parameter constraints. The teacher signal selection condition storage unit 130 stores the input teacher signal selection conditions. The CG synthesis parameter constraint storage unit 120 and the teacher signal selection condition storage unit 130 can be implemented using any storage medium.

[0026] The CG synthesis parameter generation unit 121 generates multiple CG synthesis parameters within the range of the CG synthesis parameter constraints and outputs them to the learning CG generation unit 122. The learning CG generation unit 122 generates learning data for each of the multiple CG synthesis parameters generated by the CG synthesis parameter generation unit 121.

[0027] For example, when detecting a device, the CG synthesis parameter generation unit 121 generates a CG image of the device's appearance based on CG synthesis parameters that specify the type of device, orientation, lighting direction, etc., and generates training data by superimposing it on a background image. The background image may be a real image or a CG image.

[0028] The teacher signal selection unit 131 performs a process to select items to be included in the teacher signal based on the CG synthesis parameters and the teacher signal selection conditions. The first teacher signal generation unit 132 generates a first teacher signal for the teacher signal items selected by the teacher signal selection unit 131. The second teacher signal generation unit 133 generates a second teacher signal for the teacher signal items selected by the teacher signal selection unit 131.

[0029] For example, if the CG synthesis parameters have values ​​that specify the type, orientation, and lighting direction of the device, and the type of object to be detected is specified as the first training signal, then the value of the device type becomes the first training signal. Similarly, if the lighting direction is specified as the second training signal, then the value of the lighting direction becomes the second training signal.

[0030] As will be described in more detail later, the first training signal preferably includes the type of object to be detected and the position of the object in the training data. The second training signal preferably includes at least one of the following: the positional relationship between the object to be detected and the camera, the lighting conditions for the object to be detected, and the deformation state of the object to be detected. In other words, the first training signal is information used as the output for object detection, indicating what is where. In contrast, the second training signal is supplementary information that contributes to improving the accuracy of object detection.

[0031] The second teacher signal may further include the type of object to be detected and the position of the object in the training data. In other words, the second teacher signal may contain the same information as the first teacher signal. In this case, the second inference unit outputs the information that the first inference unit should output as its inference result, and the first inference unit performs inference again based on the second inference result.

[0032] Figure 2 is a diagram of the object detection device configuration. The object detection device has an object detection unit 160 as shown in Figure 1, to which an image input unit 200 and an inference result output unit 210 are connected. In this configuration, the image input unit 200 provides the same input image to the first inference unit 161 and the second inference unit 162. The second inference unit 162 infers the state at the time of imaging from the input image and outputs it to the first inference unit 161.

[0033] The first inference unit 161 performs image recognition on the input image, taking into account the state at the time of imaging inferred by the second inference unit 162, and makes inferences about the presence and position of the object to be detected, and outputs the inference result to the inference result output unit 210. In Figure 2, the second inference unit 162 also outputs to the inference result output unit 210. External output from the second inference unit 162 is not mandatory, but output is possible if the state during imaging is required as output.

[0034] Figure 3 is an explanatory diagram of the CG synthesis parameter constraints. In the example shown in Figure 3, the CG synthesis parameter constraint 300 indicates the range of values ​​that an item can take by associating the minimum and maximum input values ​​with the item name. Note that some item names have sub-item names, in which case the minimum and maximum input values ​​are associated with each sub-item name.

[0035] The item names include the type of object to be detected, the number of objects to be detected, the position of the object to be detected, the orientation of the object to be detected, the deformation parameters of the object to be detected, the position of the camera, the direction of the camera, the number of lights, the position of the lights, the direction of the lights, the illumination range, the illuminance, the color of the lights, etc.

[0036] The type of object to be detected is, for example, "device A". The number of detection targets is the number of detection targets to include in a single training dataset. In Figure 3, a range of 1 to 3 targets is specified.

[0037] The position of the detected object indicates the spatial coordinate range in which the object to be detected is placed. When indicating the position of the detected object using the XYZ coordinate system, X, Y, and Z are used as sub-coordinate names. In Figure 3, the range of the X coordinate is 50 to 200, the range of the Y coordinate is 30 to 100, and the range of the Z coordinate is 0 to 0.

[0038] The direction of detection indicates the range of orientation of the object being detected. For example, two sub-items, horizontal angle φ and vertical angle θ, are used, with the range of φ specified as 10 degrees to 30 degrees and the range of θ as 0 degrees to 0 degrees. The deformation parameters of the object to be detected have the movable part of the object to be detected as the sub-item name. The deformation parameters vary depending on the type of object to be detected. For example, if device A has a rotating arm as the deformable part, the sub-item name should be the arm angle, with a range of 10 to 50 degrees. The deformation parameters may also be for expandable members, sliding members, etc. If there are multiple deformable parts, the number of sub-items should be increased and a range set for each.

[0039] The camera position indicates the spatial coordinates of a virtual camera, that is, the range of the viewpoint from which the CG-generated object is viewed. When indicating the camera position using the XYZ coordinate system, X, Y, and Z are used as sub-coordinate names. In Figure 3, the range of the X coordinate is 0.0m to 5.0m, the range of the Y coordinate is 5.0m to 10.0m, and the range of the Z coordinate is 3.0m to 3.5m.

[0040] The camera direction indicates the range of the camera's orientation. For example, two sub-items, horizontal angle φ and vertical angle θ, are used, with the range of φ being 0 to 360 degrees and the range of θ being 110 to 135 degrees.

[0041] The number of lights indicates the range of lights used in CG compositing. The items for lighting position, lighting direction, illumination range, illuminance, and lighting color must be specified for each light source. The lighting position corresponds to the position of the light source illuminating the CG-composited object and is specified using XYZ coordinates, similar to the position of the detection target and the camera position. The lighting direction is indicated by a horizontal angle φ and a vertical angle θ, similar to the position of the detection target and the camera orientation. The illumination range is the angle of the area illuminated by the light, for example, 30 to 45 degrees. Illuminance indicates the brightness of the light, for example, 1000 lm. The color of the light can be indicated, for example, using RGB.

[0042] Figure 4 shows a specific example of the first teacher signal selection condition. The first teacher signal selection condition 400 shown in Figure 4 specifies whether to select or not an item that can be learned as the output of the first inference. The items that can be learned as the output of the first inference include the type of object to be detected and the range of the object to be detected on the image. If the value of the selection status for any of these items is "yes", a teacher signal for that item is generated.

[0043] Specifically, if the type of object to be detected is "present," you can refer to the CG synthesis parameters and use the value of the type of object to be detected directly as the training signal. If the value for the type of detection range on the image is "present", the system refers to the CG synthesis parameters and generates a training signal by projecting the image of the detection target onto the plane of the camera's field of view, based on the spatial coordinates of the detection target object, the camera's spatial coordinates, and the camera's orientation.

[0044] Figure 5 shows a specific example of the second teacher signal selection condition. The second teacher signal selection condition 500 shown in Figure 5 specifies whether or not to select items that can be learned as the output of the second inference. Examples of items that can be learned as the output of the second inference are shown below.

[0045] "Type of detection target": This is the same as the first training signal, so the explanation is omitted. "The range of the detection target on the image" is the same as the first training signal, so the explanation is omitted. "Deformation parameters to be detected": Refer to the CG synthesis parameters and use the values ​​of the corresponding items directly as the training signal. The relative position and direction of the detected object relative to the camera can be calculated by referring to the CG synthesis parameters and based on the spatial coordinates of the object to be detected, the spatial coordinates of the camera, and the orientation of the camera. The "relative position of the lighting relative to the camera, and the relative direction of illumination" can be calculated by referring to the CG compositing parameters and based on the camera's spatial coordinates, camera orientation, lighting's spatial coordinates, and lighting orientation. "The relative position of the lighting and the relative direction of the illumination relative to the detected object" can be calculated by referring to the CG synthesis parameters and based on the spatial coordinates of the object being detected, the spatial coordinates of the lighting, and the direction of the lighting. The "illumination range" can be determined by referencing CG synthesis parameters and integrating information about the camera and multiple lighting sources. The "illuminance" parameter can be determined by referencing the CG compositing parameters and integrating information from the camera and multiple lighting sources. The "lighting color" can be determined by referencing the CG compositing parameters and integrating information about the camera and multiple lighting sources.

[0046] Figure 6 is a flowchart showing the learning process. First, the condition input unit 110 receives input for CG synthesis parameter constraint conditions and stores them in the CG synthesis parameter constraint condition storage unit 120 (S600). The condition input unit 110 also receives input for the first teacher signal selection condition and stores it in the teacher signal selection condition storage unit 130 (S601). Similarly, the condition input unit 110 receives input for the second teacher signal selection condition and stores it in the teacher signal selection condition storage unit 130 (S602).

[0047] The CG synthesis parameter generation unit 121 generates each CG synthesis parameter randomly according to the constraints of the CG generation parameters (S603). The learning CG generation unit 122 generates learning CG, which is training data, using the generated CG synthesis parameters (S604).

[0048] The first teacher signal generation unit 132 generates a first teacher signal according to the first teacher signal selection conditions using the CG synthesis parameters and the training CG (S605). This teacher signal is directly related to object detection and includes the type of object to be detected and the range of the object to be detected in the image. The second teacher signal generation unit 133 generates a second teacher signal according to the second teacher signal selection conditions using the CG synthesis parameters and the training CG (S606). This teacher signal includes teacher signals that are not directly related to object detection. For example, training signals related to the sample object would include deformation parameters of the object being detected, as well as the relative position and direction of the object relative to the camera. Similarly, training signals related to illumination would include the relative position and direction of each light source relative to the camera, as well as the relative position and direction of each light source relative to the object being detected. Other examples include illumination range, illuminance, and illumination color. Furthermore, if the second training signal includes training signals related to the object to be detected, it is desirable to also include the first training signal. For example, if the second training signal contains deformation parameters of the object to be detected, the type of object to be detected is also required as a second training signal, since the deformation of the object differs depending on the type of object to be detected. In addition, the area of ​​the object to be detected on the image is also required as a training signal, as it indicates where on the image the effect of the deformation parameters occurs. Hereafter, the second training signal, excluding the portion that overlaps with the first training signal, will be referred to as the second training signal in the narrow sense.

[0049] Next, the second inference unit 162 calculates a second inference result using the first inference parameters and training CG held at that point (S607). Here, the second inference result is numerical information corresponding to the second training signal. Since the second inference result corresponds to the input image, it can also be output as a two-dimensional map. This output will be described later.

[0050] The first inference unit 161 calculates the first inference result using the second inference result and the training CG (S608). Alternatively, the first inference result may be output using only the portion of the second inference result corresponding to the second teacher signal in the narrow sense and the training CG. The first inference result is numerical information corresponding to the first teacher signal.

[0051] Next, the first error calculation unit 170 calculates the first error using the first inference result and the first training signal (S609). The second error calculation unit 171 calculates the second error using the second inference result and the second training signal (S610).

[0052] The inference parameter update unit 180 determines whether termination conditions have been met, such as whether the error has become sufficiently small or whether the parameter updates described below have been performed a predetermined number of times (S611). If the conditions are met, the process terminates; otherwise, the parameters of the first inference unit 161 and the second inference unit 162 are updated using a method such as backpropagation, so that the first and second errors are reduced (S612). After that, the process returns to S603.

[0053] Figure 7 is a flowchart showing the object detection processing procedure. First, the image input unit 200 receives the image to be inferred as the input image (S700). The second inference unit 162 calculates a second inference result using the image to be inferred (S701). The first inference unit 161 calculates the first inference result using the second inference result and the image to be inferred (S702). The inference result output unit 210 outputs the first inference result. If it is to be viewed by a person, it can be output as a screen; if the inference result is to be used by a system, it can be output to that system via the network. At this point, a second inference result may be output. This would allow detailed information about the detected object to be provided to the aforementioned person or system.

[0054] Figure 8 shows a specific example of a training signal. In this figure, the first training signal generates the type of object to be detected and the range of the object to be detected in the image. The second training signal generates the type of object to be detected, the range of the object to be detected in the image, the deformation parameters of the object to be detected, the relative position of the object to the camera, the relative direction of the object to the camera, the relative position of illumination 1 to the camera, the relative direction of illumination 1 to the camera, the relative position of illumination 1 to the object to be detected, the relative direction of illumination 1's illumination to the object to be detected, the illuminance of illumination 1, and the illumination color of illumination 1.

[0055] Figure 9 shows a specific example of a computer-generated image. Image 900 in Figure 9 includes computer-generated images 910 and 911. Computer-generated images 910 and 911 are images generated by computer-generated images of the detected object.

[0056] CG composite image 910 and CG composite image 911 are of the same type as the detected object, but their appearance differs depending on how the light hits them. CG composite image 910 is dark, while CG composite image 911 is bright.

[0057] It is difficult for the object detection unit 160 to learn that objects that look so different are the same type of object. However, the difficulty decreases when different appearances are provided as input. Therefore, the accuracy of object detection is improved by having the second inference unit 162 infer the difference in appearance and provide the result to the first inference unit 161. The same applies to the orientation of an object relative to the camera and the deformation parameters of the object.

[0058] Figure 10 is an explanatory diagram of the inference results. In the two-dimensional map 1000 of Figure 10, each element at a position on the map consists of a second inference result and a plurality of fixed-length vectors corresponding to it. Inference results concerning the entire screen, such as the relative position of the lighting relative to the camera and the relative direction of the lighting relative to the camera, are stored in one dimension of the vector, and this extends to the entire two-dimensional map. Furthermore, if sufficient training has been performed, the inference results corresponding to the CG composite image 910 are stored in yet another dimension of the vector, and this is limited to the region 1010 of the two-dimensional map. Similarly, the inference results corresponding to the CG composite image 911 are limited to the region 1011 of the two-dimensional map.

[0059] Next, we will describe a modified example in which the object detection unit has a shared inference unit. Figure 11 is a diagram of the configuration of an object detection model learning device according to a modified example. Figure 11 differs from Figure 1 in that the object detection unit 160 further includes a shared inference unit 1100. The output of the learning CG generation unit 122 is input to the shared inference unit 1100 as training data.

[0060] The shared inference unit 1100 performs predetermined inference processing on the input training data and then outputs the results to the first inference unit 161 and the second inference unit 162. In other words, with this configuration, processing can be made more efficient by performing preprocessing that is effective in common for both the first inference unit 161 and the second inference unit 162 in a shared inference unit 1100. Other configurations and operations are the same as in Figure 1, so their explanation will be omitted.

[0061] Figure 12 is a diagram showing the configuration of an object detection device according to a modified example. Figure 12 differs from Figure 1 in that the object detection unit 160 further includes a shared inference unit 1100. The shared inference unit 1100 receives an input image from the image input unit 200, performs a predetermined inference process on the input image, and then outputs it to the first inference unit 161 and the second inference unit 162. In other words, with this configuration, processing can be made more efficient by performing preprocessing that is effective in common for both the first inference unit 161 and the second inference unit 162 in a shared inference unit 1100. Other configurations and operations are the same as in Figure 2, so we will omit the explanation.

[0062] As described above, the object detection model learning device disclosed in the embodiment comprises an object detection learning unit 150 that learns an object detection model that detects objects from an input image, and a learning data generation unit 100 that generates learning data used for the learning. The training data generation unit 100 generates training data by synthesizing images of the object to be detected using computer graphics, generates a first training signal relating to the object to be detected with respect to the training data, and generates a second training signal corresponding to the state of the object to be detected at the time of imaging with respect to the training data. The object detection model comprises a first inference unit 161 that outputs a first inference result regarding the object to be detected for the input image, and a second inference unit 162 that outputs a second inference result regarding the state of the object to be detected at the time of imaging for the input image, wherein the first inference unit 161 obtains the first inference result using the input image and the second inference result. The object detection learning unit 150 is characterized by comprising: a second error calculation unit 171 that calculates the difference between a second inference result obtained by providing the learning data to the second inference unit 162 and the second teacher signal as a second error; a first error calculation unit 170 that calculates the difference between a first inference result obtained by providing the learning data and the second inference result to the first inference unit 161 and the first teacher signal as a first error; and an inference parameter update unit 180 that updates the parameters of the second inference unit based on the second error and updates the parameters of the first inference unit based on the first error. With this configuration and operation, since the state of the object to be detected at the time of imaging is used, the learning accuracy of the object detection model can be improved, and higher learning accuracy can be expected than with learning using real images.

[0063] Furthermore, the learning data generation unit 100 receives constraints regarding the computer graphics parameters, selection conditions for the first teacher signal, and selection conditions for the second teacher signal, generates a plurality of computer graphics synthesis parameters within the range of the constraints, generates the learning data based on the computer graphics synthesis parameters, generates the first teacher signal using the computer graphics synthesis parameters and the selection conditions for the first teacher signal, and generates the second teacher signal using the computer graphics synthesis parameters and the selection conditions for the second teacher signal. With this configuration and operation, it is possible to improve the learning accuracy of the object detection model by using the parameters of CG synthesis to learn about the state at the time of imaging.

[0064] Preferably, the first teacher signal includes the type of object to be detected and the position of the object to be detected in the training data. Furthermore, it is preferable that the second teacher signal includes at least one of the following: the positional relationship between the object to be detected and the camera, the state of illumination for the object to be detected, and the state of deformation of the object to be detected. By using these parameters as training data, it is possible to efficiently learn the conditions during image acquisition and more effectively improve the accuracy of object detection learning.

[0065] The second teacher signal may further include the type of object to be detected and the position of the object to be detected in the training data. The second parameter to be inferred may be affected by the type and location of the object being detected.

[0066] Furthermore, the object detection model may further include a shared inference unit 1100, and after performing inference processing on the input image by the shared inference unit, the first inference unit 161 and the second inference unit 162 may perform processing on the inference result of the shared inference unit. In this way, by performing preprocessing that is effective in common for both the first inference unit 161 and the second inference unit 162 in a shared inference unit 1100, the processing can be made more efficient.

[0067] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace or add configurations, not just delete them.

[0068] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a recording medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored on the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the embodiment described above, and the program code itself and the recording medium on which it is stored constitute the present invention. Examples of recording media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0069] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0070] In the above-described embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. All components may be interconnected. [Explanation of symbols]

[0071] 100: Training data generation unit, 110: Condition input unit, 120: CG synthesis parameter constraint condition storage unit, 121: CG synthesis parameter generation unit, 122: Training CG generation unit, 130: Teacher signal selection condition storage unit, 131: Teacher signal selection unit, 132: First teacher signal generation unit, 133: Second teacher signal generation unit, 150: Object detection learning unit, 160: Object detection unit, 161: First inference unit, 162: Second inference unit, 170: First error calculation unit, 171: Second error calculation unit, 180: Inference parameter update unit, 200: Image input unit, 210: Inference result output unit, 1100: Shared inference unit

Claims

1. An object detection learning unit that trains an object detection model to detect objects from an input image, The system includes a learning data generation unit that generates learning data used for the aforementioned learning, The aforementioned training data generation unit is: The training data is generated by synthesizing images of the object to be detected using computer graphics. With respect to the aforementioned training data, a first training signal relating to the object to be detected is generated, With respect to the aforementioned training data, a second training signal corresponding to the state of the object to be detected at the time of imaging is generated. The aforementioned object detection model is A first inference unit outputs a first inference result regarding the object to be detected with respect to the input image, The system includes a second inference unit that outputs a second inference result regarding the state of the object to be detected at the time of imaging, based on the input image. The first inference unit obtains the first inference result using the input image and the second inference result. The object detection learning unit is A second error calculation unit calculates the difference between the second inference result obtained by providing the training data to the second inference unit and the second training signal as the second error, A first error calculation unit that provides the training data and the second inference result to the first inference unit and calculates the difference between the first inference result obtained and the first teacher signal as the first error, An object detection model learning device comprising: an inference parameter update unit that updates the parameters of the second inference unit based on the second error and updates the parameters of the first inference unit based on the first error.

2. The object detection model learning device according to claim 1, The aforementioned training data generation unit is: The constraints relating to the parameters of the computer graphics, the selection conditions for the first teacher signal, and the selection conditions for the second teacher signal are accepted. Within the scope of the aforementioned constraints, a plurality of computer graphics synthesis parameters are generated, and the training data is generated based on the computer graphics synthesis parameters. The first teacher signal is generated using the computer graphics synthesis parameters and the selection conditions for the first teacher signal. An object detection model learning device characterized by generating the second teacher signal using the computer graphics synthesis parameters and the selection conditions for the second teacher signal.

3. The object detection model learning device according to claim 1, An object detection model learning device characterized in that the first teacher signal includes the type of object to be detected and the position of the object to be detected in the training data.

4. The object detection model learning device according to claim 1, The object detection model learning device is characterized in that the second teacher signal includes at least one of the following: the positional relationship between the object to be detected and the camera, the state of illumination for the object to be detected, and the state of deformation of the object to be detected.

5. The object detection model learning device according to claim 4, The object detection model learning device is characterized in that the second teacher signal further includes the type of object to be detected and the position of the object to be detected in the training data.

6. The object detection model learning device according to claim 1, The object detection model learning device further comprises a shared inference unit, wherein after performing inference processing on the input image by the shared inference unit, the first inference unit and the second inference unit perform processing on the inference result of the shared inference unit.

7. An object detection learning method for training an object detection model that detects objects from an input image, The process involves generating training data by synthesizing images of the object to be detected using computer graphics, and The steps include generating a first training signal relating to the object to be detected with respect to the training data, The steps include generating a second training signal corresponding to the state of the object to be detected at the time of imaging using the aforementioned training data, The steps include providing the training data to the second inference unit of the object detection model and obtaining a second inference result regarding the state of the object to be detected at the time of imaging, The first inference unit of the object detection model is given the training data and the second inference result to obtain a first inference result regarding the object to be detected. The first step is to determine the difference between the first inference result and the first training signal as the first error, The steps include: determining the difference between the second inference result and the second training signal as the second error; The steps include updating the parameters of the second inference unit based on the second error and updating the parameters of the first inference unit based on the first error, A method for learning an object detection model, characterized by including the following:

Images