Learning Data Generation Device and Learning Data Generation Method

Abstract

The learning data generation apparatus (100) includes: a 3D model acquisition unit (110) that acquires 3D model information representing a 3D model of an object; a local image acquisition unit (120) that acquires local image information representing an image region of an object captured in a captured image, i.e., a local image; a texture coordinate acquisition unit (130) that acquires 2D texture coordinates for texture mapping of the local image to the 3D model based on the local image information and the 3D model information; a rendering condition acquisition unit (140) that acquires rendering condition information representing rendering conditions, which are conditions for rendering a textured 3D model for which texture mapping of the local image to the 3D model has been performed based on the 2D texture coordinates; a 2D image acquisition unit (150) that acquires 2D image information representing a 2D image by rendering the textured 3D model based on the rendering condition information; and a learning data output unit (190) that outputs 2D image information.

Description

Technical Field

[0001] This invention relates to a data generation apparatus and a method for generating data for learning. Background Technology

[0002] The captured image information is input as a descriptive variable into the trained model, enabling the trained model to infer the shape, center position, or category of an object. The captured image is an image obtained by taking a picture of the object.

[0003] In order for the trained model to make high-precision inferences, a large amount of training data is needed for the model to learn.

[0004] For example, Non-Patent Document 1 discloses a technique in which CG image information, representing a CG image obtained by taking a picture of a 3D model created by computer graphics (hereinafter referred to as "CG") using a virtual camera, is used as learning data for a learning model to learn and generate a trained model. This technique generates a trained model by using the captured image information as an explanatory variable input, and the trained model is able to infer the center position of an object reflected in the captured image.

[0005] In the technology disclosed in Non-Patent Document 1 (hereinafter referred to as "Prior Art"), a virtual camera is used to photograph 3D models created by CG, thereby generating a large amount of learning data.

[0006] Non-patent literature 1: "Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel", "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World", [online], "arXiv preprint arXiv:1703.06907", [Searched on November 12, 2003, Japan], Internet (URL: https: / / arxiv.org / abs / 1703.06907) Summary of the Invention

[0007] Existing 3D models are created through CG and are simple in shape and appearance. Therefore, existing technologies are suitable for generating trained models for inferring the shape, center position, or category of objects with simple shapes or appearances.

[0008] However, in existing technologies, in order to generate a trained model that can accurately infer the shape, center position, or category of objects with simple shapes or appearances, more training data is needed for the learning model to learn. That is, existing technologies suffer from the problem that the learning model requires a long learning time to generate a trained model that can accurately infer the shape, center position, or category of objects with simple shapes or appearances.

[0009] The present invention is proposed to solve the above-mentioned problems, and its purpose is to provide a learning data generation device that can generate learning data for objects with complex shapes or appearances, so that the learning time required to generate a trained model is shorter than that of the prior art. The trained model can infer the shape, center position or category of the object with high accuracy.

[0010] The learning data generation apparatus of the present invention comprises: a local image acquisition unit that acquires local image information representing an image region of an object captured in a captured image, i.e., a local image; a 3D model acquisition unit that acquires 3D model information representing a 3D model; a texture coordinate acquisition unit that acquires 2D texture coordinates for texture mapping of the local image shown in the local image information to the 3D model shown in the 3D model information based on the local image information acquired by the local image acquisition unit and the 3D model information acquired by the 3D model acquisition unit; a rendering condition acquisition unit that acquires rendering condition information representing rendering conditions, which are conditions for rendering a textured 3D model to which texture mapping of the local image shown in the local image information to the 3D model shown in the 3D model information has been performed based on the 2D texture coordinates acquired by the texture coordinate acquisition unit; a 2D image acquisition unit that renders the textured 3D model based on the rendering condition information acquired by the rendering condition acquisition unit to acquire 2D image information representing a 2D image; and a learning data output unit that outputs the 2D image information acquired by the 2D image acquisition unit.

[0011] The effects of the invention

[0012] According to the present invention, even for objects with complex shapes or appearances, the learning time required to generate a trained model is shorter than that of the prior art, and the trained model can infer the shape, center position or category of the object with high accuracy. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an example of the structure of the main part of an object inference system constructed using the learning data generation apparatus according to Implementation Method 1.

[0014] Figure 2 This is a block diagram illustrating an example of the structure of the main parts of the learning data generation apparatus according to Embodiment 1.

[0015] Figure 3 This is an explanatory diagram showing an example of a 3D model obtained by the 3D model acquisition unit 110 of the learning data generation apparatus according to Embodiment 1.

[0016] Figure 4A This is an explanatory diagram showing an example of a captured image obtained by the image acquisition unit of the learning data generation apparatus according to Embodiment 1. Figure 4B This indicates that the local image acquisition unit of the learning data generation apparatus according to Embodiment 1 obtains data from the background subtraction method. Figure 4A The illustration shows an example of a partial image extracted from a captured image.

[0017] Figure 5A This refers to the texture coordinate acquisition unit of the learning data generation apparatus according to Embodiment 1. Figure 3 An illustration of an example of a UV unwrapped map obtained by UV unwrapping the 3D model shown. Figure 5B This indicates that the texture coordinate acquisition unit of the learning data generation apparatus according to Embodiment 1 is located in... Figure 5A The illustration shows an example of a UV unfolded diagram after rotating and scaling the UV coordinates.

[0018] Figure 6 This is an explanatory diagram showing an example of a textured 3D model according to Embodiment 1.

[0019] Figure 7A and Figure 7B This is a diagram illustrating an example of the hardware structure of the main parts of the learning data generation device according to Embodiment 1.

[0020] Figure 8 This is a flowchart illustrating an example of the processing of the learning data generation apparatus according to Embodiment 1. Detailed Implementation

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Implementation Method 1

[0023] Reference Figures 1 to 8 The learning data generation apparatus 100 according to Embodiment 1 will be described.

[0024] Figure 1This is a block diagram illustrating an example of the structure of the main part of the object inference system 1 constructed using the learning data generation device 100 according to embodiment 1.

[0025] The object inference system 1 includes a learning data generation device 100, a storage device 10, a learning device 20, and an inference device 30.

[0026] Storage device 10 is a device for storing electronic information, having a storage medium such as SSD (Solid State Drive) or HDD (Hard Disk Drive). Storage device 10 is connected to learning data generation device 100, learning device 20, or inference device 30 via wired or wireless communication means.

[0027] The learning data generation device 100 generates learning data and outputs the generated learning data to the learning device 20 or the storage device 10. This learning data is used when performing machine learning to infer the shape, center position, or category of an object. Details of the learning data generation device 100 will be described later.

[0028] The learning device 20 acquires learning data and uses the acquired learning data to perform machine learning for inferring the shape, center position, or category of an object. Specifically, the learning device 20 acquires learning data output by the learning data generation device 100 from the learning data generation device 100 or the storage device 10 for the purpose of performing machine learning.

[0029] The learning device 20 outputs trained model information, representing the trained model corresponding to the learning result obtained by the machine learning, to the inference device 30 or the storage device 10. The trained model shown in the trained model information output by the learning device 20 is, for example, a neural network having an input layer, intermediate layers, and an output layer.

[0030] The learning device 20 is composed of, for example, a general-purpose computer such as a personal computer.

[0031] The inference device 30 reads from the storage device 10 or from the storage device 10. Figure 1The imaging device (not shown) acquires image information representing an image obtained by photographing an object that is the object of inference. Additionally, the inference device 30 acquires trained model information output by the learning device 20 from the learning device 20 or the storage device 10. The inference device 30 inputs the acquired image information as a descriptive variable into the trained model shown in the acquired trained model information, thereby enabling the trained model to infer the shape, center position, or category of the object reflected in the image shown in the image information. The inference device 30 outputs inference result information representing the result inferred by the trained model to the storage device 10 or... Figure 1 The output device is not shown in the figure. Furthermore, the output device may be a display output device such as a monitor. The output device is not limited to a display output device; it may also be a lighting device such as a lamp or a sound output device such as a speaker. The output device acquires the inference result information output by the inference device 30 and outputs the acquired inference result information in a state that can be recognized by the user, via light or sound.

[0032] The inference device 30 and the learning device 20 are each composed of, for example, a general-purpose computer such as a personal computer.

[0033] Reference Figure 2 The structure of the main parts of the learning data generation device 100 according to Embodiment 1 will be described.

[0034] Figure 2 This is an example of the structure of the main parts of the learning data generation device 100 according to Embodiment 1.

[0035] The learning data generation device 100 includes a 3D model acquisition unit 110, a local image acquisition unit 120, a texture coordinate acquisition unit 130, a rendering condition acquisition unit 140, a 2D image acquisition unit 150, and a learning data output unit 190.

[0036] The learning data generation device 100 may also include, in addition to the 3D model acquisition unit 110, the local image acquisition unit 120, the texture coordinate acquisition unit 130, the rendering condition acquisition unit 140, the 2D image acquisition unit 150, and the learning data output unit 190, an operation receiving unit 101, an image acquisition unit 121, and a tag acquisition unit 160.

[0037] Regarding the learning data generation apparatus 100 according to Embodiment 1, such as Figure 2 As shown, the structure is described with an operation receiving unit 101, a 3D model acquisition unit 110, a local image acquisition unit 120, a captured image acquisition unit 121, a texture coordinate acquisition unit 130, a rendering condition acquisition unit 140, a 2D image acquisition unit 150, a label acquisition unit 160, and a learning data output unit 190.

[0038] Operation receiving unit 101 connects to the keyboard or pointing device, etc. Figure 2 The operation receiving unit 101 receives operation signals output from the operation input device (not shown) and converts these operation signals into operation information corresponding to the operation signals. Specifically, the operation receiving unit 101 receives operation signals output from the operation input device when the user operates the operation input device, and converts these operation signals into operation information corresponding to the operation signals.

[0039] The operation receiving unit 101 outputs the transformed operation information to the 3D model acquisition unit 110, the local image acquisition unit 120, the captured image acquisition unit 121, and the rendering condition acquisition unit 140, etc.

[0040] The 3D model acquisition unit 110 acquires 3D model information representing a 3D (hereinafter referred to as "3D") model. The 3D model acquisition unit 110 acquires 3D model information, for example, by reading it from the storage device 10. The 3D model acquisition unit 110 may also pre-store the 3D model information. Alternatively, the 3D model acquisition unit 110 may acquire 3D model information based on operation information output by the operation receiving unit 101. More specifically, for example, a user specifies 3D model information stored in the storage device 10 by operating an operation input device. The operation receiving unit 101 receives the operation signal representing the specified 3D model information, converts it into operation information corresponding to the operation signal, and outputs the converted operation information to the 3D model acquisition unit 110. The 3D model acquisition unit 110 acquires the 3D model information desired by the user by acquiring the operation information from the operation receiving unit 101 and reading the user-specified 3D model information from the storage device 10.

[0041] Figure 3 This is an explanatory diagram showing an example of a 3D model obtained by the 3D model acquisition unit 110 of the learning data generation apparatus 100 according to Embodiment 1.

[0042] Specifically, Figure 3 It is a diagram that visualizes the 3D model as a 2D image by using computer graphics (hereinafter referred to as "CG"). The 3D model information obtained by the 3D model acquisition unit 110 is used to visualize the 3D model.

[0043] The local image acquisition unit 120 acquires local image information representing the image region in the captured image that is the object to be inferred, i.e., the local image.

[0044] Specifically, for example, the local image acquisition unit 120 acquires local image information by reading local image information from the storage device 10 that has pre-stored local image information.

[0045] More specifically, for example, the local image acquisition unit 120 acquires local image information specified by the user through the operation input device based on the operation information acquired by the operation receiving unit 101.

[0046] The image acquisition unit 121 acquires image information representing the captured image of the object being inferred.

[0047] Specifically, for example, the image acquisition unit 121 acquires the image information by reading the image information from the storage device 10 that has pre-stored the image information.

[0048] More specifically, for example, the image acquisition unit 121 acquires the image information specified by the user through the operation input device based on the operation information acquired by the operation receiving unit 101.

[0049] When the learning data generation apparatus 100 includes an image acquisition unit 121, the local image acquisition unit 120 may extract local image information representing the captured image shown by the image information acquired by the image acquisition unit 121. For example, it may extract the foreground using a background subtraction method, extract a rectangular region containing the extracted foreground region from the captured image, and thereby obtain local image information representing the image region of the captured object in the captured image, i.e., a local image. Since the method of extracting the foreground from the image using the background subtraction method is a known technique, its description is omitted. Alternatively, the local image acquisition unit 120 may extract a rectangular region containing the foreground region from the captured image using an SSD (Single Shot MultiBox Detector) or the like. Since the method of extracting a rectangular region containing the foreground region from the image using an SSD or the like is a known technique, its description is omitted.

[0050] Figure 4 is an explanatory diagram showing an example of a local image extracted by the local image acquisition unit 120 of the learning data generation apparatus 100 according to Embodiment 1 from a captured image shown by the captured image information obtained by the captured image acquisition unit 121 using the background subtraction method.

[0051] Specifically, Figure 4A This is an explanatory diagram showing an example of a captured image displayed by the image acquisition unit 121 of the learning data generation apparatus 100 according to Embodiment 1. Additionally, Figure 4B This indicates that the local image acquisition unit 120 of the learning data generation apparatus 100 according to Embodiment 1 obtains data from the background subtraction method. Figure 4A The illustration shows an example of a partial image extracted from a captured image.

[0052] As shown in Figure 4, the local image acquisition unit 120 acquires images from... Figure 4A As an example, the captured image is used to extract a rectangular region containing the captured object, i.e., the foreground region, through background subtraction. Figure 4B The image shown here is a partial image, which is presented as an example.

[0053] As described above, the local image acquisition unit 120 is configured to extract a local image from the captured image information obtained by the captured image acquisition unit 121, and obtain local image information representing the local image. As a result, the learning data generation device 100 can automate the generation of local image information.

[0054] The texture coordinate acquisition unit 130 acquires 2D texture coordinates for mapping the local image shown by the local image information to the 3D model shown by the 3D model information based on the local image information acquired by the local image acquisition unit 120 and the 3D model acquisition unit 110.

[0055] Specifically, for example, the texture coordinate acquisition unit 130 performs UV unwrapping on the 3D model shown in the 3D model information and acquires 2D texture coordinates, i.e., UV coordinates, for mapping the local image shown in the local image information to the 3D model that has undergone UV unwrapping.

[0056] Since the method for obtaining UV coordinates for texturing an image to a 3D model that has undergone UV unwrapping is a well-known technique, its description is omitted.

[0057] The texture coordinate acquisition unit 130 can also perform coordinate transformation on the acquired UV coordinates by rotating, paralleling, or scaling up or down at least one of these methods, and obtain the transformed UV coordinates as 2D texture coordinates for mapping the local image to the 3D model.

[0058] For example, UV coordinates can be transformed into transformed UV coordinates using the following equation (1).

[0059]

[0060] Here, (U, V) are the UV coordinates, (U′, V′) are the transformed UV coordinates, and offset is... u and offset v The amount of movement that makes the UV coordinates move in parallel, θ is the angle that makes the UV coordinates rotate, and α is the magnification (reduction) ratio that makes the UV coordinates magnified or reduced.

[0061] Figure 5 shows the texture coordinate acquisition unit 130 of the learning data generation apparatus 100 according to Embodiment 1. Figure 3 An illustration of an example of a UV unwrapped map obtained by UV unwrapping the 3D model shown.

[0062] Specifically, Figure 5A This refers to the texture coordinate acquisition unit 130 of the learning data generation apparatus 100 according to Embodiment 1. Figure 3 This is an explanatory diagram of an example of a UV unwrapped pattern obtained by UV unwrapping the shown 3D model. Additionally, Figure 5B This indicates that the texture coordinate acquisition unit 130 of the learning data generation apparatus 100 according to Embodiment 1 is in Figure 5A The illustration shows an example of a UV unfolded diagram after rotating and scaling the UV coordinates.

[0063] The rendering condition acquisition unit 140 acquires rendering condition information representing the rendering conditions based on the 2D texture coordinates acquired by the texture coordinate acquisition unit 130. The rendering conditions are for rendering a textured 3D model that has performed texture mapping from the local image shown by the local image information to the 3D model shown by the 3D model information.

[0064] Figure 6 This is an explanatory diagram showing an example of a textured 3D model according to Embodiment 1.

[0065] Specifically, Figure 6 The textured 3D model shown is based on Figure 3 The following is an example of a 3D model's UV unwrapping diagram, shown as an example. Figure 5B The UV unfolding diagram shown will be Figure 4B The local image shown as an example is directed towards... Figure 3 The 3D model shown as an example has undergone texture mapping.

[0066] For example, the rendering condition acquisition unit 140 acquires the rendering condition information by reading the rendering condition information from the storage device 10 that stores the rendering condition information in advance.

[0067] Specifically, for example, the rendering condition information obtained by the rendering condition acquisition unit 140 is information representing the conditions when a textured 3D model in CG space is photographed by a virtual camera.

[0068] More specifically, for example, the rendering condition acquisition unit 140 acquires information such as the position or pose of the 3D model in CG space, including the size of the 3D model including the bounding box in CG space, the position or pose of the virtual camera in CG space, or the position of the light source in CG space or the color of the light emitted by the light source, as rendering condition information.

[0069] Furthermore, the rendering condition information obtained by the rendering condition acquisition unit 140 is not limited to one; the rendering condition acquisition unit 140 obtains multiple rendering condition information with different rendering conditions.

[0070] Furthermore, the method by which the rendering condition acquisition unit 140 acquires rendering condition information is not limited to the method of acquiring rendering condition information by reading rendering condition information from the storage device 10 through the rendering condition acquisition unit 140.

[0071] For example, the rendering condition acquisition unit 140 acquires rendering conditions by reading information from the storage device 10 that represents a mathematical formula that can determine rendering conditions. These rendering conditions include the position or pose of the 3D model in CG space shown in the 3D model information acquired by the 3D model acquisition unit 110, the size of the 3D model including the bounding box in CG space, the position or pose of the virtual camera in CG space, or the position of the light source in CG space or the color of the light emitted by the light source. The rendering condition acquisition unit 140 may also acquire rendering condition information by substituting predefined values ​​into the parameters included in the mathematical formula shown in the information.

[0072] Here, the value that the rendering condition acquisition unit 140 substitutes into the parameter is not limited to one. For example, it can be multiple values ​​that are different from each other. The rendering condition acquisition unit 140 can also substitute multiple values ​​that are different from each other as predefined values ​​into the parameter in turn to determine the rendering conditions.

[0073] The 2D image acquisition unit 150 renders a textured 3D model based on the rendering condition information acquired by the rendering condition acquisition unit 140 to obtain 2D image information representing a 2D image.

[0074] Specifically, when the rendering condition acquisition unit 140 acquires each of a plurality of different rendering condition information, the 2D image acquisition unit 150 renders the textured 3D model based on the rendering condition information acquired by the rendering condition acquisition unit 140 to obtain 2D image information representing a 2D image.

[0075] The learning data output unit 190 outputs the 2D image information acquired by the 2D image acquisition unit 150.

[0076] Specifically, the learning data output unit 190 outputs the 2D image information obtained by the 2D image acquisition unit 150 each time the rendering condition acquisition unit 140 obtains each of the multiple different rendering condition information.

[0077] For example, the learning data output unit 190 outputs 2D image information to the storage device 10 or the learning device 20.

[0078] The learning device 20 acquires 2D image information output by the learning data output unit 190 as learning data, and uses the acquired learning data to perform machine learning to generate a trained model for inferring the shape, center position, or category of an object.

[0079] The learning data generation device 100, having the structure described above, is able to output multiple 2D image information based on a single local image information, based on local image information representing the image region of the captured object in the captured image obtained by capturing an object.

[0080] The learning device 20 uses the 2D image information output by the learning data generation device 100 as learning data to perform machine learning. Since the 2D image information contains local images, even when the shape or appearance of the object being inferred is complex, the learning time required to generate the trained model is shorter than that of the prior art. The trained model can infer the shape, center position or category of the object with high accuracy.

[0081] That is, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies, and the trained model can infer the shape, center position or category of the object with high accuracy.

[0082] In addition, the 2D image acquisition unit 150 may acquire, in addition to acquiring 2D image information, supplementary image information such as segmented images or depth images corresponding to the 2D image shown by the 2D image information.

[0083] Since the methods for obtaining segmented or depth images by rendering textured 3D models are well-known techniques, the description is omitted.

[0084] The learning data generation device 100 may also have a tag acquisition unit 160.

[0085] The tag acquisition unit 160 acquires tag information representing the tag associated with the 2D image information acquired by the 2D image acquisition unit 150.

[0086] When the learning data generation device 100 has a tag acquisition unit 160, the learning data output unit 190 outputs, in addition to the 2D image information shown by the 2D image acquisition unit 150, the tag information acquired by the tag acquisition unit 160 and the 2D image information in association.

[0087] Specifically, for example, when the rendering condition acquisition unit 140 acquires each of a plurality of different rendering condition information, the learning data output unit 190 outputs the 2D image information acquired by the 2D image acquisition unit 150 and the tag information acquired by the tag acquisition unit 160 in association to the storage device 10 or the learning device 20.

[0088] The learning device 20 acquires 2D image information and associated label information output by the learning data output unit 190 as learning data, and uses the acquired learning data to perform machine learning to generate a trained model for inferring the shape, center position, or category of an object.

[0089] For example, the tag acquisition unit 160 acquires partial rendering information of at least a portion of the rendering conditions shown in the rendering condition information used when the 2D image acquisition unit 150 acquires 2D image information, and uses this as tag information. Since the rendering condition information has been described previously, its description is omitted.

[0090] The tag information obtained by the tag acquisition unit 160 is not limited to partial rendering information.

[0091] For example, if the 2D image acquisition unit 150 acquires additional image information, such as a segmented image or a depth image, that corresponds to the 2D image shown by the 2D image information in addition to acquiring 2D image information, the tag acquisition unit 160 may also acquire the additional image information acquired by the 2D image acquisition unit 150 as tag information.

[0092] The learning data generation device 100, having the structure described above, is able to output multiple information groups that associate 2D image information and label information based on a local image, representing the image region of the captured object in a captured image obtained by capturing an object.

[0093] The learning device 20 uses the 2D image information output by the learning data generation device 100 as learning data to perform machine learning. Since the 2D image information contains local images, even if the shape or appearance of the object being inferred is complex, the learning time required to generate the trained model is shorter than that of the prior art. The trained model can infer the shape, center position or category of the object with high accuracy.

[0094] That is, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies, and the trained model can infer the shape, center position or category of the object with high accuracy.

[0095] In addition, the learning device 20 uses label information as learning data in addition to the 2D image information output by the learning data generation device 100 to perform machine learning. As a result, the learning time required to generate a trained model is shorter compared to the case where only 2D image information is used as learning data for machine learning.

[0096] That is, by generating information sets that associate 2D image information and label information, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. The trained model can infer the shape, center position or category of an object with high accuracy.

[0097] Reference Figure 7A and Figure 7B The hardware structure of the main parts of the learning data generation device 100 according to Embodiment 1 will be described.

[0098] Figure 7A and 7B This is a diagram illustrating an example of the hardware structure of the main parts of the learning data generation apparatus 100 according to Embodiment 1.

[0099] like Figure 7A As shown, the learning data generation device 100 is composed of a computer, which has a processor 201 and a memory 202. The memory 202 stores programs for enabling the computer to function as an operation receiving unit 101, a 3D model acquisition unit 110, a local image acquisition unit 120, a captured image acquisition unit 121, a texture coordinate acquisition unit 130, a rendering condition acquisition unit 140, a 2D image acquisition unit 150, a tag acquisition unit 160, and a learning data output unit 190. By reading and executing the programs stored in the memory 202 from the processor 201, the operation receiving unit 101, the 3D model acquisition unit 110, the local image acquisition unit 120, the captured image acquisition unit 121, the texture coordinate acquisition unit 130, the rendering condition acquisition unit 140, the 2D image acquisition unit 150, the tag acquisition unit 160, and the learning data output unit 190 are realized.

[0100] In addition, such as Figure 7B As shown, the learning data generation device 100 can also be configured by a processing circuit 203. In this case, the processing circuit 203 can also perform the functions of the operation receiving unit 101, the 3D model acquisition unit 110, the local image acquisition unit 120, the captured image acquisition unit 121, the texture coordinate acquisition unit 130, the rendering condition acquisition unit 140, the 2D image acquisition unit 150, the tag acquisition unit 160, and the learning data output unit 190.

[0101] Alternatively, the learning data generation device 100 may also be composed of a processor 201, a memory 202, and a processing circuit 203 (not shown). In this case, the processor 201 and the memory 202 may perform some of the functions of the operation receiving unit 101, the 3D model acquisition unit 110, the local image acquisition unit 120, the captured image acquisition unit 121, the texture coordinate acquisition unit 130, the rendering condition acquisition unit 140, the 2D image acquisition unit 150, the tag acquisition unit 160, and the learning data output unit 190, while the processing circuit 203 performs the remaining functions.

[0102] Processor 201 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), microprocessor, microcontroller, or DSP (Digital Signal Processor).

[0103] The memory 202 may use, for example, semiconductor memory or a hard disk. More specifically, the memory 202 may use RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), SSD (Solid State Drive) or HDD (Hard Disk Drive), etc.

[0104] The processing circuit 203 may use, for example, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration).

[0105] Reference Figure 8 The operation of the learning data generation device 100 according to Embodiment 1 will be explained.

[0106] Figure 8 This is a flowchart illustrating an example of the processing of the learning data generation apparatus 100 according to Embodiment 1.

[0107] The data generation device 100 is used to repeatedly execute the process of the flowchart, for example.

[0108] First, in step ST801, the 3D model acquisition unit 110 acquires 3D model information.

[0109] Next, in step ST811, the texture coordinate acquisition unit 130 acquires the 2D texture coordinates, i.e., the UV coordinates.

[0110] Next, in step ST812, the texture coordinate acquisition unit 130 acquires the transformed UV coordinates.

[0111] Next, in step ST821, the image acquisition unit 121 acquires the image information.

[0112] Next, in step ST822, the local image acquisition unit 120 acquires local image information.

[0113] Next, in step ST831, the rendering condition acquisition unit 140 acquires rendering condition information.

[0114] Next, in step ST832, the 2D image acquisition unit 150 acquires 2D image information.

[0115] Next, in step ST833, the 2D image acquisition unit 150 acquires the accompanying image information.

[0116] Next, in step ST834, the tag acquisition unit 160 acquires tag information.

[0117] Next, in step ST835, the learning data output unit 190 outputs the 2D image information and label information in association.

[0118] After performing the processing in step ST835, the learning data generation device 100 ends the processing of the flowchart, for example, by returning to the processing in step ST801 and repeating the processing of the flowchart.

[0119] The learning data generation device 100 may also, when the 2D image acquisition unit 150 repeatedly renders using the 3D model information acquired by the 3D model acquisition unit 110 in step ST801, end the flowchart processing after executing the processing of step ST835, return to the processing of step ST811 or step ST812, and repeat the processing of the flowchart.

[0120] Alternatively, when the 2D image acquisition unit 150 repeatedly renders based on the UV coordinates obtained by the texture coordinate acquisition unit 130 in step ST811, after executing the processing of step ST835, the flowchart processing can be terminated and returned to the processing of step ST812 or step ST821 to repeat the processing of the flowchart.

[0121] Alternatively, when the 2D image acquisition unit 150 repeatedly renders based on the transformed UV coordinates obtained by the texture coordinate acquisition unit 130 in step ST812, after executing the processing of step ST835, the processing of the flowchart ends and returns to the processing of step ST821 to repeat the processing of the flowchart.

[0122] Alternatively, when the local image acquisition unit 120 repeatedly acquires local image information using the image acquisition unit 121 acquired in step ST821, after executing the processing in step ST835, the flowchart processing ends and returns to the processing in step ST822 to repeat the flowchart processing.

[0123] Alternatively, the learning data generation device 100 may repeatedly acquire rendering condition information in the rendering condition acquisition unit 140, and repeatedly render the 2D image acquisition unit 150 for each rendering condition information repeatedly acquired by the rendering condition acquisition unit 140. After executing the processing in step ST835, the processing of the flowchart ends, and the process returns to the processing in step ST831, and the processing of the flowchart is repeated.

[0124] Furthermore, in this flowchart, step ST812 can be omitted if the texture coordinate acquisition unit 130 does not have the function of acquiring transformed UV coordinates, step ST821 can be omitted if the local image acquisition unit 120 does not have the function of extracting local images from captured images, and step ST833 can be omitted if the 2D image acquisition unit 150 does not have the function of acquiring accompanying image information.

[0125] Furthermore, as long as the processing of step ST801 is performed before the processing of step ST811, the processing of step ST811 is performed before the processing of step ST812, and the processing of step ST821 is performed before the processing of step ST822, the order of processing from step ST801 to step ST822 is arbitrary.

[0126] As described above, the learning data generation apparatus 100 includes: a 3D model acquisition unit 110, which acquires 3D model information representing a 3D model; a local image acquisition unit 120, which acquires local image information representing an image region, i.e., a local image, of an object captured in a captured image; a texture coordinate acquisition unit 130, which acquires 2D texture coordinates for texture mapping of the local image shown by the local image information to the 3D model shown by the 3D model information, based on the local image information acquired by the local image acquisition unit 120 and the 3D model information acquired by the 3D model acquisition unit 110; and a rendering condition acquisition unit 14. 0, which obtains rendering condition information representing rendering conditions based on the 2D texture coordinates obtained by the texture coordinate acquisition unit 130. The rendering conditions are conditions for rendering a textured 3D model that has performed texture mapping from the local image shown by the local image information to the 3D model shown by the 3D model information; 2D image acquisition unit 150, which obtains 2D image information representing 2D images by rendering the textured 3D model based on the rendering condition information obtained by the rendering condition acquisition unit 140; and learning data output unit 190, which outputs the 2D image information obtained by the 2D image acquisition unit 150.

[0127] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0128] In addition, as described above, the learning data generation device 100 is configured to include a tag acquisition unit 160 in addition to the above structure. The tag acquisition unit 160 acquires tag information that represents tags related to the 2D image information acquired by the 2D image acquisition unit 150. The learning data output unit 190 outputs the tag information acquired by the tag acquisition unit 160 in association with the 2D image information, in addition to the 2D image information acquired by the 2D image acquisition unit 150.

[0129] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0130] Furthermore, as described above, the learning data generation apparatus 100 is configured such that, in the above structure, the 2D image acquisition unit 150 acquires, in addition to acquiring 2D image information, also acquires accompanying image information representing a segmented image or a depth image corresponding to the 2D image shown by the 2D image information, and the tag acquisition unit 160 acquires the accompanying image information acquired by the 2D image acquisition unit 150 as tag information.

[0131] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0132] Furthermore, as described above, the learning data generation apparatus 100 is configured such that, in the above structure, the tag acquisition unit 160 acquires partial rendering information representing at least a portion of the rendering conditions shown in the rendering condition information used when the 2D image acquisition unit 150 acquires 2D image information, and uses this as tag information.

[0133] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0134] In addition to the above structure, the learning data generation apparatus 100 also includes an image acquisition unit 121, which acquires image information representing an image of a captured object. The local image acquisition unit 120 extracts the foreground from the captured image using the background subtraction method based on the captured image information acquired by the image acquisition unit 121, and extracts a rectangular region containing the extracted foreground region from the captured image, thereby acquiring local image information representing the image region of the captured object in the captured image, i.e., the local image.

[0135] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies, and without the need for pre-generating local image information. The trained model can infer the shape, center position, or category of an object with high accuracy.

[0136] Furthermore, as described above, the learning data generation apparatus 100 is configured such that, in the above structure, the texture coordinate acquisition unit 130 performs UV unwrapping on the 3D model shown in the 3D model information and acquires 2D texture coordinates, i.e., UV coordinates, for texture mapping of the local image shown in the local image information to the 3D model that has undergone UV unwrapping.

[0137] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0138] Furthermore, as described above, the learning data generation apparatus 100 is configured such that, in the above structure, the texture coordinate acquisition unit 130 performs coordinate transformation on the acquired UV coordinates by rotating, paralleling, or scaling up or down, and obtains the transformed UV coordinates, i.e., the transformed UV coordinates, as 2D texture coordinates for mapping local images to 3D models.

[0139] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0140] Furthermore, as described above, the learning data generation apparatus 100 is configured such that, in the above structure, the rendering condition acquisition unit 140 acquires the conditions when the textured 3D model in CG space is photographed by the virtual camera. That is, it acquires information representing at least one of the following: the position and pose of the 3D model in CG space as shown by the 3D model information acquired by the 3D model acquisition unit 110; the size of the 3D model including the bounding box in CG space; the position and pose of the virtual camera in CG space; and the position of the light source in CG space and the color of the light emitted by the light source.

[0141] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0142] Variations of Implementation Method 1

[0143] When the object to be inferred is one, the learning data generation apparatus 100 according to Embodiment 1 outputs 2D image information, or outputs 2D image information and tag information associated with the 2D image information.

[0144] The learning data generation device 100 can also output 2D image information, or output 2D image information and tag information associated with the 2D image information, when there are multiple objects that are the objects of inference.

[0145] The following describes the case where the learning data generation device 100 outputs 2D image information, or outputs 2D image information and tag information associated with the 2D image information, when there are multiple objects that are the objects of inference.

[0146] Specifically, for example, the 3D model acquisition unit 110 acquires 3D model information corresponding to each of the multiple objects that are the objects of inference. That is, the 3D model acquisition unit 110 acquires a number of 3D model information that is equivalent to the number of objects that are the objects of inference.

[0147] Furthermore, the local image acquisition unit 120 acquires, for example, local image information corresponding to each of the multiple objects that are to be inferred. That is, the local image acquisition unit 120 acquires a number of local image information that is equivalent to the number of objects that are to be inferred.

[0148] The texture coordinate acquisition unit 130, for example, acquires 2D texture coordinates for each 3D model information based on multiple local image information acquired by the local image acquisition unit 120 and multiple 3D model information acquired by the 3D model acquisition unit 110, for mapping the local image shown by the local image information corresponding to the 3D model information to the 3D model shown by each of the multiple 3D model information.

[0149] Specifically, for example, the texture coordinate acquisition unit 130 performs UV unwrapping on the 3D models shown by the multiple 3D model information, and acquires 2D texture coordinates, i.e., UV coordinates, for each 3D model information to perform texture mapping on the local image shown by the local image information corresponding to the 3D model information to each of the multiple 3D models that have undergone UV unwrapping.

[0150] The preferred texture coordinate acquisition unit 130 performs coordinate transformation on the UV coordinates obtained for each 3D model by rotating, paralleling, or scaling up or down, and obtains the transformed UV coordinates, i.e., the transformed UV coordinates, as 2D texture coordinates for mapping local images to 3D models.

[0151] As described above, the texture coordinate acquisition unit 130 performs coordinate transformation on the UV coordinates acquired for each 3D model by rotating, paralleling, or scaling up or down, thereby enabling the learning data generation device 100 to configure textured 3D models corresponding to multiple objects as inference objects in a variety of ways in CG space.

[0152] The rendering condition acquisition unit 140 acquires rendering condition information representing the rendering conditions based on the 2D texture coordinates acquired by the texture coordinate acquisition unit 130 for each 3D model information. The rendering conditions are used to simultaneously render multiple textured 3D models that have performed texture mapping on local images shown by local image information corresponding to the 3D model information to the 3D models shown by the multiple 3D model information.

[0153] The 2D image acquisition unit 150 acquires 2D image information representing a 2D image by rendering textured 3D models corresponding to multiple objects that are inference objects, based on the rendering condition information acquired by the rendering condition acquisition unit 140.

[0154] The learning data output unit 190 outputs the 2D image information acquired by the 2D image acquisition unit 150.

[0155] By configuring it as described above, even when there are multiple objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than the prior art. The trained model can infer the shape, center position, or category of each of the multiple objects with high accuracy.

[0156] Other variations of implementation method 1

[0157] The learning data generation apparatus 100 according to Embodiment 1 obtains 3D model information corresponding to the object to be inferred, renders a textured 3D model that performs texture mapping on the image area (i.e., local image) of the object captured to the 3D model shown in the 3D model information, and outputs 2D image information or outputs 2D image information and label information associated with the 2D image information.

[0158] Alternatively, the learning data generation device 100 may, in addition to the 3D model information corresponding to the object that is being inferred, also acquire background model information representing the 3D model corresponding to the background of the object, i.e., the background model. At the same time, it may render a 3D model with background texture that has performed texture mapping from the background image to the background model, and a 3D model with texture that has performed texture mapping from a local image of the object that is being inferred to the 3D model, thereby outputting 2D image information, or outputting 2D image information and label information associated with the 2D image information.

[0159] The following describes a situation where the learning data generation device 100 simultaneously renders a 3D model with a background texture that has performed texture mapping from a background image to a background model, and a 3D model with a texture that has performed texture mapping from a local image of an object that is the object of inference to a 3D model, thereby outputting 2D image information, or outputting 2D image information and label information associated with the 2D image information.

[0160] Specifically, for example, the 3D model acquisition unit 110 acquires 3D model information (hereinafter referred to as "object model") representing a 3D model corresponding to the object being inferred (hereinafter referred to as "object model"), and background model information representing a 3D model, i.e., a background model, corresponding to the background of the object.

[0161] Additionally, the local image acquisition unit 120 acquires, for example, local image information (hereinafter referred to as "object local image information") representing a local image of an object that has been captured and is to be inferred (hereinafter referred to as "object local image"), and local image information (hereinafter referred to as "background image information") representing a local image of an image region in the captured image where no object was captured.

[0162] The texture coordinate acquisition unit 130, for example, acquires 2D texture coordinates for texture mapping of the object local image shown in the object local image information to the object model shown in the object model information, based on object local image information acquired by the local image acquisition unit 120 and object model information acquired by the 3D model acquisition unit 110. Additionally, the texture coordinate acquisition unit 130, for example, acquires 2D texture coordinates for texture mapping of the background image shown in the background image information to the background model shown in the background model information, based on background image information acquired by the local image acquisition unit 120 and background model information acquired by the 3D model acquisition unit 110.

[0163] Specifically, for example, the texture coordinate acquisition unit 130 performs UV unwrapping on the object model shown in the object model information to obtain 2D texture coordinates, i.e., UV coordinates, for mapping the object local image shown in the object local image information to the object model that has undergone UV unwrapping. Additionally, the texture coordinate acquisition unit 130 performs UV unwrapping on the background model shown in the background model information to obtain 2D texture coordinates, i.e., UV coordinates, for mapping the background image shown in the background image information to the background model that has undergone UV unwrapping.

[0164] The preferred texture coordinate acquisition unit 130 performs a coordinate transformation on the UV coordinates obtained by UV unwrapping the object model, by rotating, paralleling, or scaling up or down, to obtain the transformed UV coordinates, which are then used as 2D texture coordinates for mapping a local image of the object to the object model. Alternatively, the preferred texture coordinate acquisition unit 130 performs a coordinate transformation on the UV coordinates obtained by UV unwrapping the background model, by rotating, paralleling, or scaling up or down, to obtain the transformed UV coordinates, which are then used as 2D texture coordinates for mapping a background image to the background model.

[0165] As described above, the texture coordinate acquisition unit 130 performs coordinate transformation on the UV coordinates obtained by UV unwrapping each of the object model and the background model by rotating, paralleling, scaling up or down, thereby enabling the learning data generation device 100 to configure a variety of textured 3D models corresponding to the object as the inference object and textured 3D models corresponding to the background of the object, i.e., textured 3D models with background, in the CG space.

[0166] The rendering condition acquisition unit 140 acquires rendering condition information representing the rendering conditions, which are conditions for simultaneously rendering a textured 3D model corresponding to the object being inferred and a background textured 3D model corresponding to the background of the object.

[0167] The 2D image acquisition unit 150 acquires 2D image information representing a 2D image by simultaneously rendering a textured 3D model corresponding to the object being inferred and a background textured 3D model corresponding to the background of the object, based on the rendering condition information acquired by the rendering condition acquisition unit 140.

[0168] The learning data output unit 190 outputs the 2D image information acquired by the 2D image acquisition unit 150.

[0169] By configuring it as described above, even for objects with complex shapes or appearances, the learning data generation device 100 can generate a trained model with a shorter learning time than existing technologies. This trained model can infer the shape, center position, or category of an object with high accuracy.

[0170] Furthermore, within the scope of its disclosure, the present invention allows for free combination of various embodiments or arbitrary modification of the constituent elements of various embodiments, or arbitrary omission of constituent elements in various embodiments.

[0171] Industrial applicability

[0172] The learning data generation device of the present invention is suitable for object inference systems, learning systems or inference systems, etc.

[0173] Explanation of the label

[0174] 1. Object inference system; 10. Storage device; 20. Learning device; 30. Inference device; 100. Learning data generation device; 101. Operation receiving unit; 110. 3D model acquisition unit; 120. Local image acquisition unit; 121. Image acquisition unit; 130. Texture coordinate acquisition unit; 140. Rendering condition acquisition unit; 150. 2D image acquisition unit; 160. Tag acquisition unit; 190. Learning data output unit; 201. Processor; 202. Memory; 203. Processing circuit.

Claims

1. A data generation device for learning, characterized in that, have: The 3D model acquisition unit acquires 3D model information representing the object's 3D model. The local image acquisition unit acquires local image information representing a local image, which is an image region in the captured image where the object is captured; The texture coordinate acquisition unit acquires 2D texture coordinates for mapping the local image shown by the local image information to the 3D model shown by the 3D model information, based on the local image information acquired by the local image acquisition unit and the 3D model information acquired by the 3D model acquisition unit. The rendering condition acquisition unit acquires rendering condition information representing the rendering conditions, which are conditions for rendering a textured 3D model to which texture mapping has been performed on the local image shown by the local image information to the 3D model shown by the 3D model information based on the 2D texture coordinates acquired by the texture coordinate acquisition unit. A 2D image acquisition unit, which, based on the rendering condition information acquired by the rendering condition acquisition unit, renders the textured 3D model to obtain 2D image information representing a 2D image; and The learning data output unit outputs the 2D image information acquired by the 2D image acquisition unit.

2. The learning data generation device according to claim 1, characterized in that, It has a tag acquisition unit that acquires tag information representing tags related to the 2D image information acquired by the 2D image acquisition unit. In addition to the 2D image information acquired by the 2D image acquisition unit, the learning data output unit also outputs the tag information acquired by the tag acquisition unit in association with the 2D image information.

3. The learning data generation device according to claim 2, characterized in that, In addition to acquiring the 2D image information, the 2D image acquisition unit also acquires accompanying image information representing the segmented image or depth image corresponding to the 2D image shown by the 2D image information. The tag acquisition unit acquires the accompanying image information acquired by the 2D image acquisition unit as the tag information.

4. The learning data generation device according to claim 2, characterized in that, The tag acquisition unit acquires at least a portion of the rendering conditions shown in the rendering condition information used when the 2D image acquisition unit acquires the 2D image information, and uses this as the tag information.

5. The learning data generation device according to claim 1, characterized in that, It has an image acquisition unit that acquires image information representing an image of the object being captured. The local image acquisition unit extracts the foreground using the background subtraction method for the captured image shown by the captured image information obtained by the captured image acquisition unit, and extracts a rectangular region containing the extracted foreground region from the captured image, thereby obtaining the local image information representing the image region in the captured image where the object was captured, i.e., the local image.

6. The learning data generation device according to claim 1, characterized in that, The texture coordinate acquisition unit performs UV unwrapping on the 3D model shown in the 3D model information to obtain the 2D texture coordinates, i.e., UV coordinates, for mapping the local image shown in the local image information to the 3D model that has undergone UV unwrapping.

7. The learning data generation apparatus according to claim 6, characterized in that, The texture coordinate acquisition unit performs coordinate transformation on the acquired UV coordinates by rotating, paralleling, and scaling up or down at least one of them, and obtains the transformed UV coordinates, i.e., the transformed UV coordinates, as the 2D texture coordinates used for texture mapping of the local image to the 3D model.

8. The learning data generation device according to claim 1, characterized in that, The rendering condition acquisition unit acquires the conditions when the textured 3D model in CG space is photographed by a virtual camera. Specifically, it acquires information representing at least one of the following: the position and pose of the 3D model in CG space as shown by the 3D model information acquired by the 3D model acquisition unit; the size of the 3D model including the bounding box in CG space; the position and pose of the virtual camera in CG space; and the position of the light source in CG space and the color of the light emitted by the light source.

9. A method for generating learning data, characterized in that, have: The 3D model acquisition process involves obtaining 3D model information representing the object. The local image acquisition step involves acquiring local image information representing a local image, where the local image is the image region in the captured image where the object is captured. In the texture coordinate acquisition step, the texture coordinate acquisition unit obtains 2D texture coordinates for mapping the local image shown by the local image information to the 3D model shown by the 3D model information based on the local image information obtained through the local image acquisition step and the 3D model information obtained through the 3D model acquisition step. The rendering condition acquisition step involves the rendering condition acquisition unit acquiring rendering condition information representing the rendering conditions. These rendering conditions are used to render a textured 3D model that has undergone texture mapping from the local image shown by the local image information to the 3D model shown by the 3D model information based on the 2D texture coordinates obtained through the texture coordinate acquisition step. In the 2D image acquisition step, the 2D image acquisition unit renders the textured 3D model based on the rendering condition information obtained through the rendering condition acquisition step to obtain 2D image information representing a 2D image; and The learning data output step involves the learning data output unit outputting the 2D image information obtained through the 2D image acquisition step.

Images