Data generation device and data generation method

Abstract

To provide a data generation device and a data generation method capable of reducing man-hours required for collection of learning data for a learning-based method, by generating an image and teacher information, which are the learning data of the learning-based method, from original learning data when a change is made in a camera placement or a camera main body.SOLUTION: A data generation device includes: a pseudo image generation part 110 which generates a pseudo image 112 of an object 101 corresponding to a second parameter 108 which is a parameter of a virtual camera by inputting the second parameter, to a space constitution device 111 trained with at least an image 102 of the object and a first parameter 105, which is a parameter of a real camera when an image of the object is taken; and a teacher information update part 115 which updates teacher information corresponding to the first parameter in a teacher information storage part 114 to teacher information corresponding to the second parameter using the first parameter, the second parameter and three-dimensional information of the object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a data generation device and a data generation method for generating learning data used for learning an area estimation method for an object in an image.

Background Art

[0002] One of the tasks targeted by industrial robots such as transport carts is the transportation work of basket carts used in warehouses and factories. The transportation work of basket carts is an operation in which parts and the like are placed on a basket cart and transported to a shipping location or an assembly location in accordance with a slip or an assembly procedure manual in a warehouse or a factory. Although it is a simple operation, it is a heavy operation and places a physical burden on the operator. In order to improve the efficiency of the transportation work of basket carts, an expansion of the application of a transport cart equipped with an area estimation function for an object in an image by image processing is expected.

[0003] The area estimation for an object in an image has also greatly improved in accuracy with the development of deep learning technology. The knowledge has been applied in the field of autonomous driving. However, problems specific to learning-based methods still remain.

[0004] For example, consider a system in which an area used for towing a basket cart is estimated using a learning-based method for an image of a basket cart taken by a camera, and the basket cart is towed by a transport cart based on the estimation result. The learning data at this time can be an image in which an object such as a basket cart appears and teacher information indicating a specific area such as an area used for towing the basket cart.

[0005] If the camera body or camera placement in this system is changed, the modified camera will acquire images in which the object appears differently from the image intended to be used as input for area estimation using a learning-based method. For example, changing the camera body changes the camera's focal length. When the camera's focal length changes, the area occupied by the object in the image changes, even if the same camera placement is used for shooting. Also, changing the camera placement changes the relative position and orientation relationship between the camera and the object. When the camera placement changes, the way the object appears in the image changes, even if the same camera is used for shooting.

[0006] Learning-based methods assume that region estimation is performed by inputting images in which the object is depicted in the same way as the images used for training data. Therefore, when performing region estimation on images in which the object is depicted differently from those used for training data collection, there is a concern that the estimation accuracy will decrease. To prevent this decrease in estimation accuracy, one method is to recollect training data and retrain the model after making changes to the camera. However, collecting training data is problematic because it requires time and effort for image capture and labeling the captured images with region information used for towing the trolley, which serves as training information.

[0007] As background technologies for this field, for example, there are technologies such as those described in Patent Document 1 and Patent Document 2.

[0008] The abstract of Patent Document 1 discloses that "a style transfer network is trained to perform style transfer, which converts any realistic input image into a converted image, using an image captured by a target camera as a style target image. The converted image has the content of the input image, maintains the realism of the input image, and has a style that matches the style of the style target image. Using the trained style transfer network, the training images of the original training dataset are converted into converted training images, and each converted training image is labeled with the training label of the corresponding training image in the original training dataset to create an augmented training dataset, and a deep neural network (DNN) is trained to perform a specific task using the augmented training dataset."

[0009] Furthermore, the abstract of Patent Document 2 discloses that "the spatial environment of the target real environment is reproduced in three dimensions within a virtual space, including dynamic objects, and the success or failure of the machine learning estimation results in the real environment is automatically determined by comparing the positional relationships of objects existing in the virtual space with the estimation results obtained by machine learning in the real environment."

[0010] Furthermore, paragraphs 0037 to 0041 of Patent Document 2 describe that if object detection fails, images and annotation data for expanding the training data are generated according to the cause of the error, and the object detection model is retrained.

[0011] In addition to Patent Documents 1 and 2, a technique called NeRF (Neural Radiance Fields) is known. NeRF is a technique that uses images of an object and the parameters of the camera used to capture the object to train a neural network that represents the captured three-dimensional space, and generates a realistic image of the object as if it were captured by cameras in a virtual arrangement. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] Japanese Patent Publication No. 2019-32821 [Patent Document 2] Japanese Patent Publication No. 2022-81081 [Overview of the project] [Problems that the invention aims to solve]

[0013] As mentioned above, a problem arises when changes are made to the camera: recollecting training data to prevent a decrease in the estimation accuracy of learning-based methods requires a significant amount of effort.

[0014] Patent Document 1 describes a method for creating an augmented training dataset by style-transforming the training images of the original training dataset, which are realistic input images. However, since it only performs style-transformation, it has the problem of not being able to generate corresponding images when the camera position changes. Similarly, Patent Document 1 also has the problem of not being able to handle changes in camera position because it uses the same training labels as the training information before style-transformation.

[0015] In Patent Document 2, the images used to augment the training data are artificial images that reproduce the scene using three-dimensional computer graphics, such as Figures 6(c) and 10(c) of Patent Document 2. The annotation data used to augment the training data are annotation images that are correctly color-coded with bounding boxes, such as Figure 6(d) of Patent Document 2, and with colors corresponding to each object in Figure 10(c) of Patent Document 2, such as Figure 10(d) of Patent Document 2. Artificial images taken in a virtual space, such as those in Patent Document 2, will look different from actual, realistic images due to differences in color, shading, reflections, and light sources. Therefore, there is a problem that using non-realistic artificial images as training data may reduce the estimation accuracy of the learning-based method for actual, realistic images.

[0016] Furthermore, Patent Document 2 generates region information that serves as training information by projecting the object's existence region, as shown in Figure 10(d) of Patent Document 2. This presents a problem in that it cannot generate region information that represents only a part of the object, rather than the entire object, such as specific region information like the region used for towing a trolley.

[0017] Furthermore, while NeRF is a technology that can generate realistic images after a camera placement change, it cannot generate training data for the changed camera placement. Therefore, there is a problem in that the generation of training data must be done manually.

[0018] Therefore, the problem that the present invention aims to solve is to provide a data generation device and a data generation method that can reduce the effort required to collect training data for a learning-based method by generating images and teacher information, which are training data for a learning-based method, from the original training data when the camera arrangement or the camera body is changed. [Means for solving the problem]

[0019] To solve the above problems, the data generation device of the present invention includes, for example, an image receiving unit that receives an image of an object, a first parameter receiving unit that receives a first parameter which is the parameter of the actual camera when the image of the object was taken, a second parameter receiving unit that receives a second parameter which is the parameter of a virtual camera, a three-dimensional information receiving unit that receives three-dimensional information of the object, a teacher information storage unit that receives and stores teacher information corresponding to the first parameter, and a pseudo-image generation unit that generates a pseudo-image of the object corresponding to the second parameter by inputting the second parameter into a spatial constructor that has been trained using at least the image and the first parameter, and the first parameter, the second parameter and the three-dimensional information and before Teacher information corresponding to the first parameter in the teacher information storage unit Using Training information corresponding to the second parameter generate Teacher information Generate It is characterized by comprising a part and a part.

[0020] Furthermore, the data generation method of the present invention is, for example, A data generation method performed by an information processing device,An image reception step for receiving an image of an object, a first parameter reception step for receiving a first parameter that is a parameter of a real camera when the image of the object is taken, a second parameter reception step for receiving a second parameter that is a parameter of a virtual camera, a three-dimensional information reception step for receiving three-dimensional information of the object, a teacher information storage step for receiving and storing teacher information corresponding to the first parameter, and a spatial configurator learned using at least the image and the first parameter, by inputting the second parameter, a pseudo-image generation step for generating a pseudo-image of the object corresponding to the second parameter, the first parameter, the second parameter, and the three-dimensional information and before The teacher information corresponding to the first parameter Using The teacher information corresponding to the second parameter generate The teacher information to be obtained Generate A step, characterized by comprising the above.

Effect of the Invention

[0021] According to the present invention, even when the camera arrangement or the camera body is changed, as learning data for a learning-based method, an image and teacher information, which are the learning data after the change related to the camera, can be generated from the original learning data, and the labor for collecting the learning data for the learning-based method can be reduced.

[0022] Problems, configurations, and effects other than the above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0023] [Figure 1] It is a functional block diagram showing a schematic configuration of a data generation device according to Example 1. [Figure 2] It is a functional block diagram showing a schematic configuration of a data generation device according to Example 2. [Figure 3] It is a functional block diagram showing a schematic configuration of a data generation device according to Example 3. [Figure 4]This is a functional block diagram showing the schematic configuration of the data generation device in Example 4. [Figure 5] This figure shows an example of a system that performs tasks to which the data generation apparatus and data generation method of the embodiment are applicable. [Figure 6] This figure shows an example of an image captured by a camera in the system shown in Figure 5. [Figure 7] This diagram illustrates the challenges in the system shown in Figure 5. [Figure 8] This diagram illustrates the challenges in the system shown in Figure 5. [Figure 9] This is an example of an information processing device that constitutes the data generation device of Examples 1 to 4. [Modes for carrying out the invention]

[0024] The embodiments of the present invention will be described below with reference to the drawings. In each figure and each embodiment, the same or similar components are denoted by the same reference numerals, and redundant explanations are omitted.

[0025] First, referring to Figures 5 to 8, we will describe an example of a system that performs the tasks to which the data generation apparatus and data generation method of the embodiment are applicable, and the challenges it presents.

[0026] Figure 5 shows an example of a system that performs tasks to which the data generation apparatus and data generation method of the embodiment are applied.

[0027] Here, we will explain the operation of towing a cage cart 701 in a warehouse or factory using a transport cart 700 as an example, but it is not limited to this. For example, other operations may also be applied, such as when an automated robot estimates the picking area of ​​an object and performs picking to pick up goods in a warehouse.

[0028] As shown in Figure 5, the transport trolley 700 has a camera 702 for photographing the cage cart 701 and an end effector 703 for towing the cage cart 701.

[0029] Figure 6 shows an example of an image captured by a camera in the system shown in Figure 5.

[0030] The transport trolley 700 determines the occupied area 707, which is the area used for towing the cart 701, within the captured camera image 706, in order to determine the towing position of the cart 701. Here, the occupied area 707 is the area of ​​interest within the cart 701, which is the object in the camera image 706. This can be the entire object or a part of it, but here we will explain using the case where it is a part of the object as an example.

[0031] One method for determining the towing position is to select a 3D position (e.g., the bottom of the cart) where towing is stable, provided that a sensor is placed separately from the camera 702 to acquire depth information, thereby enabling the acquisition of depth information corresponding to the occupied area 707. For estimating the area used for towing the cart 701, a machine learning-based method using a Deep Neural Network (DNN) is employed. The images used as training data for the DNN and the labeling of the towing area as training information are performed manually, for example, using an annotation tool.

[0032] In towing a cage cart using such a transport trolley 700, as shown in Figure 5, a region estimation DNN is created using training data that includes camera images 706 taken of the cage cart 701 being towed from the position and orientation of the camera 702 assumed by the transport trolley 700, and training information that labels the occupied region 707. Then, using this trained region estimation DNN, the occupied region 707 can be estimated by inputting camera images 706 taken of the cage cart 701 by camera 702 during the actual operation.

[0033] Figures 7 and 8 illustrate the challenges in the system shown in Figure 5.

[0034] As shown in Figure 7, during the development or after the introduction of the transport trolley 700, there may be cases where it is desired to change to a new trolley, such as a heavy-duty trolley 704, which was not initially anticipated during development or introduction, due to changes in operational policies or expansion of the range of items being operated. Here, we illustrate the case where the heavy-duty trolley 704 is similar in shape to the trolley 701 but is larger in size. If the weight of the new heavy-duty trolley 704 is large, or if the shape of the heavy-duty trolley 704 prevents it from being towed by the end effector 703, it may be necessary to change the end effector 703 to a heavy-duty end effector 705 in order to tow the heavy-duty trolley 704, and in conjunction with this change, the placement of the camera 702 may be changed or the camera 702 may be replaced with a different camera.

[0035] Furthermore, as shown in Figure 8, it is conceivable that the original cart 701 can also be transported using the modified heavy-duty end effector 705 and the repositioned camera 702, which were originally designed for the heavy-duty cart 704.

[0036] Furthermore, even if the heavy-duty cage cart 704 or the heavy-duty end effector 705 is not changed, it is possible that the camera 702 may be repositioned or replaced with a different camera during the development or after the introduction of the transport trolley 700.

[0037] Thus, if at least one of the following occurs: a change related to camera 702 (such as a change in the placement of camera 702 or a change to a different camera) or a change in the object, the way the object is depicted will differ between the images used to train the region estimation DNN and the images taken after the change related to camera 702. If the region estimation DNN trained on training data collected before the change is used as input to images in which the object is depicted differently, there is a concern that the accuracy of the occupied region estimation will decrease.

[0038] To avoid failures in estimating the occupied area 707 after changes to camera 702 or the object, one possible method is to re-label the camera image 706 taken of the object and the occupied area 707 after the changes to camera 702 or the object, and then retrain the model. However, this method requires the effort of collecting training data again each time there are changes to camera 702 or the object.

[0039] Therefore, regarding changes to camera 702, the effort required to collect training data can be reduced by applying the data generation apparatus and data generation method described in Examples 1 to 4 below. Furthermore, regarding changes to the target object, the effort required to collect training data can be reduced by applying the data generation apparatus and data generation method described in Example 4 below. [Examples]

[0040] Figure 1 is a functional block diagram showing the schematic configuration of the data generation device in Example 1.

[0041] As shown in Figure 1, the data generation device 100 of this embodiment is configured to include, as its main components, an image receiving unit 104, a first parameter receiving unit 107, a second parameter receiving unit 109, a pseudo-image generation unit 110, a three-dimensional information receiving unit 113, a teacher information storage unit 114, and a teacher information update unit 115. The pseudo-image generation unit 110 also includes a spatial constructor 111.

[0042] Next, the operation of the data generation device 100 and the data generation method will be explained.

[0043] In the image reception step, the image reception unit 104 receives an image 102 of the object 101. Image 102 is at least one image of the object 101 taken by the camera 702. Image 102 may be newly taken, but data collected before any changes to the camera 702 can also be used, for example, for training the region estimation DNN used in the system described in Figure 5. Using collected data can reduce man-hours. In addition to image 102, an additional image 103 may also be input to the image reception unit 104. Image 102 and the additional image 103 are used to create the spatial constructor 111, which will be described later.

[0044] Additional image 103 may be, for example, an image taken by varying the position and orientation of camera 702 based on the position and orientation of camera 702 when each of the images included in image 102 was taken, or an image taken by determining the position and orientation of camera 702 independently of the position and orientation of camera 702 when each of the images included in image 102 was taken. Additional image 103 does not need to have corresponding training information.

[0045] Image 102 and additional image 103 are captured, for example, using a camera 702 on equipment where one or more cameras 702 with known relative position and orientation are fixed, or using a camera on a robot equipped with a camera 702. When using a robot's camera 702, it is sufficient, for example, that the mounting orientation of the camera 702 as seen from the robot's reference coordinate system is known. Alternatively, a person may use the camera 702 to take the images.

[0046] The first parameter receiving unit 107 receives the first parameter 105, which is the parameter of the actual camera when the image 102 of the object 101 was taken, during the first parameter receiving step. The first parameter receiving unit 107 may also receive an additional first parameter 106, which is the parameter of the actual camera when the additional image 103 was taken. Here, "actual camera" means a camera in real space.

[0047] The first parameter 105 and the additional first parameter 106 are parameters that include at least information about the position, orientation, and focal length of the camera 702. The first parameter 105 and the additional first parameter 106 may further include parameters such as the image center and resolution. For the position and orientation of the camera 702, known values ​​are used if, for example, the relative position and orientation at the time of capturing image 102 and additional image 103 are known. If the images are captured using a robot, the change in the camera's position and orientation between captured images can be calculated from the robot's movement information. Alternatively, the camera's position and orientation at the time of capture may be estimated from multiple images using the Structure from Motion (SfM) method. Furthermore, if the camera 702 is equipped with an Internal Measurement Unit (IMU), the values ​​obtained from the IMU may be used for estimation. The camera's focal length and image center may be the design values ​​of the camera 702, or they may be estimated using the SfM method.

[0048] The second parameter receiving unit 109 receives the second parameter 108, which is a parameter of the virtual camera, in the second parameter receiving step. The second parameter 108 of the virtual camera consists of the same parameters as the first parameter 105 and the additional first parameter 106. For example, the second parameter receiving unit 109 receives the modified camera parameters for camera 702 as shown in Figure 8. Here, a virtual camera refers to a camera in the virtual space handled by the data generation device 100.

[0049] The pseudo-image generation unit 110 first creates a spatial constructor 111 in the spatial constructor creation step using image 102, additional image 103, first parameter 105, and additional first parameter 106. The spatial constructor 111 reconstructs the three-dimensional space of the captured environment from the images of image 102 and additional image 103 and the camera-related parameters of first parameter 105 and additional first parameter 106. For the reconstruction of the three-dimensional space, for example, a three-dimensional reconstruction method using the SfM method, which estimates the position and orientation of the camera at the time of shooting from multiple images, and the MVS (Multi View Stereo) method, which reproduces a three-dimensional point cloud of the scene using multiple images, or the NeRF method, which learns a neural network that represents the captured three-dimensional space using images of the object and parameters of the camera that took the picture, and generates a realistic image of the object taken from a camera in a virtual arrangement. In this embodiment, the case using the NeRF method is described, but other methods for three-dimensional reconstruction may also be used. Note that the use of additional image 103 and additional first parameter 106 is optional, and the spatial constructor 111 may be created using image 102 and first parameter 105.

[0050] The pseudo-image generation unit 110 then generates a pseudo-image 112 of the object 101 corresponding to the second parameter 108 by inputting the second parameter 108 into the spatial constructor 111, which has been trained using at least the image 102 and the first parameter 105 in the pseudo-image generation step.

[0051] The pseudo-image 112 is an image output with a virtual resolution by placing a camera with a virtual focal length and image center at the camera position and orientation of the second parameter 108 of the virtual camera. Since the pseudo-image 112 is a virtual image taken with camera 702 at the modified position and orientation of camera 702, as shown in Figure 8, for example, it can be used as training data, and the effort required to collect training data can be reduced. In addition, since the spatial constructor 111 can generate a photorealistic pseudo-image 112, it can reduce the possibility of a decrease in estimation accuracy compared to using non-photorealistic artificial images that reproduce the scene using 3D computer graphics as training data.

[0052] The three-dimensional information receiving unit 113 receives three-dimensional information of the object 101 in the three-dimensional information receiving step. The three-dimensional information can, but is not limited to, a three-dimensional model, a pseudo-depth image, or a depth image.

[0053] The teacher information storage unit 114 receives and stores teacher information corresponding to the first parameter 105 in the teacher information storage step. Teacher information can, but is not limited to, region information or pose information related to the object 101. The teacher information corresponding to the first parameter 105 is associated with the image 102 received by the image receiving unit 104. The teacher information corresponding to the first parameter 105 may be newly created, but it is also possible to use teacher information created before changes to the camera 702 for training the region estimation DNN used in the system described in Figure 5. Using pre-created data reduces man-hours. Furthermore, when using region information as teacher information, it is also possible to use region information that indicates a part of the object 101.

[0054] In the teacher information update step, the teacher information update unit 115 uses the first parameter 105, the second parameter 108 of the virtual camera, and the three-dimensional information of the object 101 to update the teacher information in the teacher information storage unit 114 to the teacher information 116 corresponding to the second parameter 108. Since the teacher information 116 is, for example, the teacher information corresponding to a pseudo-image 112 taken by camera 702 at the changed position and orientation of camera 702 as shown in Figure 8, it can be used as training data, and the effort required to collect training data can be reduced. The teacher information 116 is, for example, the region information of the object 101 corresponding to the pseudo-image 112, but is not limited to this.

[0055] As described above, according to the data generation device 100 and data generation method of this embodiment, even if the camera arrangement or the camera body is changed, images and teacher information, which are modified learning data related to the camera, can be generated from the original learning data as learning data for the learning-based method, thereby reducing the effort required to collect learning data for the learning-based method.

[0056] In the following embodiments, specific configuration examples and functions of the data generation device 100 will be described. [Examples]

[0057] Figure 2 is a functional block diagram showing the schematic configuration of the data generation device in Example 2.

[0058] Example 2 is a more detailed embodiment of Example 1. From Example 2 onward, explanations that overlap with Example 1 will be omitted.

[0059] As shown in Figure 2, this embodiment is a specific example in which the data generation device 100 receives a three-dimensional model 201 of the object 101 as three-dimensional information received by the three-dimensional information receiving unit 113, the training information received by the training information storage unit 114 corresponding to the first parameter 105 as posture information 202 of the object 101, and the training information 116 corresponding to the second parameter 108 as region information of the pseudo-image 112.

[0060] A three-dimensional model 201 of the object 101 can be obtained, for example, by photographing the object 101 using a three-dimensional scanner. The three-dimensional model 201 of the object 101 can be created, for example, by scanning a specific part of the object 101 that is to be designated as the occupied area in image 102, or by scanning the entire object 101. If the three-dimensional model 201 of the object 101 is created by scanning the entire object 101, it may include area information of the object 101 that is to be designated as the occupied area in image 102.

[0061] The posture information 202 can be obtained, for example, by a person using an annotation tool to select two-dimensional points on image 102 that correspond to characteristic three-dimensional points of the three-dimensional model 201 of the object 101, and then solving a PnP (Perspective-n-Point) problem.

[0062] Next, we will explain the process by which the teacher information update unit 115 updates the posture information 202, which is teacher information corresponding to the first parameter 105, to the teacher information 116 corresponding to the second parameter 108.

[0063] Let M be the coordinate system of the 3D model 201 of object 101, Cam_i be the coordinate system of camera 702 (where i is the index corresponding to the image belonging to image 102), VCam be the coordinate system of the virtual camera, and Base be the reference coordinate system. Base is a coordinate system set at a characteristic location in the space where the image was taken, for example, on the floor or desk where object 101 is placed.

[0064] Let T be the information indicating position and orientation. Using a superscript before ^ and a subscript after _, the orientation information 202 corresponding to the first parameter 105 is expressed as Cam_i^T_M, the information indicating the position and orientation of the first parameter 105 is expressed as Cam_i^T_Base, and the information indicating the position and orientation of the second parameter 108 of the virtual camera is expressed as VCam^T_Base. The representations of Cam_i^T_M, Cam_i^T_Base, and VCam^T_Base are, for example, homogeneous transformation matrices of 4x4.

[0065] First, the teacher information update unit 115 uses equation (1) to convert Cam_i^T_M, which is the attitude information 202 corresponding to the first parameter 105 input to the teacher information update unit 115, into VCam^T_M, which is the attitude information expressed in the coordinate system (VCam) of the virtual camera.

[0066]

number

[0067] Next, using L as the region, equation (2) is used to find VCam^L_M, which is the region on the image plane occupied by the three-dimensional model 201 of the object 101 as seen from the virtual camera. This VCam^L_M is output as training information 116 corresponding to the second parameter 108.

[0068]

number

[0069] In equation (2), A represents parameter information, and A_VCam is the information indicating the focal length, image center, and resolution of the second parameter 108 of the virtual camera. The three-dimensional model 201 of the object 101 is denoted as Model_201. projection is a function that represents the three-dimensional model 201 of the object 101 in the coordinate system (VCam) of the virtual camera, and projects the three-dimensional model 201 of the object 101 onto the image plane of the virtual camera using the parameter information A_VCam of the virtual camera and the orientation information VCam^T_M in the coordinate system of the virtual camera obtained in equation (1), thereby generating VCam^L_M, which is the region on the image plane occupied by the three-dimensional model 201 of the object 101 as seen from the virtual camera.

[0070] The region VCam^L_M is, for example, region information of the pseudo-image 112, and is a binary image represented in image format, where the pixel values ​​of the region information part relating to the object 101 are 0 and the other regions are 1.

[0071] Furthermore, if the three-dimensional model 201 of object 101 is generated by scanning the entire object 101 and includes region information of object 101 that you want to occupy in image 102, then when projecting the three-dimensional model 201 of object 101 onto the image plane of the virtual camera using projection, you may project only the relevant region information and generate it as region VCam^L_M.

[0072] Through the above process, the posture information 202, which is the training information corresponding to the first parameter 105, can be updated to the training information 116 corresponding to the second parameter 108, thereby reducing the effort required to collect training data. [Examples]

[0073] Figure 3 is a functional block diagram showing the schematic configuration of the data generation device in Example 3.

[0074] Example 3 is another embodiment that further elaborates on Example 1.

[0075] As shown in Figure 3, in this embodiment, the data generation device 100 is a specific example in which the pseudo-image generation unit 110 inputs a first parameter 105 to the spatial constructor 111 to generate a pseudo-depth image 301 of the object 101 corresponding to the first parameter 105, the three-dimensional information of the object 101 received by the three-dimensional information receiving unit 113 is the pseudo-depth image 301 of the object 101 generated by the spatial constructor 111, the training information storage unit 114 received by the training information storage unit 114 is the training information 302 corresponding to the first parameter 105, and the training information corresponding to the second parameter 108 is the training information of the pseudo-image 112.

[0076] First, the processing of the pseudo-image generation unit 110 will be explained.

[0077] Similar to Embodiment 1, the pseudo-image generation unit 110 inputs the second parameter 108 to the spatial constructor 111 to generate a pseudo-image 112 of the object 101 corresponding to the second parameter 108. This pseudo-image 112 can be used as training data, similar to Embodiment 1.

[0078] In this embodiment, the pseudo-image generation unit 110 generates a pseudo-depth image 301 of the object 101 corresponding to the first parameter 105 by inputting the first parameter 105 to the spatial constructor 111, in addition to the above processing. This pseudo-depth image 301 can be used as three-dimensional information of the object 101 received by the three-dimensional information receiving unit 113. At this time, a pseudo-image 112 of the object 101 corresponding to the first parameter 105 can also be generated, but this is not used in this embodiment.

[0079] If, for example, the NeRF method is used for the spatial constructor 111, the spatial constructor 111 can generate a pseudo-depth image 301 corresponding to the pseudo-image 112 in addition to the pseudo-image 112, and can therefore be used as the spatial constructor 111 in this embodiment.

[0080] The region information 302, which is training information corresponding to the first parameter 105, can be obtained, for example, by a person using an annotation tool to create a mask image of a specific region of the object 101 on the image 102.

[0081] Next, we will explain the process by which the teacher information update unit 115 updates the area information 302, which is teacher information corresponding to the first parameter 105, to the teacher information 116 corresponding to the second parameter 108.

[0082] In addition to the variable representation described in Example 2, information indicating the focal length, image center, and resolution of the first parameter 105 is represented as A_Cam_i. Furthermore, the region information 302 corresponding to the first parameter 105 is represented as L_Cam_i. L_Cam_i is, for example, a binary image represented in image format, where the pixel values ​​of the region information portion related to the object 101 are 0, and the other regions are 1. Also, D is a pseudo-depth image, and the pseudo-depth image 301 corresponding to the first parameter 105, which is used as three-dimensional information of the object 101, is represented as D_Cam_i. Note that the pseudo-depth image 301 corresponding to image 102 can be generated by inputting the first parameter 105 (A_Cam_i and Cam_i^T_Base) into the spatial constructor 111.

[0083] First, the teacher information update unit 115 uses equation (3) to obtain a three-dimensional point cloud (PC_Cam_i) corresponding to the region information 302 for the object 101 corresponding to the first parameter 105. PC is a three-dimensional point cloud. deprojection is a function that obtains the pixel values ​​of the pseudo-depth image 301 (D_Cam_i) corresponding to the region information 302 (L_Cam_i) and calculates the three-dimensional points using the parameter information (A_Cam_i) of the first parameter 105.

[0084]

number

[0085] Next, using equation (4), we find the region VCam^L_PC that the three-dimensional point cloud PC_Cam_i obtained in equation (3) occupies on the image plane when viewed from the virtual camera. projection_PC is a function that converts the three-dimensional point cloud PC_Cam_i into a point cloud viewed from the virtual camera coordinates VCam using (VCam^T_Base)·(Cam_i^T_Base)^(-1), and projects this converted point cloud onto the virtual camera's image plane using the virtual camera's parameter information A_VCam to create the region that the point cloud occupies on the image plane. The region VCam^L_PC obtained in equation (4) can be used as training information 116 corresponding to the second parameter 108, so it may also be output as training information 116 corresponding to the pseudo-image 112.

[0086]

number

[0087] On the other hand, depending on the combination of the virtual camera's second parameter 108, the first parameter 105, and the object 101, the region VCam^L_PC obtained by equation (4) may actually correspond to the shadowed part of the object 101 (the back side of the object 101 as seen from the virtual camera), and may be an area that is not actually visible in the image. Therefore, the following additional processing may also be considered.

[0088] First, the teacher information update unit 115 uses equation (5) to convert the three-dimensional point cloud PC_Cam_i, which corresponds to the region information 302 related to the object 101, into a point cloud PC_VCam, which is represented in virtual camera coordinates.

[0089]

number

[0090] Furthermore, the pseudo-image generation unit 110 uses the spatial constructor 111 to obtain a pseudo-depth image (a pseudo-depth image corresponding to pseudo-image 112) corresponding to the second parameter 108 of the object 101, and inputs it to the training information update unit 115. Specifically, the second parameter 108 (A_VCam and VCam^T_Base) is input to the spatial constructor 111 to generate a pseudo-depth image (D_VCam) corresponding to pseudo-image 112.

[0091] Next, for each pixel representing the region information of region VCam^L_PC, the corresponding pixel value of the pseudo-depth image D_VCam and the z-coordinate value of the three-dimensional point in the point cloud PC_VCam, which is represented by virtual camera coordinates, are obtained. Then, if the obtained pixel value of the pseudo-depth image D_VCam is smaller than the z-coordinate value of the three-dimensional point in the point cloud PC_VCam, the three-dimensional point in the point cloud PC_VCam is considered to be located in the shaded area of ​​object 101, and the corresponding region information is excluded from region VCam^L_PC.

[0092] Finally, a GUI or similar tool is used to verify that the created pseudo-image 112 and the corresponding training information 116 have been correctly generated. By excluding incorrect pseudo-image 112 and the corresponding training information 116 from the training data through verification using a GUI or similar tool, it is possible to prevent mislearning of the relationship between the existing towed object, the trolley car 701, and the training information, thereby preventing a decrease in estimation accuracy regarding the trolley car 701.

[0093] In this embodiment, a pseudo-depth image 301 generated by the spatial constructor 111 was used as the three-dimensional information of the object 101 received by the three-dimensional information receiving unit 113, but it is also possible to use depth information obtained by other methods. For example, if the camera 702 is a camera that can acquire depth images, the depth image obtained by the camera 702 may be used.

[0094] Through the above process, the domain information 302, which is the training information corresponding to the first parameter 105, can be updated to the training information 116 corresponding to the second parameter 108, thereby reducing the effort required to collect training data. [Examples]

[0095] Figure 4 is a functional block diagram showing the schematic configuration of the data generation device in Example 4.

[0096] Example 4 is an example in which an image augmentation function is added to Example 1. While this explanation uses Example 1 as an example, it can also be applied to Example 2 or Example 3.

[0097] As shown in Figure 4, the data generation device 100 of this embodiment has an image augmentation unit 401.

[0098] In the image augmentation step, the image augmentation unit 401 uses a pseudo-image 112 of the object 101 corresponding to the second parameter 108 and training information 116 corresponding to the second parameter 108 (training information 116 corresponding to the pseudo-image 112) to perform image processing on the pseudo-image 112 to generate an augmented pseudo-image 402 that is different from the pseudo-image 112.

[0099] Image processing can include, for example, uniformly changing the pixel values ​​within a region of the pseudo-image 112 corresponding to the region information indicated by the training information 116 corresponding to the second parameter 108, such as changing the hue, saturation, and brightness of the pixel values ​​to a fixed value. Another example of image processing is scaling up or down within a region of the pseudo-image 112 corresponding to the region information indicated by the training information 116 corresponding to the second parameter 108.

[0100] Using this embodiment, for example, if a heavy-duty cage cart 704, as shown in Figure 7, looks similar to an existing towable cage cart 701, it is possible to reduce the effort required to collect training data for the heavy-duty cage cart 704 by generating augmented pseudo-images 402 as training data for the heavy-duty cage cart 704. Furthermore, if applied to an existing towable cage cart 701, it is also possible to reduce the effort required to collect training data for the existing towable cage cart 701. [Examples]

[0101] Figure 9 shows an example of an information processing device that constitutes the data generation device of Examples 1 to 4.

[0102] Example 5 is an example illustrating an example of the hardware configuration of an information processing device (computer) used to realize the data generation device 100 of Examples 1 to 4.

[0103] As shown in Figure 9, the information processing device 1000 of this embodiment includes a processor 1001, a main memory 1002, an auxiliary memory 1003, an input device 1004, an output device 1005, and a communication device 1006. These are connected to each other via communication means such as a bus (not shown).

[0104] Furthermore, the information processing device 1000 may constitute part of another system to which the machine learning system is applied, such as a microcomputer implemented in a robotic arm (manipulator). Alternatively, the information processing device 1000 may be implemented using virtual information processing resources, such as a cloud server provided by a cloud system.

[0105] The processor 1001 is composed of components such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), and an AI chip.

[0106] The main memory 1002 is a device for storing programs and data, and is, for example, ROM (Read Only Memory), RAM (Random Access Memory), or non-volatile memory (NVRAM (Non-Volatile RAM)).

[0107] The auxiliary storage device 1003 is, for example, a hard disk drive, an SSD (Solid State Drive), an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage system, an IC (Integrated Circuit) card, a reader / writer for recording media such as SD cards and optical recording media, or the storage area of ​​a cloud server. Programs and data can be read into the auxiliary storage device 1003 via a recording media reader or a communication device 1006. Programs and data stored in the auxiliary storage device 1003 are read into the main memory 1002 as needed.

[0108] The input device 1004 is an interface that accepts input from an external source, such as a keyboard, mouse, touch panel, card reader, or voice input device.

[0109] The output device 1005 is an interface that outputs various information such as processing progress and processing results. The output device 1005 may be, for example, a display device that visualizes the above information (liquid crystal monitor, LCD (Liquid Crystal Display), graphics card, etc.), a device that converts the above information into sound (speaker, etc.), or a device that converts the above information into text (printer, etc.). Alternatively, for example, the information processing device 1000 may be configured to input and output information to and from other devices via the communication device 1006.

[0110] The communication device 1006 is a device that enables communication with other devices. The communication device 1006 is a wired or wireless communication interface that enables communication with other devices via communication means such as a LAN (Local Area Network), WAN (Wide Area Network), or the Internet, and is, for example, a NIC (Network Interface Card), wireless communication module, USB (Universal Serial Bus) module, or serial communication module.

[0111] The aforementioned image receiving unit 104, first parameter receiving unit 107, second parameter receiving unit 109, pseudo-image generation unit 110, three-dimensional information receiving unit 113, teacher information storage unit 114, teacher information update unit 115, and image augmentation unit 401 are realized, for example, by the processor 1001 of the information processing device 1000 reading and executing a program stored in the main memory 1002.

[0112] Furthermore, the main memory 1002 and auxiliary memory 1003 store the image 102 and additional image 103, the first parameter 105 and additional first parameter 106, the second parameter 108, various parameters that define the spatial constructor 111 of the pseudo-image generation unit 110, the pseudo-image 112 and pseudo-depth image 301, the three-dimensional model 201 of the object 101, posture information 202 and region information 302, augmented pseudo-image 402, and training information 116 corresponding to the second parameter 108.

[0113] Each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each of these functions.

[0114] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs, or other storage devices, or in recording media such as IC cards, SD cards, or DVDs.

[0115] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, although the above embodiments were described in relation to a cart, other objects may also be used. Furthermore, the above embodiments are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. It is also possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. [Explanation of Symbols]

[0116] 100...Data generation device 101...Target object 102...Image 103... Additional images 104...Image Reception Department 105...First parameter 106... Additional first parameter 107...First Parameter Reception Department 108...Second parameter 109...Second Parameter Reception Section 110...Pseudo image generation unit 111...Space constructor 112... Pseudo-image 113...3D Information Reception Department 114...Teacher Information Storage Unit 115...Teacher information update department 116... Teacher information corresponding to the second parameter 201... Three-dimensional model of the object 202... Posture information 301...Pseudo depth image 302...Area information 401...Image padding section 402...Inflated / fake images 700... Transport cart 701... Cart 702...Camera 703... End Effector 704... Cart for heavy loads 705... End effector for heavy loads 706...Camera image 707...Occupied area 1000... Information Processing Device 1001 Processor 1002...Main memory 1003...Auxiliary storage device 1004...Input device 1005...Output device 1006...Communication equipment

Claims

1. An image receiving unit that receives images of the object, A first parameter receiving unit that receives first parameters which are the parameters of the actual camera when the image of the object was taken, A second parameter receiving unit that receives the second parameter, which is a parameter of the virtual camera, A three-dimensional information receiving unit that receives three-dimensional information of the aforementioned object, A teacher information storage unit that receives and stores teacher information corresponding to the first parameter, A pseudo-image generation unit generates a pseudo-image of the object corresponding to the second parameter by inputting the second parameter into a spatial constructor that has been trained using at least the image and the first parameter, A data generation device comprising: a teacher information generation unit that generates teacher information corresponding to the second parameter using the first parameter, the second parameter, the three-dimensional information, and the teacher information corresponding to the first parameter in the teacher information storage unit.

2. In claim 1, The three-dimensional information of the object is a three-dimensional model of the object, The training information corresponding to the first parameter is the posture information of the object. A data generation device characterized in that the training information corresponding to the second parameter is the region information of the pseudo-image.

3. In claim 1, The pseudo-image generation unit generates a pseudo-depth image of the object corresponding to the first parameter by inputting the first parameter to the spatial constructor. The three-dimensional information of the object is the pseudo-depth image of the object, The training information corresponding to the first parameter is the region information of the object. A data generation device characterized in that the training information corresponding to the second parameter is the region information of the pseudo-image.

4. In claim 1, A data generation device characterized by having an image augmentation unit that generates an augmented pseudo-image by performing image processing on the pseudo-image of the object corresponding to the second parameter and training information corresponding to the second parameter.

5. In claim 1, The data generation device is characterized in that the first parameter and the second parameter each include at least information about the position, orientation, and focal length of the actual camera or the virtual camera.

6. In claim 1, The spatial constructor is a data generation device characterized by generating realistic pseudo-images.

7. In claim 1, A data generation device characterized in that the training information corresponding to the second parameter is region information indicating a part of the object in the pseudo-image.

8. In claim 1, The data generation device is characterized in that the image is an image taken using equipment on which one or more cameras whose relative position and orientation are known are fixed, or a robot having a camera, wherein the mounting orientation of the camera as seen from the robot's reference coordinate system is known.

9. A data generation method performed by an information processing device, An image reception step that receives an image of the object, A first parameter receiving step that receives a first parameter which is a parameter of the actual camera when the image of the object was taken, A second parameter reception step that accepts a second parameter, which is a parameter of the virtual camera, A three-dimensional information receiving step that receives three-dimensional information of the object, A teacher information storage step that receives and stores teacher information corresponding to the first parameter, A pseudo-image generation step involves inputting the second parameter into a spatial constructor that has been trained using at least the image and the first parameter, thereby generating a pseudo-image of the object corresponding to the second parameter. A data generation method characterized by comprising: a teacher information generation step of generating teacher information corresponding to the second parameter using the first parameter, the second parameter, the three-dimensional information, and the teacher information corresponding to the first parameter.

Images