A synthetic data template library generation method for object pose estimation

Abstract

The application provides a kind of synthesis data template library generation method for object pose estimation, first based on Unity3D engine builds a synthesis scene for template library generation, mainly including light setting and the script binding of camera frame-by-frame shooting;The method for generating camera viewpoint based on Fibonacci grid, while based on the improved Unity Perception modifies the rotation angle of camera to realize the shooting of virtual camera;Target area acquisition algorithm based on dynamic amplification and region clipping, introduces amplification coefficient, can meet the clipping of different size target objects, ensure that the clipping area can completely contain target objects, as far as possible to reduce the interference produced by background and non-target objects.The application is different from the existing template library generation method, only uses the image of real scene without using the related annotation of real object, at the same time, the application is more friendly to the user who cannot obtain the pose information of real object due to the limitation of acquisition device.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a synthetic data template library generation method for object pose estimation. BACKGROUND

[0002] Object pose estimation is a hot research problem in the field of computer vision, and can be mainly divided into key point-based object pose estimation, coordinate-based object pose estimation and template-based object pose estimation according to the implementation mode. Object pose estimation has a wide range of applications in robot grasping, general virtual reality and other fields. For example, in the industry, a mechanical arm can grasp a target object by predicting the pose information of an industrial part; in the field of general virtual reality, a virtual model can be superimposed according to the pose information of a real object.

[0003] Common target objects such as screws and wrenches are weak texture objects, and the key point-based and coordinate-based object pose estimation cannot accurately predict the pose information of weak texture objects. Therefore, the template-based pose estimation method is often used to predict the pose of weak texture objects. The generation of the template library is crucial and can directly affect the accuracy of the pose estimation of weak texture objects. Therefore, a method is needed to generate an effective template library to better solve the pose estimation task of weak texture objects.

[0004] The generation of the template library mainly includes three steps: target image generation, target template generation and image-template matching. In the target image generation, the existing method is implemented according to the pose information of a real image and a target detection algorithm; in the target template generation, the existing method is implemented by recursively segmenting a icosahedron based on rendering, replacing each triangle of the icosahedron, which is the largest convex polyhedron, with four almost equilateral triangles and performing multiple iterations to select the camera viewpoint, and sampling the 3D model. In the image-template matching, the matching of the target image and the target template is implemented by using a nearest neighbor algorithm for processing. The existing method uses a lot of data labeling of real images, and in the case of limited devices, it is impossible to obtain real pose information.

[0005] At present, the template-based object pose estimation algorithm is relatively mature, but there is less research on the method of generating the template library. Most of the methods use the data labeled according to the real object pose information to generate the template library. However, in the case of limited devices, it is impossible to obtain the labeled data of real objects, so the existing target image generation method and image-template matching cannot meet the demand. For the generation of the template, the existing scheme mostly depends on the rendering platform, selects the camera viewpoint through rendering, and constantly modifies the size of the polyhedron to generate templates of different scales, which takes a long time and is relatively complex. SUMMARY

[0006] To address the aforementioned issues, this invention provides a method for generating a synthetic data template library oriented towards object pose estimation. This method is a novel synthetic data template library generation method based on the Unity3D engine. It does not rely on the pose information annotation of real objects, and by improving Unity Perception, it can quickly achieve the positioning of the camera viewpoint to generate the template library.

[0007] A method for generating a synthetic data template library for object pose estimation includes:

[0008] Step 1: Construct a synthetic scene using the LineMOD Occlusion dataset; including:

[0009] Step 1.1: Extract texture information of the target object's 3D model from the LineMOD Occlusion dataset using MeshLab;

[0010] Step 1.2: Import the texture information and the 3D model together into Unity to obtain the target object 3D model with texture information;

[0011] Step 1.3: Using the target object's 3D model as the center, set up 8 linear light sources to illuminate the target object's 3D model;

[0012] Step 1.4: Use the script of Perception Camera in Unity Perception to bind the camera and complete the construction of the composite scene;

[0013] Step 2: Generate the target image and target template using a camera viewpoint generation method based on Fibonacci grids; including:

[0014] Step 2.1: Place the 3D model of the target object into the newly created blank scene, and adjust the position of the camera and the 3D model of the target object so that the camera is directly above the target object;

[0015] Step 2.2: Calculate the sampling points on the surface of the sphere based on the Fibonacci grid. The specific formula is as follows:

[0016]

[0017] Where n is the number of sampling points, N is the total number of sampling points, and x n y n z n These are the coordinate axes corresponding to the nth sampling point. R is the radius of the sphere to be sampled;

[0018] Step 2.2: Read the virtual camera position in each frame and change the virtual camera position to the sampling point position calculated in Step 2.2;

[0019] Step 2.3: Set the virtual camera rotation angle; specifically, obtain the coordinates of the virtual camera and the target object 3D model, and calculate the vector pointing from the virtual camera to the target object 3D model; in Unity, the virtual camera's field of view points to its Z-axis direction, so by calling the function transform.forward, which points forward in the world coordinate system, we ensure that the virtual camera always points to the target object 3D model, thus setting the camera rotation angle.

[0020] Step 2.4: Using the camera viewpoint obtained in Step 2.2, capture images of the model from different angles and save them as the target template and the target image, respectively; specifically:

[0021] Step 2.4.1: Target template generation. Using semantic segmentation annotation and object detection annotation in Unity Perception as annotators, add label configuration fields to the 3D model, obtain the 2D bounding box and mask annotation of the target object, set the number of iterations, generate the label of the target template and store it.

[0022] Step 2.4.2: Target image generation. Import a model that can be placed on the target object (such as a table model) and place it directly below the target object. Use texture randomization to set the newly imported model (such as a table model). Configure it according to step 2.4.1 to obtain the label of the target image and store it.

[0023] Step 3: Obtain the target region based on a dynamically scaled region cropping algorithm; specifically including:

[0024] Step 3.1: Transform the coordinates of the target region; including:

[0025] Step 3.1.1: Extract the data of the target template and target image, including the center coordinates of the object, the size of the generated bounding box, and the name of the object;

[0026] Step 3.1.2: Process the extracted data and convert it into a rectangular box that OpenCV can process for subsequent image cropping;

[0027] Step 3.1.3: Calculate the starting and ending coordinates of the trimming area:

[0028]

[0029] Among them, (x start ,y start (x) represents the starting coordinates of the clipping region. end ,y end ) represents the ending coordinates of the clipping region, w represents the width of the extracted target object bounding box, and h represents the length of the extracted target object bounding box;

[0030] Step 3.2: Dynamically zoom in on the cropped area:

[0031]

[0032]

[0033] Among them, (x start ,y start (x) represents the starting coordinates of the cropped area after dynamic zoom-in. end ,y end ) represents the ending coordinates of the cropped region after dynamic zoom-in, w represents the width of the extracted target object bounding box, h represents the length of the extracted target object bounding box, and margin... w To increase the width of one side of the initial cropped area by a number of pixels, margin h To increase the pixel value by the length of one side of the initial cropping region, w max h is the width of the bounding box of the 3D model of the largest object. max τ is the length of the bounding box of the 3D model of the largest object, and τ is the magnification factor.

[0034] Step 3.3: Crop and save the cropping region obtained in Step 3.2. Specifically, call the OpenCV image reading function `imread` to read the (x) region from Step 3.2. start ,y start ) and (x end ,y end The image is cropped by forming an array, and finally the image writing function imwrite in OpenCV is called to write the cropped image to a folder to obtain the final target object region.

[0035] Step 4: Write the path name and file name of the target image and target template obtained in Step 3.3 into a .txt file in sequence and save it to form an image-template matching and generate the final template library.

[0036] The beneficial effects of this invention are:

[0037] This invention proposes a synthetic data template library generation method for object pose estimation. In target image generation, only images of the real scene are needed, without requiring the pose information of the real objects. Even with limited equipment, target images can still be generated using this method. In target template generation, the camera viewpoint is selected based on a Fibonacci grid, ensuring both uniform selection of the camera viewpoint and the speed of template generation; generating 1500 images takes only about 2 minutes. Regarding the image-template matching process, this invention uses the same synthetic scene in both template generation and image generation, thus eliminating the need for additional matching using nearest neighbor algorithms; image-template matching can be completed simply by following the generation order. Attached Figure Description

[0038] Figure 1 This is a flowchart of a method for generating a synthetic data template library for object pose estimation in this invention.

[0039] Figure 2 A schematic diagram of the scenario of this invention is provided, in which the positions of the eight bounding box corners represent the positions of the linear light source;

[0040] Figure 3 This is a diagram showing the implementation of camera viewpoint selection in this invention. The object in the middle is the 3D model of the target object, and the other objects are preview images of the camera viewpoint. Taking Ape as an example.

[0041] Figure 4 This is an example of an image-template matching graph generated by the present invention, wherein the first row is the target image and the second row is the target template, taking Ape as an example. Detailed Implementation

[0042] The invention will be further explained below with reference to the accompanying drawings and specific implementation examples.

[0043] This invention provides a novel method for generating synthetic data template libraries for object pose estimation, developed using the Unity3D engine. This method utilizes only real-world scene images without requiring annotations of real objects. It acquires texture information using MeshLab and imports it into the Unity3D engine, incorporating a Fibonacci mesh for camera viewpoint selection. Unity Perception, a perception toolkit, is used to obtain the target images and corresponding templates from the synthetic data. A dynamic zoom and region cropping method is provided to remove background and non-target object interference from the target images and templates, using the final cropped and matched images and templates as the final generated template library. The generated template library provides strong support for subsequent template-based object pose estimation network training. This invention provides 1500 image-template pairs for each object type, including 1500 target images and 1500 target templates.

[0044] like Figure 1 As shown, a method for generating a synthetic data template library for object pose estimation includes:

[0045] Synthetic Scene Construction: Synthetic data has become a new research hotspot in recent years. This invention uses Unity, MeshLab, and texture randomization techniques to create synthetic data. To better serve the object pose estimation task, the LineMOD Occlusion dataset, commonly used in object pose estimation, was used for constructing the synthetic scene. This dataset contains eight types of objects, labeled as follows: glue, eggbox, duck, cat, driller, can, ape, and holepunch.

[0046] The LineMOD Occlusion dataset provides .obj files of 3D models. However, these files lack texture data during Unity import, resulting in the imported model lacking the texture of the target object. Therefore, this invention first extracts texture information from the 3D model using MeshLab. Specifically, this involves importing the .obj file of the 3D model into MeshLab, using a point cloud normal vector calculation tool to obtain point cloud normal vectors, outputting a mesh model based on the Poisson reconstruction method, calling the parametric mesh model interface, and exporting the texture information. Importing the texture information along with the 3D model into Unity yields a 3D model of the target object with texture information.

[0047] The construction of the composite scene mainly includes the setting of virtual lighting and virtual cameras. In this invention, in order to preserve sufficient features of the target model, eight linear light sources are set around the target 3D model to illuminate it, such as... Figure 2 As shown. To achieve frame-by-frame image acquisition of the 3D model, the Perception Camera virtual perception camera in Unity Perception is used to bind the camera.

[0048] Step 1: Construct a synthetic scene using the LineMOD Occlusion dataset; including:

[0049] Step 1.1: Extract texture information of the target object's 3D model from the LineMOD Occlusion dataset using MeshLab;

[0050] Step 1.2: Import the texture information and the 3D model together into Unity to obtain the target object 3D model with texture information;

[0051] Step 1.3: Using the target object's 3D model as the center, set up 8 linear light sources to illuminate the target object's 3D model;

[0052] Step 1.4: Use the script of Perception Camera in Unity Perception to bind the camera and complete the construction of the composite scene;

[0053] This invention provides a camera viewpoint generation method based on a Fibonacci grid. The method involves uniformly sampling Fibonacci grid points on a sphere, using these sampling points as camera positions, and modifying the camera rotation angle to generate the camera viewpoint. Furthermore, it utilizes Unity Perception to store the image information and annotation information of the target image and target template.

[0054] Place the imported 3D model of the target object from step 1.1 into a newly created blank scene. Set the skybox material in the new scene to None and adjust the background color to black, so that the model is in a completely black environment in the Game view, which facilitates subsequent template generation. To ensure that the template is clearly and accurately displayed in the scene, this invention uses 8 simulated parallel lights to provide illumination for the target 3D model. Since the shadows cast by the simulated object when it encounters light can interfere with the training of the subsequent template-based pose estimation network, the shadow type needs to be set to No Shadows. Finally, adjust the camera position and the position of the target 3D model so that the camera is directly above the target object to complete the scene construction.

[0055] Camera viewpoint generation based on Fibonacci grid: Compared to latitude and longitude-based sampling of spherical surface points, Fibonacci grids can sample spherical surface points more uniformly, resulting in a more comprehensive range of viewpoints for the template. This invention uses Fibonacci grids to sample spherical surface points, using these uniformly sampled points as the viewpoints of a virtual camera to acquire images from different perspectives. A schematic diagram of the camera position is shown below. Figure 3 As shown.

[0056] Step 2: Generate the target image and target template using a camera viewpoint generation method based on Fibonacci grids; including:

[0057] Step 2.1: Place the 3D model of the target object into the newly created blank scene, and adjust the position of the camera and the 3D model of the target object so that the camera is directly above the target object;

[0058] Step 2.2: Calculate the sampling points on the surface of the sphere based on the Fibonacci grid. The specific formula is as follows:

[0059]

[0060] Where n is the number of sampling points, N is the total number of sampling points, and x n y n z n These are the coordinate axes corresponding to the nth sampling point. R is the radius of the sphere to be sampled, and in this invention, it is the distance between the virtual camera and the target 3D model;

[0061] Step 2.2: Read the virtual camera position in each frame and change the virtual camera position to the sampling point position calculated in Step 2.2;

[0062] Step 2.3: Set the virtual camera rotation angle; specifically, obtain the coordinates of the virtual camera and the target object 3D model, and calculate the vector pointing from the virtual camera to the target object 3D model; in Unity, the virtual camera's field of view points to its Z-axis direction, so by calling the function transform.forward, which points forward in the world coordinate system, we ensure that the virtual camera always points to the target object 3D model, thus setting the camera rotation angle.

[0063] Step 2.4: Using the camera viewpoint obtained in Step 2.2, capture images of the model from different angles and save them as the target template and the target image, respectively; specifically:

[0064] Step 2.4.1: Target template generation. Using semantic segmentation annotation and object detection annotation in Unity Perception as annotators, add label configuration fields to the 3D model imported in Step 1.1, obtain the 2D bounding box and mask annotation of the target object, set the iteration number to 1500, generate the label of the target template and store it in a JSON file;

[0065] Step 2.4.2: Target Image Generation. Import a model that can place the target object (e.g., a table model) directly below the target object. Utilize texture randomization, i.e., by calling Unity Perception tools (Unity Perception package) to change the texture of the newly imported model (e.g., the table model), thereby changing the environment in which the target image is located. Then, configure according to Step 2.4.1 to obtain the target image's label and store it in a JSON file.

[0066] This invention provides a region cropping algorithm based on dynamic magnification to obtain the target region. Using this algorithm, it can be ensured that the cropped image can contain the complete target object and that the features of the target object can be effectively extracted, avoiding the failure of subsequent pose estimation network training due to the target object being too small.

[0067] Step 3: Obtain the target region based on a dynamically scaled region cropping algorithm; specifically including:

[0068] Step 3.1: Transform the coordinates of the target region; including:

[0069] Step 3.1.1: Extract the JSON files of the target template and target image respectively and extract the data. The specific data to be extracted includes the center coordinates of the object, the size of the generated bounding box, and the name of the object;

[0070] Step 3.1.2: Process the extracted data and convert it into a rectangular box that OpenCV can process for subsequent image cropping;

[0071] Step 3.1.3: OpenCV needs to obtain the starting and ending coordinates of the clipping region, that is, the coordinates of the top left and bottom right corners of the target object region;

[0072] Calculate the starting and ending coordinates of the clipping region:

[0073]

[0074] Among them, (x start ,y start (x) represents the starting coordinates of the clipping region. end ,y end) represents the ending coordinates of the clipping region, w represents the width of the extracted target object bounding box, and h represents the length of the extracted target object bounding box;

[0075] Since there are 8 target objects in the synthesized image, each with a different size, if a fixed-size cropping region is used for some of the larger objects, incomplete cropping may occur. Therefore, this invention selects the size of the bounding box of the 3D model of the largest object as the size of the cropping region. Considering the versatility of this method, this invention also introduces a magnification factor τ, which can further expand the cropping region according to actual needs.

[0076] Step 3.2: Dynamically zoom in on the cropped area:

[0077]

[0078]

[0079] Among them, (x start ,y start (x) represents the starting coordinates of the cropped area after dynamic zoom-in. end ,y end ) represents the ending coordinates of the cropped region after dynamic zoom-in, w represents the width of the extracted target object bounding box, h represents the length of the extracted target object bounding box, and margin... w To increase the width of one side of the initial cropped area by a number of pixels, margin h To increase the pixel value by the length of one side of the initial cropping region, w max h is the width of the bounding box of the 3D model of the largest object. max τ is the length of the bounding box of the 3D model of the largest object, and τ is the magnification factor used to dynamically adjust the clipping area, making the method universal.

[0080] Step 3.3: Crop and save the cropping region obtained in Step 3.2. Specifically, call the OpenCV image reading function `imread` to read the (x) region from Step 3.2. start ,y start ) and (x end ,y end The image is cropped by forming an array. Finally, the image is written to a folder using the OpenCV image writing function `imwrite`. The final target object region is then obtained, and the generated result is as follows. Figure 4 As shown;

[0081] Step 4: Write the path name and file name of the target image and target template obtained in Step 3.3 into a .txt file in sequence and save it to form an image-template matching and generate the final template library.

[0082] This invention provides a template library generation method for pose estimation tasks. First, a composite scene for template library generation is built based on the Unity3D engine, mainly including lighting settings and script binding for frame-by-frame camera capture. A method for generating the camera viewpoint based on a Fibonacci mesh is used, while a virtual camera capture is achieved by modifying the camera's rotation angle using an improved Unity Perception. A target region acquisition algorithm based on dynamic zoom and region clipping is introduced, incorporating a zoom factor to accommodate target objects of different sizes, ensuring the clipped region completely contains the target object while minimizing interference from the background and non-target objects. This invention differs from existing template library generation methods by using only images of the real scene without requiring annotations of real objects. Furthermore, this invention is more user-friendly for users who cannot obtain real object pose information due to limitations in their acquisition equipment.

Claims

1. A method for generating a synthetic data template library for object pose estimation, characterized in that, include: Step 1: Construct a synthetic scene using the LineMOD Occlusion dataset; Step 2: Generate the target image and target template using a camera viewpoint generation method based on Fibonacci grids; Step 2.1: Place the 3D model of the target object into the newly created blank scene, and adjust the position of the camera and the 3D model of the target object so that the camera is directly above the target object; Step 2.2: Calculate the sampling points on the surface of the sphere based on the Fibonacci grid. The specific formula is as follows: ； in, For sampling points, The total number of sampling points. The first The corresponding coordinate axes of each sampling point , To sample the radius of the sphere; Step 2.2: Read the virtual camera position in each frame and change the virtual camera position to the sampling point position calculated in Step 2.2; Step 2.3: Set the virtual camera rotation angle; specifically, obtain the coordinates of the virtual camera and the target object 3D model, and calculate the vector pointing from the virtual camera to the target object 3D model; in Unity, the virtual camera's field of view points to its Z-axis direction, so by calling the function transform.forward, which points forward in the world coordinate system, we ensure that the virtual camera always points to the target object 3D model, thus setting the camera rotation angle. Step 2.4: Use the camera view obtained in Step 2.2 to capture images of the model from different angles, and save them as the target template and the target image respectively; Step 2.4.1: Target template generation. Using semantic segmentation annotation and object detection annotation in Unity Perception as annotators, add label configuration fields to the 3D model, obtain the 2D bounding box and mask annotation of the target object, set the number of iterations, generate the label of the target template and store it. Step 2.4.2: Target image generation. Import a model that can be placed under the target object and set it up using texture randomization. Configure it according to step 2.4.1 to obtain the label of the target image and store it. Step 3: Obtain the target region based on a dynamically scaled region cropping algorithm; Step 4: Write the path name and file name of the acquired target image and target template into a .txt file in sequence and save it to form an image-template matching and generate the final template library.

2. The method for generating a synthetic data template library for object pose estimation according to claim 1, characterized in that, Step 1 includes: Step 1.1: Extract texture information of the target object's 3D model from the LineMOD Occlusion dataset using MeshLab; Step 1.2: Import the texture information and the 3D model together into Unity to obtain the target object 3D model with texture information; Step 1.3: Using the target object's 3D model as the center, set up 8 linear light sources to illuminate the target object's 3D model; Step 1.4: Use the script of Perception Camera in Unity Perception to bind the camera and complete the construction of the composite scene.

3. The method for generating a synthetic data template library for object pose estimation according to claim 1, characterized in that, Step 3 includes: Step 3.1: Transform the coordinates of the target region; Step 3.2: Dynamically zoom in on the cropped area: ； ； in, These are the starting coordinates of the cropped area after dynamic zoom-in. These are the end coordinates of the cropped area after dynamic zoom-in. The width of the bounding box of the extracted target object. The length of the bounding box of the extracted target object. To increase the pixel value by one side width of the initial cropped area, To increase the pixel value by the length of one side of the initial cropping region, This is the width of the bounding box of the 3D model of the largest object. This is the length of the bounding box of the 3D model of the largest object. This is the magnification factor; Step 3.3: Crop and save the cropping region obtained in Step 3.

2. Specifically, call the OpenCV image reading function `imread` to read the image from Step 3.

2. and An array is formed to perform cropping. Finally, the image is written to a folder using the OpenCV image writing function `imwrite`, and the final target object region is obtained.

4. The method for generating a synthetic data template library for object pose estimation according to claim 3, characterized in that, Step 3.1 includes: Step 3.1.1: Extract the data of the target template and target image, including the center coordinates of the object, the size of the generated bounding box, and the name of the object; Step 3.1.2: Process the extracted data and convert it into a rectangular box that OpenCV can process for subsequent image cropping; Step 3.1.3: Calculate the starting and ending coordinates of the trimming area: ； in, The starting coordinates of the clipping region. The coordinates of the end of the clipping region. The width of the bounding box of the extracted target object. This is the length of the bounding box of the extracted target object.

Images