Data set forming method for target detection

Abstract

The invention provides a data set forming method for target detection. The data set forming method comprises four main steps of building a background image library and a three-dimensional model library, estimating three-dimensional attitude information of a to-be-detected target, reconstructing a scene, projecting the target, rendering data and generating labels. In the background image library establishment stage, internal and external parameters and illumination information of a camera are collected. When three-dimensional attitude estimation is carried out, prediction is carried out by using background images, camera parameters, target position coordinates and point cloud data. And then reconstructing a scene in three-dimensional modeling software, and projecting a target. And finally, rendering the scene, and generating a data set and a label. According to the method, three-dimensional modeling software is used for synthesizing data, and the three-dimensional attitude in the scene is estimated, so that the problems of low authenticity of synthesized data, high label cost, difficulty in obtaining real data of a specific scene and the like are solved, the accuracy and stability of target detection are greatly improved, the label cost is reduced, the specific scene can be simulated, and the target detection efficiency is improved. The problem of data acquisition is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for synthesizing target detection data. Background Art

[0002] Object detection is an important task in the field of computer vision, which is used to identify and locate specific objects in images or videos. In the process of developing and optimizing object detection algorithms, a large amount of labeled data is essential. These labeled data are usually manually labeled by human annotators, which is a time-consuming, labor-intensive and costly process. To solve this problem, many researchers have begun to explore the use of synthetic data to train object detection algorithms to reduce their reliance on real labeled data.

[0003] Synthetic data is an image or video that simulates a real scene by generating a virtual image or video on a computer. There are two main ways to create synthetic data, one is through a mapping scheme, and the other is to use 3D software to render the model. However, both of these methods of object detection data synthesis have some limitations and challenges.

[0004] Patent CN111091167 pastes a pre-prepared target image onto a background image. The problem with this method is that it often cannot accurately simulate effects such as lighting, shadows, and the three-dimensional perspective relationship between the target and the scene, so the generated synthetic image often has a significant gap with the real scene.

[0005] Patent CN110084304 uses 3D software rendering to generate more realistic synthetic images through model rendering. However, this method requires a lot of manual input, such as setting the model's position, direction, lighting conditions, and rendering parameters, which makes this method inefficient and difficult to achieve large-scale data synthesis. Therefore, finding a new target detection data synthesis method that can overcome these problems has become an important demand. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a data set synthesis method for target detection. The present invention uses the background image, the point cloud data of the target to be detected, the internal and external parameters of the camera, and the position coordinates of the target to be detected in the image to predict the pose and scaling of the model through a three-dimensional pose estimation model that can handle multiple inputs, and then performs modeling and rendering through three-dimensional modeling software, which solves the problems of authenticity of synthetic data, high cost of data annotation, and difficulty in obtaining real data in specific scenarios. The specific implementation steps are as follows: Step S10: Establish a background image library and a 3D model library. First, collect 2D images of the actual environment, confirm the internal and external parameters of the camera and the background lighting information, and create a background image library. At the same time, construct a 3D model of the pre-detection target to create a 3D model library.

[0007] Step S101: Create a background image library. Collect two-dimensional images in the actual environment as backgrounds, and collect and record parameter information related to these background images. The specific steps are as follows: First, we use a high-resolution camera to photograph a calibration plate of known size and shape. By taking photos of the calibration plate from different angles and positions, the camera is calibrated to obtain the camera's internal parameters.

[0008] Next, we use the calibrated camera to shoot in various environments and lighting conditions to obtain as rich and diverse background scenes as possible. During the shooting process, we also place a calibration plate of known size and shape in the camera's field of view and take a photo of this calibration plate. By identifying the feature points in the photo, we can establish and solve mathematical equations containing external parameters to calibrate the camera's external parameters. These external parameters describe the position and orientation of the camera coordinate system relative to the world coordinate system.

[0009] During the shooting process, we also record the type, position, direction and intensity of the light sources in the scene in detail. We first determine and record the type of light source (such as natural light or artificial light), then accurately measure and record the position of the light source through measurement tools or lighting models, then estimate the direction of the light source by observing the affected area of ​​the light and the direction of the shadows, and finally use a photometer or other light measurement device to measure and record the intensity of the light source.

[0010] Finally, we save the captured background images and the collected information in the background image library so that we can simulate and render the scene more accurately in the subsequent steps.

[0011] Step S102: Create a 3D model library. Construct a 3D model of the target to be detected and create a 3D model library. There are two main ways to create a 3D model. One is to create it from scratch using specialized 3D modeling software. The other is to use a 3D scanner to convert real-world objects into digital models.

[0012] Step S20: Confirm the 3D pose information of the target to be detected. We use a multi-input deep neural network model, namely the 3D pose estimation model, to predict the 3D pose of the target to be detected in the current background image. The model receives the background image, the point cloud data of the target to be detected, the internal and external parameters of the camera, and the position coordinates of the target to be detected in the image as input, and then extracts features through the image processing subnetwork, the point cloud processing subnetwork and the camera parameter processing subnetwork respectively and splices these features together, and then further extracts the high-dimensional information therein, and finally outputs the confidence of the target to be detected on the background image, as well as the 3D rotation information and scaling ratio of the model.

[0013] Step S201: Background image processing subnetwork. The subnetwork extracts high-level features from the background image through a convolutional neural network (CNN). The choice of CNN can include models such as ResNet and VGG. First, the background image needs to be preprocessed to a suitable size. After feature extraction by CNN, we can get a one-dimensional feature vector.

[0014] Step S202: Point cloud processing subnetwork. The subnetwork uses PointNet to extract information from three-dimensional point cloud data. First, the point cloud data is preprocessed, including centering (making all points surround the origin) and normalization (ensuring that all points are within the unit sphere). Then, PointNet is used to extract features to obtain a one-dimensional feature vector. PointNet is a neural network designed for processing unordered point cloud data, which can directly extract global features from point clouds.

[0015] Step S203: Camera parameter and coordinate processing subnetwork. The subnetwork uses a fully connected network (FCN) to process the camera's internal and external parameters and points in the camera coordinate system. For the camera's internal and external parameters, we can directly normalize them to the range of [-1, 1] and input them into the FCN. For the position coordinates of the target to be detected in the image, we convert the coordinates into coordinates with the center of the image as the origin, convert and normalize them to the range of [-1, 1] and input them into the FCN. The FCN can be composed of three fully connected layers. Finally, the output of the FCN is a one-dimensional feature vector.

[0016] Step S204: Feature fusion and output. We aggregate the outputs of the three sub-networks, and then output the confidence, the three-dimensional rotation information of the model, and the scaling ratio of the model through two fully connected layers. The number of nodes in these two fully connected layers can be set to 512 and 5, where the 5 output nodes correspond to the confidence, the three-dimensional rotation information of the model, and the scaling ratio of the model.

[0017] Step S30: Scene reconstruction and target projection. In this step, the corresponding scene is first established in the 3D modeling software using the background image and related parameters. Then, the target to be detected is projected into a 2D image in the scene using the information predicted by the model in step S20.

[0018] Step S301: scene reconstruction: in the 3D modeling software, the camera position and viewing angle are reconstructed by using the camera parameters, and the scene is re-illuminated by using the recorded light source information.

[0019] Step S302: Target projection. In this step, we first use the known camera information and the position coordinates of the target to be detected in the image to project the target into the reconstructed two-dimensional scene image. Then, we use the information predicted by the model in step S20 to adjust the three-dimensional rotation information and scaling of the target to obtain more realistic data.

[0020] Step S40: Rendering data and generating labels. Use 3D modeling software to render the reconstructed scene with the detection target, obtain a 2D image and label with the target, and obtain label data.

[0021] Step S401: Rendering a synthetic image. Using the scene pre-configured in step S30, a synthetic image is generated by rendering through a 3D modeling software. During the rendering process, a synthetic image with a target is generated according to different combinations of targets and scenes.

[0022] Step S402: Rendering the target image. Rendering is performed using 3D modeling software for the target to be detected in the scene. During the rendering process, only the target to be detected is rendered, and the background is set to be transparent.

[0023] Step S403: Obtain detection frame information. The rendered image is processed to obtain the detection frame coordinates of the target. The category of the rendered target is combined with the detection frame information to form complete annotation information.

[0024] Compared with the prior art, the present invention proposes a data set synthesis method for target detection, which has many advantages: 1. The present invention uses a two-dimensional picture of a real scene as the background, and considers the camera's internal and external parameters and the background's lighting information during the three-dimensional rendering process, so that the generated synthetic data is closer to the real scene, thereby improving the authenticity of the data.

[0025] 2. For some specific scenarios or targets, it may be difficult to obtain a large amount of real data, but the method of the present invention can solve this problem by simulating and generating data for these scenarios or targets.

[0026] 3. Through the automatic three-dimensional pose prediction and target rendering process, the present invention can automatically generate the location label of the target, greatly reducing the cost of manual labeling.

[0027] 4. The method of the present invention can generate synthetic data on a large scale, thereby improving the efficiency of data generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The figure is a flow chart of a data set synthesis method for target detection according to the present invention.

[0029] Figure 2 Schematic diagram of the three-dimensional pose estimation model of the present invention.

[0030] Figure 3 The figure is a flowchart of the image annotation of the present invention. DETAILED DESCRIPTION

[0031] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0032] like Figure 1 As shown, the present invention provides a data set synthesis method for target detection, characterized in that it includes the following steps: Step S10: Establish a background image library and a 3D model library. First, collect a 2D picture of the actual environment as the background, and collect and record the internal and external parameters of the camera and the background lighting information to create a background image library. At the same time, construct a 3D model of the pre-detection target to create a 3D model library.

[0033] Step S101: Create a background image library. Collect two-dimensional images in the actual environment as backgrounds, and collect and record parameter information related to these background images. The specific steps are as follows: First, we use a high-resolution camera to photograph a calibration plate of known size and shape. By taking photos of the calibration plate from different angles and positions, the camera is calibrated to obtain the camera's internal parameters.

[0034]

[0035] in, is the camera internal parameter, and is the focal length of the camera, and are the coordinates of the optical center of the camera.

[0036] Next, we use the calibrated camera to shoot in various environments and lighting conditions to obtain as rich and diverse background scenes as possible. During the shooting process, we also place a calibration plate of known size and shape in the camera's field of view and take a photo of this calibration plate. By identifying the feature points in the photo, we can establish and solve mathematical equations containing external parameters to calibrate the camera's external parameters. These external parameters describe the position and orientation of the camera coordinate system relative to the world coordinate system.

[0037]

[0038] in, is the camera extrinsic parameter, is the rotation matrix of the camera extrinsic parameters, is the translation vector of the camera extrinsic parameter.

[0039] During the shooting process, we also record the type, position, direction and intensity of the light sources in the scene in detail. We first determine and record the type of light source (such as natural light or artificial light), then accurately measure and record the position of the light source through measurement tools or lighting models, then estimate the direction of the light source by observing the affected area of ​​the light and the direction of the shadows, and finally use a photometer or other light measurement device to measure and record the intensity of the light source.

[0040] Finally, we save the captured background images and the collected information in the background image library so that we can simulate and render the scene more accurately in the subsequent steps.

[0041] Step S102: Create a 3D model library. Construct a 3D model of the target to be detected and create a 3D model library. There are two main ways to create a 3D model. One is to create it from scratch using specialized 3D modeling software. The other is to use a 3D scanner to convert real-world objects into digital models.

[0042] Step S20: Estimate the three-dimensional posture information of the target to be detected. We use a multi-input deep neural network model, namely a three-dimensional posture estimation model, to predict the three-dimensional posture of the target to be detected in the current background image. The model receives the background image, the internal and external parameters of the camera, the position coordinates of the target to be detected in the image, and the point cloud data of the target to be detected as input, and then extracts features through the image processing subnetwork, the camera parameter processing subnetwork and the point cloud processing subnetwork respectively and splices these features together, and then further extracts the high-dimensional information therein, and finally outputs the confidence of the target to be detected on the background image, as well as the three-dimensional rotation information and scaling ratio of the model. The schematic diagram of the three-dimensional posture estimation model is shown below. Figure 2 As shown, the specific implementation process is as follows: Step S201: Background image processing subnetwork. The subnetwork extracts high-level features from the background image through a convolutional neural network (CNN). The choice of CNN can include models such as ResNet and VGG. First, the background image needs to be preprocessed to a suitable size, such as 512x512 pixels. After feature extraction by ResNet50, we can get a one-dimensional feature vector of size 2048.

[0043] Step S202: Camera parameter and coordinate processing subnetwork. The subnetwork uses a fully connected network (FCN) to process the internal and external parameters of the camera and the points in the camera coordinate system. For the internal and external parameters of the camera, we can directly normalize them to the range of [-1, 1] and input them into the FCN. For the position coordinates of the target to be detected in the image, we convert the coordinates into coordinates with the center of the image as the origin, convert and normalize them to the range of [-1, 1] and input them into the FCN. FCN can be composed of three fully connected layers, the first layer contains 256 nodes, the second layer contains 512 nodes, and the third layer has 1024 nodes, using ReLU as the activation function. Finally, the output of FCN is a one-dimensional feature vector of size 1024.

[0044] Step S203: Point cloud processing subnetwork. The subnetwork uses PointNet to extract information from three-dimensional point cloud data. First, the point cloud data is preprocessed, including centering (making all points surround the origin) and normalization (ensuring that all points are within the unit sphere). Then, PointNet is used to extract features to obtain a one-dimensional feature vector of size 1024. PointNet is a neural network designed for processing unordered point cloud data, which can directly extract global features from point clouds.

[0045] Step S204: Feature fusion and output. We aggregate the outputs of the three sub-networks to form a feature vector of size (2048+1024+1024)=4096. Next, the confidence, the three-dimensional rotation information of the model, and the scaling ratio of the model are output through two fully connected layers. The number of nodes in these two fully connected layers can be set to 512 and 5, where the 5 output nodes correspond to the confidence, the three-dimensional rotation information of the model, and the scaling ratio of the model.

[0046] The confidence level refers to the probability that the corresponding position of the input coordinates in the background image may exist. The three-dimensional rotation information of the model consists of the pitch angle, yaw angle, and roll angle. The scaling ratio of the model refers to the scaling ratio that should be performed when the model is projected on the background image.

[0047] Step S30: Scene reconstruction and target projection. In this step, the corresponding scene is first established in the 3D modeling software using the background image and related parameters. Then, the target to be detected is projected into a 2D image in the scene using the information predicted by the model in step S20.

[0048] Step S301: scene reconstruction. Reconstruct the camera position and viewing angle using camera parameters in the 3D modeling software, and re-illuminate the scene using the recorded light source information. The specific implementation steps are as follows: First, the background image is imported into the 3D modeling software. Then, the position and viewing angle of the camera are determined by the camera's external parameters, and then the camera's focal length and principal point coordinates are set by the camera's internal parameters.

[0049] The external parameters of a camera usually include a rotation matrix and a translation vector, which describe the transformation from the world coordinate system to the camera coordinate system. This transformation is usually expressed as:

[0050] in, is the coordinate of the point in the camera coordinate system, is the coordinate of the point in the world coordinate system, is the rotation matrix of the camera extrinsic parameters, is the translation vector of the camera extrinsic parameter.

[0051] If we want to find the position of the camera in world coordinates, let is zero, Solve,

[0052] in, yes The transpose of .

[0053] To get the angle of the camera in world coordinates, we need to use the rotation matrix in the camera's external parameters. To get the angle, we need to convert from the rotation matrix to Euler angles. For the XYZ order (i.e., rotate around the X axis first, then around the Y axis, and finally around the Z axis), the conversion formula may be as follows:

[0054] At the same time, the scene is re-illuminated according to the recorded light source information. First, the light source type is set to the recorded type in the 3D modeling software. Then, the position and direction of the light source are adjusted. Finally, the light intensity is adjusted according to the recorded light source intensity to complete the scene re-illumination in the 3D modeling software.

[0055] Step S302: Target projection. In this step, we first use the known camera information and the position coordinates of the target to be detected in the image to project the target into the reconstructed two-dimensional scene image. Then, we use the information predicted by the model in step S20 to adjust the three-dimensional rotation information and scaling of the target to obtain more realistic data. The specific implementation steps are as follows: First, the position coordinates of the target to be detected in the image are multiplied by the camera's intrinsic parameter matrix The inverse matrix of reprojects the coordinates in the image coordinate system into the camera coordinate system.

[0056]

[0057] in, are the camera coordinates, is the inverse of the internal parameter matrix, are image coordinates.

[0058] Then, the coordinates in the camera coordinate system are converted to world coordinates. This step can be achieved by the following formula:

[0059] in, yes The transposed matrix represents the inverse of the rotation relationship between the camera coordinate system and the world coordinate system. is the camera's translation vector, are world coordinates.

[0060] After obtaining the world coordinates of the target to be detected, the target model is placed in the 3D modeling software according to the world coordinates.

[0061] Finally, according to the three-dimensional rotation information and scaling ratio of the target to be measured obtained in step S20, we can adjust the target model to achieve the construction of a three-dimensional rendering scene with accurate position, appropriate size and natural posture.

[0062] Step S40: Rendering data and generating labels. Use 3D modeling software to render the reconstructed scene with the detection target, obtain a 2D image and annotation with the target, and obtain the annotation data. The process is as follows: Figure 3 shown.

[0063] Step S401: Rendering a synthetic image. Using the scene pre-configured in step S30, a synthetic image is generated by rendering through a 3D modeling software. During the rendering process, a synthetic image with a target is generated according to different combinations of targets and scenes.

[0064] Step S402: Render the target image. Use 3D modeling software to render the target to be detected in the scene. During the rendering process, only the target to be detected is rendered, and the background is set to transparent. The final output image is in PNG format and has an RGBA transparent channel. The rendered target image is recorded as .

[0065] Step S403: Obtain detection frame information. Processing is performed to extract the A channel of the RGBA format image. In this channel, the part with a value of 0 represents the background, and the part with a value other than 0 represents the target to be detected. By finding the minimum and maximum values ​​of the coordinates of the part with a value other than 0, the detection frame coordinates of the target can be obtained. By combining the category of the rendered target with the detection frame information, complete annotation information can be formed.

[0066] Usually, one synthetic image may contain 1 to 5 targets. When multiple targets are contained, steps S402 and S403 need to be repeatedly performed for each target to mark and obtain detection frame information.

Claims

1. A data set synthesis method for target detection, characterized in that: The following steps are involved: Step S10 establishes a background image library and a three-dimensional model library, which includes collecting a two-dimensional image in an actual environment as a background, collecting and recording the internal and external parameters of the camera and the lighting information of the background, thereby creating a background image library, and constructing a three-dimensional model of the pre-detection target, thereby creating a three-dimensional model library; Step S20 estimates the three-dimensional posture information of the target to be detected, which includes using a three-dimensional posture estimation model that can process multiple inputs to predict the three-dimensional posture of the target to be detected in the current background image through the background image, the point cloud data of the target to be detected, the internal and external parameters of the camera, and the position coordinates of the target to be detected in the image; Step S30 scene reconstruction and target projection, which includes using the background image and related parameters to establish a corresponding scene in the three-dimensional modeling software, and then projecting the target to be detected into a two-dimensional image in the scene through the information predicted by the model in step S20; Step S40 renders data and generates labels. This step includes using three-dimensional modeling software to render the reconstructed scene with the target to be detected, obtain a two-dimensional image with the target and corresponding annotations, and then obtain the annotation data.

2. The method according to claim 1, wherein in step S20, the three-dimensional posture estimation model includes an image processing subnetwork, a camera parameter processing subnetwork and a point cloud processing subnetwork, which are respectively used to process the input background image, the internal and external parameters of the camera, the position coordinates of the target to be detected in the image and the point cloud data of the target to be detected, extract the features and splice them together, and finally output the confidence of the target to be detected on the background image, as well as the three-dimensional rotation information and scaling ratio of the model.

Images