Automatic driving streetscape unsupervised 4D automatic labeling method based on three-dimensional reconstruction
By combining general pre-trained visual models and 3D reconstruction technology with the DBSCAN algorithm, unsupervised 4D annotation of street scenes for autonomous driving was achieved, solving the problems of high human resource consumption and domain differences in existing technologies, and improving the efficiency and accuracy of autonomous driving data annotation.
Patent Information
- Application Number
- CN202411091554.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-09
AI Technical Summary
Existing autonomous driving data annotation technologies rely on ground truth annotation, which suffers from high human resource consumption, long time consumption, and domain differences, and cannot achieve fully unsupervised 4D annotation.
Image information processing is performed using a general pre-trained visual model. Combined with image segmentation and depth estimation, denoising is achieved through 3D reconstruction and the DBSCAN algorithm, thus realizing unsupervised 4D automatic annotation.
This technology enables 3D object detection using only image information without ground truth annotations, avoiding the cost of manual annotation, solving the domain difference problem, and improving efficiency and accuracy.
Smart Images

Figure CN118967934A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of autonomous driving data processing, and in particular to an unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction. Background Art
[0002] At this stage, the research and development of autonomous driving is no longer just in the academic research stage, and the industry has given feedback on many demands for the practice of technology. Among them, the most basic one is that for the collected real data, it is necessary to annotate it, including but not limited to the 4D box and semantic information of the objects in the scene (the 4D box here refers to the length, width, height coordinates and time dimensions, that is, it is necessary to know the specific position of each object at different times). 4D annotation is a data annotation method. It not only contains all the information of traditional 2D annotation, but also introduces dimensions such as height and time to make the annotated data richer and more complete. In the field of autonomous driving, 4D annotation has broad application prospects, but this work can only be done manually or semi-automatically, which means that the data annotator needs to spend a lot of manpower, financial resources and time. Similar annotated data sets can refer to waymo and nuScenes.
[0003] The existing data annotation technology solution is to use data-driven training neural network methods to train the model based on a part of the existing 2D / 3D / 4D annotated data. After the training, the model can complete the target function, that is, input frame-by-frame images and point clouds to the model, and the model outputs frame-by-frame 3D information of the object. For example, BEVfusion and Auto4D have done a good job of object detection under fixed data sets. By training on a certain data set, the model can detect the data in the current data set domain, and the output effect is good, but it cannot directly reason on other data (non-training data domain); VSRD uses an implicit reconstruction method to reconstruct a single frame image. The reasoning process only needs a 2D image as input to output a 3D box, but due to its single-frame reconstruction characteristics, this solution still relies on true value data such as 2D boxes and object category identity information, and cannot be completely unsupervised.
[0004] In summary, existing technologies often rely on 3D ground truth annotations and 2D ground truth annotations for target data to be detected, and none of them can be completely unsupervised. In addition, most technologies have domain differences, that is, they cannot directly reason on unknown or unseen data (the detection accuracy of forced reasoning is almost 0). Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction, which can realize 3D object detection only by using image information without true value labeling.
[0006] The purpose of the present invention can be achieved by the following technical solution: A method for unsupervised 4D automatic labeling of street scenes for autonomous driving based on three-dimensional reconstruction, comprising the following steps:
[0007] S1, performing information processing on the input frame-by-frame images to obtain pixel-by-pixel instance information and pseudo point cloud;
[0008] S2, performing three-dimensional reconstruction based on the instance information and the pseudo point cloud to obtain a reconstructed scene;
[0009] S3. Sample the reconstructed scene frame by frame to obtain instance-by-instance point clouds, perform denoising, and output frame-by-frame 3D annotation boxes, thereby obtaining a 4D annotation result for the entire sequence consisting of frame-by-frame images.
[0010] Furthermore, the step S1 specifically uses a general pre-trained visual model to process information on each frame of the image.
[0011] Furthermore, the general pre-trained visual model includes an image segmentation model and a depth estimation model, wherein the image segmentation model is used to obtain semantic information of each pixel in the image, thereby obtaining instance information;
[0012] The depth estimation model is used to obtain the distance information of each pixel in the image and convert it into a pseudo point cloud format.
[0013] Furthermore, the working process of the image segmentation model is:
[0014] For image data Where k is the acquisition frame at different times, and j is the different images at the same time;
[0015] Will Input into the image segmentation model to obtain pixel-by-pixel instance information The value in is the instance ID to which the current pixel belongs. Figure 1 The same instance information is used to convert the image Calculate the 2D box according to the example, and project the area of the 2D box into the internal latent space of the image segmentation model to obtain the feature vector corresponding to each box in each image. m is the number of frames of a single image in the current frame, which is also the number of instances of the current image;
[0016] Then measure the Euclidean distance between multi-view feature vectors To determine whether the current two boxes belong to the same instance, if the distance is less than the threshold, they are considered to belong to the same instance, so that the instances of all views of the current frame are unified. The number of instances per frame is M, which is the total number of instances under all views of the current frame;
[0017] Continue to measure the Euclidean distance between feature vectors of different frames right Updated to enable on-instance tracking, making instances uniform across the scene.
[0018] Furthermore, the working process of the depth estimation model is:
[0019] For image data Where k is the acquisition frame at different times, and j is the different images at the same time;
[0020] Will Input into the depth estimation model to obtain pixel-by-pixel depth information, that is, depth information, which is then converted into a pseudo point cloud format.
[0021] Furthermore, the step S2 specifically adopts an implicit surface reconstruction method streetsurf based on signed distance field, and uses instance information to perform cross-frame full scene instance reconstruction.
[0022] Furthermore, the specific process of step S2 includes:
[0023] StreetSurf memorizes and overfits the required information by encoding scene information into latent space vectors. The loss functions used include pixel-by-pixel color regularization. and depth regularization is the color information of the current point predicted by the reconstruction network, is the depth information predicted by the reconstruction network;
[0024] In addition, for instance information, an additional multilayer perceptron model (Multilayer Perceptron, MLP) is used to predict instance information, and an additional instance regularization is used is the instance information predicted by the reconstruction network.
[0025] Furthermore, step S3 includes the following steps:
[0026] S31, sampling the reconstructed scene frame by frame, collecting dense spatial point coordinates, and querying the instance information corresponding to these spatial point coordinates to obtain the instance point cloud data PC frame by frame k ;
[0027] S32, performing denoising processing on the acquired instance point cloud to filter out outliers, and then calculating the 3D annotation box frame by frame according to the instance category.
[0028] Furthermore, the step S32 specifically adopts DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to perform denoising processing.
[0029] Furthermore, the calculation formula of the 3D annotation box in step S32 is:
[0030]
[0031] Among them, R k is the 3D annotation box result corresponding to the k-th frame image, z represents the z-th annotation box, and M is the total number of annotation boxes.
[0032] Compared with the prior art, the present invention has the following advantages:
[0033] The present invention processes the input frame-by-frame images to obtain pixel-by-pixel instance information and pseudo point clouds for three-dimensional reconstruction to obtain a reconstructed scene, then samples the reconstructed scene frame by frame to obtain instance-by-instance point clouds, performs denoising, and outputs frame-by-frame 3D annotation frames, thereby obtaining a 4D annotation result for the entire sequence of frame-by-frame images. This enables unsupervised automatic annotation of images, requiring only 2D time-series images and camera parameters to annotate the scene with object frames.
[0034] The present invention uses a universal pre-trained visual model to process information on each frame of the image, wherein the universal pre-trained visual model includes an image segmentation model and a depth estimation model. The image segmentation model is used to obtain the semantic information of each pixel in the image, and then the instance information is obtained; the depth estimation model is used to obtain the distance information of each pixel in the image and convert it into a pseudo point cloud format. The present invention uses a pre-trained universal visual model to process information, which not only avoids the model's need for true values, but also avoids the problem of data domain differences because these universal visual models are not trained under a single data set.
[0035] The present invention further utilizes a tracking method to perform similarity matching on frame-by-frame data based on the acquired semantic information, so as to achieve consistency of instance information across frames, that is, the identity information of the same instance at different times is the same, which is beneficial to subsequently unifying the instances of the entire scene.
[0036] Taking into account that the pseudo point cloud is not dense enough, the present invention designs an implicit surface reconstruction method streetsurf based on signed distance field, and uses instance information to reconstruct the full scene instance across frames. In addition, for the reconstructed scene, frame-by-frame sampling is performed to obtain denser three-dimensional semantics and instance point clouds. The point cloud is denoised and outliers are filtered out according to instance categories using the DBSCAN clustering algorithm, so that an accurate 3D frame can be calculated. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0038] Figure 2 Schematic diagram of the application process of the embodiment. DETAILED DESCRIPTION
[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] Example
[0041] like Figure 1 As shown, an unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction includes the following steps:
[0042] S1, performing information processing on the input frame-by-frame images to obtain pixel-by-pixel instance information and pseudo point cloud;
[0043] S2, performing three-dimensional reconstruction based on the instance information and the pseudo point cloud to obtain a reconstructed scene;
[0044] S3. Sample the reconstructed scene frame by frame to obtain instance-by-instance point clouds, perform denoising, and output frame-by-frame 3D annotation boxes, thereby obtaining a 4D annotation result for the entire sequence consisting of frame-by-frame images.
[0045] This embodiment applies the above solution and is mainly divided into three parts:
[0046] First, the input frame-by-frame images are processed, and the general pre-trained visual models SAM (Segment Anything Model) and Depth-Anything (depth estimation model) are used to obtain the semantic information and distance information of each pixel in the image. These general visual models are not trained on a single data set, so there is no problem of data domain differences. For the acquired semantic information, this solution also uses tracking methods to perform similarity matching on frame-by-frame data to achieve consistency of instance information across frames, that is, the identity information of the same instance is the same at different times.
[0047] The second part is to reconstruct the acquired information in three dimensions, and use the streetsurf reconstruction method to perform implicit surface reconstruction to reconstruct the surface information and semantic instance information of the street scene.
[0048] The last part is to sample the reconstructed scene frame by frame to obtain denser 3D semantics and instance point clouds, and use the DBSCAN clustering algorithm to denoise and filter out outliers according to instance categories to obtain accurate 3D frames.
[0049] Specifically, given a set of collected image data Where k is the acquisition frame at different times, and j is a different image at the same time, such as the front left and right views.
[0050] like Figure 2 As shown, in order to obtain the information required for reconstruction, Input to the segmenter S (i.e., image segmentation model) to obtain pixel-by-pixel instance information The number in represents which instance the current pixel belongs to. Specifically, S is a detector pre-trained on general data. It can be used directly by just loading its weights, without domain bias. Figure 1 The same instance information is used to convert the image Calculate the 2D box according to the example, and project the area of the 2D box into the internal latent space of S respectively, and obtain the feature vector corresponding to each box in each image m is the number of boxes, which is also the number of instances in the current image. Next, the Euclidean distance between the multi-view feature vectors is measured To determine whether the current two boxes belong to the same instance, if the distance is less than the threshold, it is considered that the similarity is very high and they belong to the same instance, so that the instances of all views of the current frame are unified, and the number of instances per frame is M. Similarly, continue to measure the Euclidean distance between feature vectors of different frames Updated It is possible to implement tracking on instances, making the instances of the entire scene unified.
[0051] In addition to obtaining instance information, the image is also input into a pre-trained depth information detector D (i.e., depth estimation model), which is used in a similar way to the detector S, and the pre-trained model can be used directly. In this way, pixel-by-pixel depth information, i.e., pseudo point cloud, can be obtained for subsequent 3D reconstruction.
[0052] Although we already have a pseudo point cloud, we can theoretically calculate the detection box directly. However, considering that the pseudo point cloud is not dense enough, the accuracy of directly calculating the detection box is not high and the effect is not good. Therefore, we use the implicit surface reconstruction method Streetsurf based on the signed distance field, and use the instance information to reconstruct the full scene instance across frames. Streetsurf encodes the scene information into a latent space vector to memorize and overfit the required information. The loss function used is mainly pixel-by-pixel color regularization. and depth regularization is the color information of the current point predicted by the reconstruction network, is the depth information predicted by the reconstruction network. In addition, for instance information, an additional multi-layer perceptron model (MLP) is used to predict instance information, and an additional instance regularization is used is the instance information predicted by the reconstruction network.
[0053] After the reconstruction network (i.e. 3D reconstruction) is trained, dense spatial point coordinates are collected, and the instance information corresponding to these point coordinates is queried from the network to obtain the frame-by-frame instance point cloud data PC k . Since the reconstruction network is not perfect and has errors, these point clouds will have some outliers that interfere with subsequent calculations. Therefore, this solution uses the DBSCAN algorithm to filter out these noise points. Finally, the 3D annotation box can be calculated frame by frame according to the example:
[0054] To verify the effectiveness of this solution, this embodiment was tested on the internationally accepted waymo dataset, and the results are shown in the following table:
[0055]
[0056] The results show that the unsupervised method proposed in this scheme has the advantages of high efficiency, low cost and accurate results.
[0057] In summary, this solution can detect 3D objects using only image information without ground truth annotation (referring to 2D / 3D / 4D object detection frames and corresponding semantic information) for any data. By inputting frame-by-frame street view images and corresponding camera posture parameters, the 3D frames and instance semantic information of each object in the street view image can be output. This solution uses a pre-trained general visual model to process information, which not only avoids the model's need for ground truth, but also avoids domain difference problems. This solution realizes unsupervised automatic image annotation, and only requires 2D time-series images and camera parameters to annotate the scene with object frames. Compared with the existing technology, this solution does not require 2D frame and semantic annotation at all, and is essentially unsupervised, directly saving the cost of manual annotation and data preprocessing, improving efficiency and saving a lot of computing resources.
Claims
1. An unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction, characterized in that: The following steps are involved: S1, performing information processing on the input frame-by-frame images to obtain pixel-by-pixel instance information and pseudo point cloud; S2, performing 3D reconstruction based on the instance information and the pseudo point cloud to obtain a reconstructed scene; S3. Sample the reconstructed scene frame by frame to obtain instance-by-instance point clouds, perform denoising, and output frame-by-frame 3D annotation boxes, thereby obtaining a 4D annotation result for the entire sequence consisting of frame-by-frame images.
2. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 1, characterized in that: The step S1 specifically uses a general pre-trained visual model to process information on each frame of the image.
3. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 2, characterized in that: The general pre-trained visual model includes an image segmentation model and a depth estimation model, wherein the image segmentation model is used to obtain semantic information of each pixel in the image, and then obtain instance information; The depth estimation model is used to obtain the distance information of each pixel in the image and convert it into a pseudo point cloud format.
4. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 3, characterized in that: The working process of the image segmentation model is: For image data Where k is the acquisition frame at different times, and j is the different images at the same time; Will Input into the image segmentation model to obtain pixel-by-pixel instance information The value in is the instance ID to which the current pixel belongs. In order to obtain consistent instance information across frames and views, the image Calculate the 2D box according to the example, and project the area of the 2D box into the internal latent space of the image segmentation model to obtain the feature vector corresponding to each box in each image. m is the number of frames of a single image in the current frame, which is also the number of instances of the current image; Then measure the Euclidean distance between multi-view feature vectors {p,q}∈m, to determine whether the current two boxes belong to the same instance. If the distance is less than the threshold, they are considered to belong to the same instance, so that the instances of all views of the current frame are unified. The number of instances per frame is M, which is the total number of instances under all views of the current frame; Continue to measure the Euclidean distance between feature vectors of different frames {p,q}∈M, for Updated to enable on-instance tracking, making instances uniform across the scene.
5. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 4, characterized in that: The working process of the depth estimation model is as follows: For image data Where k is the acquisition frame at different times, and j is the different images at the same time; Will Input into the depth estimation model to obtain pixel-by-pixel depth information, that is, depth information, which is then converted into a pseudo point cloud format.
6. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 5, characterized in that: The step S2 specifically adopts the implicit surface reconstruction method streetsurf based on the signed distance field, and uses the instance information to perform cross-frame full scene instance reconstruction.
7. The unsupervised 4D automatic labeling method for street scenes of autonomous driving based on three-dimensional reconstruction according to claim 6, characterized in that: The specific process of step S2 includes: StreetSurf memorizes and overfits the required information by encoding scene information into latent space vectors. The loss functions used include pixel-by-pixel color regularization. and depth regularization is the color information of the current point predicted by the reconstruction network, is the depth information predicted by the reconstruction network; In addition, for instance information, an additional multi-layer perceptron model is used to predict instance information and an additional instance regularization is used. is the instance information predicted by the reconstruction network.
8. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 7, characterized in that: The step S3 comprises the following steps: S31, sampling the reconstructed scene frame by frame, collecting dense spatial point coordinates, and querying the instance information corresponding to these spatial point coordinates to obtain the instance point cloud data PC frame by frame k ; S32, performing denoising processing on the acquired instance point cloud to filter out outliers, and then calculating the 3D annotation box frame by frame according to the instance category.
9. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 8, characterized in that: The step S32 specifically uses the DBSCAN algorithm to perform denoising.
10. The unsupervised 4D automatic labeling method for autonomous driving street scenes based on three-dimensional reconstruction according to claim 9, characterized in that: The calculation formula of the 3D annotation frame in step S32 is: Among them, R k is the 3D annotation box result corresponding to the k-th frame image, z represents the z-th annotation box, and M is the total number of annotation boxes.
Citation Information
Patent Citations
Unsupervised monocular three-dimensional target detection method based on video sequence and pre-training instance segmentation
CN116129318A
Systems and methods for generating and using visual datasets for training computer vision models
US20220414928A1
Cited By
3D target detection method based on instance-scene fusion enhancement
CN120107536A
3D object detection method based on instance-scene fusion enhancement
CN120107536B
