Three-dimensional perception model training method and electronic equipment

Abstract

The invention provides a three-dimensional perception model training method and electronic equipment, and the method comprises the steps: inputting two-dimensional images corresponding to different visual angles into a three-dimensional perception model, enabling the three-dimensional perception model to obtain corresponding two-dimensional feature information according to the two-dimensional images, converting the two-dimensional feature information into a three-dimensional space, obtaining three-dimensional feature information, and carrying out the training of the three-dimensional perception model; rendering a two-dimensional image according to the three-dimensional feature information to obtain a corresponding depth map and a color map, obtaining a corresponding pseudo-depth map according to the two-dimensional image, and determining a training loss value of the three-dimensional perception model according to the two-dimensional image and the corresponding depth map, color map and pseudo-depth map; and training the three-dimensional perception model according to the training loss value until the three-dimensional perception model meets the target requirement, and ending the training of the three-dimensional perception model. Therefore, the training of the three-dimensional perception model can be realized according to the two-dimensional images corresponding to different visual angles, the model training cost is effectively reduced, and the complexity and the calculation amount of the training of the three-dimensional perception model are reduced.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to a three-dimensional perception model training method and electronic device. Background Technology

[0002] 3D perception is a crucial foundational task in the field of computer vision. Taking intelligent driving as an example, 3D perception aims to reconstruct and understand 3D scenes using perception data from various sensors such as cameras and radar, and generate 3D perception results so that vehicles can perform path planning, intelligent vehicle control, and other tasks based on these results.

[0003] The implementation of 3D perception typically involves training a 3D perception model using pre-trained data to enable the model to perceive 3D information. Current technologies mostly employ multimodal sensing data from various sensors such as cameras and radar as pre-training data. However, due to the high deployment cost of LiDAR and its susceptibility to adverse weather conditions, the point cloud data acquired by LiDAR can be inaccurate. Complex preprocessing of the LiDAR point cloud data is required before training the 3D perception model, resulting in high pre-training data costs. Furthermore, training the 3D perception model using multimodal sensing data increases the network complexity and computational overhead of the 3D perception model.

[0004] Therefore, existing methods for training 3D perception models suffer from high costs in acquiring pre-training data, high complexity in training 3D perception models, and large computational overhead. Summary of the Invention

[0005] This application provides a method and electronic device for training a 3D perception model, which can reduce the cost of acquiring pre-training data and reduce the training complexity and computational overhead of the 3D perception model.

[0006] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a method for training a three-dimensional perception model. This method includes: acquiring two-dimensional images corresponding to different viewpoints; inputting the two-dimensional images into a three-dimensional perception model, enabling the model to obtain two-dimensional feature information corresponding to the two-dimensional images, and converting the two-dimensional feature information into three-dimensional space to obtain three-dimensional feature information; rendering the two-dimensional images based on the three-dimensional feature information to obtain a depth map and a color map corresponding to the two-dimensional images; obtaining a pseudo-depth map corresponding to the two-dimensional images; determining a training loss value for the three-dimensional perception model based on the two-dimensional images and their corresponding depth map, color map, and pseudo-depth map; and training the three-dimensional perception model based on the training loss value until the model meets the target requirements, thus ending the training of the three-dimensional perception model.

[0007] By adopting the above technical solution, only a two-dimensional image is input, and three-dimensional feature information is generated from the two-dimensional image to obtain the corresponding depth map. Thus, when training the 3D perception model, only images are used as pre-training data, eliminating the need to acquire point cloud data, reducing the cost of acquiring pre-training data. Furthermore, by reducing the amount of pre-training data required for 3D perception model training, the complexity and computational load of 3D perception model training are further reduced.

[0008] Furthermore, a pseudo-depth map is introduced. Based on the two-dimensional image and its corresponding depth map, color map, and pseudo-depth map, the training loss value of the three-dimensional perception model is determined. Then, based on the training loss value, the three-dimensional perception model is updated and trained, which effectively improves the perception accuracy of the three-dimensional perception model.

[0009] In one possible implementation of the first aspect described above, the method further includes: determining multiple three-dimensional points on multiple rays corresponding to different scattering directions with the ray origin as the origin in three-dimensional space, and determining the distance information of each three-dimensional point relative to the ray origin; obtaining the three-dimensional coordinate information of each three-dimensional point based on the coordinate information of the origin, and the scattering direction and distance information corresponding to each three-dimensional point; rendering a two-dimensional image through the three-dimensional feature information to obtain a depth map corresponding to the two-dimensional image, including: obtaining the three-dimensional feature information of each three-dimensional point from the three-dimensional feature information through the nearest neighbor interpolation method; determining the signed distance value of each three-dimensional point based on the three-dimensional feature information and the three-dimensional coordinate information of each three-dimensional point using a first multilayer perceptron; determining the opacity of each three-dimensional point based on the signed distance value of each three-dimensional point through volume rendering; determining the transmittance of the ray reaching the corresponding three-dimensional point based on the opacity of each three-dimensional point on the ray; determining the depth value of the pixel corresponding to each ray based on the distance information and opacity of each three-dimensional point on each ray, and the transmittance of the ray reaching the corresponding three-dimensional point; and rendering a two-dimensional image based on the depth value of each pixel to generate a depth map corresponding to the two-dimensional image.

[0010] By employing the above technical solution, three-dimensional point sampling is performed on the image converted from a two-dimensional image to a three-dimensional image to obtain the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point. Based on the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point, a signed distance value of each three-dimensional point is generated, and the opacity of each three-dimensional point is determined based on the signed distance value. Based on the opacity of each three-dimensional point on the light ray, the transmittance of the light ray reaching the corresponding three-dimensional point is determined. Furthermore, based on the distance information and opacity of each three-dimensional point on each light ray, as well as the transmittance of the light ray reaching the corresponding three-dimensional point, the depth value of the pixel point corresponding to each light ray is determined. In this way, by sampling three-dimensional points and determining the depth value, the two-dimensional image is finally rendered based on the depth value of each pixel point, generating a depth map that better reflects the depth value of the original two-dimensional image.

[0011] In one possible implementation of the first aspect above, the depth value of a pixel is obtained as follows:

[0012]

[0013] Where, depth i w is the depth value of the i-th pixel. k s represents the depth weight of the k-th 3D point on the ray corresponding to the i-th pixel. k For the distance information of the k-th 3D point, T k Let α be the transmittance of light reaching the k-th three-dimensional point. j Let α be the opacity of a ray from its origin to the k-th 3D point. k Let φ be the opacity of the k-th 3D point. s () is the logic density distribution function, φ sDF (p k φ is the signed distance value of the k-th 3D point. SDF (p k+1 ) represents the signed distance value of the (k+1)th 3D point.

[0014] In one possible implementation of the first aspect above, rendering a two-dimensional image based on three-dimensional feature information to obtain a color map corresponding to the two-dimensional image includes: determining a first color value for each three-dimensional point based on the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point on each ray of the two-dimensional image using a second multilayer perceptron; determining a second color value for the pixel corresponding to each ray based on the first color value of each three-dimensional point on each ray; and rendering the two-dimensional image based on the second color value and depth value of each pixel to generate a color map.

[0015] In one possible implementation of the first aspect described above, determining the training loss value of the 3D perception model based on the 2D image and the corresponding depth map, color map, and pseudo-depth map includes: determining the photometric consistency loss value of the 3D perception model based on the 2D image and the corresponding depth map; determining the rendering loss value of the 3D perception model based on the 2D image and the corresponding color map; determining the relative proximity loss value of the 3D perception model based on the corresponding pseudo-depth map and depth map; and determining the training loss value of the 3D perception model based on the photometric consistency loss value, the rendering loss value, and the relative proximity loss value.

[0016] By adopting the above technical solution, the loss of photometric consistency loss, rendering loss, and relative proximity loss is determined, and the loss of photometric perception, rendering color perception, and rendering depth perception in the scene perception process of the 3D perception model is fully considered. Then, the training loss value of the 3D perception model is determined based on the photometric consistency loss value, rendering loss value, and relative proximity loss value. In this way, the 3D perception model can be updated more comprehensively based on the training loss value, thereby improving the performance of the 3D perception model.

[0017] In one possible implementation of the first aspect described above, when there are multiple frames of two-dimensional images corresponding to different viewpoints, the method further includes: determining the camera intrinsic matrix and camera extrinsic matrix of the camera used to capture the two-dimensional images; determining the photometric consistency loss value of the three-dimensional perception model based on the two-dimensional images and the depth maps corresponding to the two-dimensional images, including: projecting each pixel of the first frame of two-dimensional images onto the second frame of two-dimensional images based on the depth map, camera intrinsic matrix, and camera extrinsic matrix corresponding to the first frame of two-dimensional images for each viewpoint, wherein the first frame of two-dimensional images and the second frame of two-dimensional images are any two adjacent frames of two-dimensional images in the multiple frames of two-dimensional images; determining the third color value of each pixel of the first frame of two-dimensional images and the fourth color value of each pixel of the projected second frame of two-dimensional images; determining the target sampling point from multiple pixels, and determining the photometric consistency loss value of the three-dimensional perception model based on the third color value and the fourth color value of the target sampling point corresponding to each two-dimensional image.

[0018] Using the above technical solution, three-dimensional point projection and color value determination are performed on two adjacent two-dimensional images to determine the photometric consistency loss value based on the color values ​​of the two adjacent two-dimensional images. This allows for the supervision of the depth of multiple images perceived by the three-dimensional perception model.

[0019] In one possible implementation of the first aspect above, determining the photometric consistency loss value of the three-dimensional perception model based on the third color value and the fourth color value of the target sampling point corresponding to each two-dimensional image includes: determining the first loss value corresponding to each two-dimensional image based on the third color value and the fourth color value of the target sampling point corresponding to each two-dimensional image; and determining the minimum loss value from the first loss value as the photometric consistency loss value of the three-dimensional perception model.

[0020] In one possible implementation of the first aspect above, determining the rendering loss value of the three-dimensional perception model based on the two-dimensional image and the color map corresponding to the two-dimensional image includes: determining the second loss value of each pixel between the color map and the two-dimensional image based on the second color value of each pixel in the color map and the third color value of the corresponding pixel in the two-dimensional image; and determining the rendering loss value of the three-dimensional perception model based on the second loss value of each pixel.

[0021] By employing the above technical solution, rendering loss calculations are performed on the color map and two-dimensional image generated by the 3D perception model, which can supervise the color rendering of the 3D perception model.

[0022] In one possible implementation of the first aspect described above, generating a pseudo-depth map corresponding to a two-dimensional image from the two-dimensional image includes: generating a pseudo-depth map of the two-dimensional image based on a depth estimation network; determining the relative proximity loss value of a three-dimensional perception model based on the pseudo-depth map and the depth map corresponding to the two-dimensional image includes: determining multiple pixel pairs corresponding to the depth map and the pseudo-depth map; determining a first relative proportion of each pixel pair corresponding to the depth map based on the depth value of each pixel pair in the depth map; determining a second relative proportion of each pixel pair corresponding to the pseudo-depth map based on the depth value of each pixel pair in the pseudo-depth map; and determining the relative proximity loss value of the three-dimensional perception model based on the first relative proportion and the second relative proportion.

[0023] By adopting the above technical solution, loss is calculated based on the depth values ​​of the pseudo-depth map and the depth map generated by the 3D perception model to obtain the relative proximity loss value, which can supervise the depth rendering of the 3D perception model.

[0024] In one possible implementation of the first aspect above, each pixel pair includes a first pixel and a second pixel, and the relative proximity loss value of the 3D perception model is obtained in the following way:

[0025]

[0026] Where a is the first pixel, a′ is the second pixel, d(a) is the depth value of the first pixel in the depth map, d(a′) is the depth value of the second pixel in the depth map, d′(a) is the depth value of the first pixel in the pseudo-depth map, and d′(a′) is the depth value of the second pixel in the pseudo-depth map. This represents the first relative proportion of pixel pairs corresponding to the depth map. is the second relative scale of the pixel pair corresponding to the pseudo-depth map, and s is the scaling factor. This is the first constraint result of applying a three-class classification constraint to the first relative proportion based on the Sigmoid function. This is the second constraint result based on the Sigmoid function to apply a three-class classification constraint to the second relative proportion.

[0027] In one possible implementation of the first aspect above, converting two-dimensional feature information to three-dimensional space to obtain three-dimensional feature information includes: generating multi-scale two-dimensional image feature information based on the two-dimensional feature information using a feature pyramid network; establishing a three-dimensional space, projecting voxels of the three-dimensional space onto the two-dimensional image to obtain two-dimensional coordinate information corresponding to each projected pixel; and sampling the multi-scale two-dimensional image feature information based on the two-dimensional coordinate information to obtain three-dimensional feature information.

[0028] By adopting the above technical solution, a three-dimensional space is established, and a two-dimensional image is converted into a three-dimensional space to obtain three-dimensional feature information. In this way, three-dimensional perception can be achieved by only inputting a two-dimensional image.

[0029] In one possible implementation of the first aspect described above, the pre-training data consists of data from an autonomous driving dataset and a navigation planning dataset; the two-dimensional feature information is obtained by extracting features from the two-dimensional image using the backbone network included in the three-dimensional perception model, with an initial learning rate of 2×10⁻⁶. -4 The depth map is obtained by rendering a 2D image based on 3D feature information using a rendering decoder included in the 3D perception model. The initial learning rate of the rendering decoder is 2×10. -5 The three-dimensional points in the two-dimensional image include a first preset number of three-dimensional points and a second preset number of three-dimensional points, wherein the first preset number of three-dimensional points are points uniformly distributed in the two-dimensional image, and the second preset number of three-dimensional points are important points in the two-dimensional image; the objective requirement is that the training period of the three-dimensional perception model reaches a preset training period, or the loss function converges.

[0030] Secondly, this application also discloses a three-dimensional perception model, which is obtained based on the three-dimensional perception model training method provided by any of the implementation methods in the first aspect.

[0031] Thirdly, this application also discloses an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to enable the electronic device to implement the three-dimensional perception model training method provided by any of the implementations of the first aspect.

[0032] Fourthly, this application also discloses a computer-readable storage medium storing a computer program that can be executed by an electronic device to implement the three-dimensional perception model training method provided by any of the implementations of the first aspect.

[0033] Fifthly, this application also discloses a computer program product, including a computer program that, when executed by an electronic device, implements the three-dimensional perception model training method provided by any of the implementations of the first aspect. Attached Figure Description

[0034] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0035] Figure 1 A schematic diagram illustrating the principle of a 3D perception model training method provided by existing technology;

[0036] Figure 2 A schematic flowchart of a three-dimensional perception model training method provided in an embodiment of this application;

[0037] Figure 3 A schematic diagram illustrating the principle of training various modules based on a 3D perception model, as provided in the embodiments of this application;

[0038] Figure 4 A schematic diagram illustrating the principle of the three-dimensional perception model training method provided in this application embodiment;

[0039] Figure 5A A schematic diagram illustrating the relationship between the amount of pre-trained data and the metric value of the 3D perception model, provided for embodiments of this application;

[0040] Figure 5B A schematic diagram comparing the perception results of the three-dimensional perception model provided in the embodiments of this application and existing three-dimensional perception models;

[0041] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] 3D perception is a crucial foundational task in computer vision, aiming to reconstruct and understand 3D scenes using various sensors such as cameras and radar (e.g., LiDAR, millimeter-wave radar). Current 3D perception methods typically initialize with pre-trained weights from existing supervised learning 3D perception models to obtain a trained 3D perception model, enabling the use of the feature representation capabilities of existing 3D perception models for 3D perception.

[0043] like Figure 1As shown in section (a), this section describes a supervised pre-training method utilizing off-the-shelf 3D perception models. This involves inputting labeled datasets (e.g., image data) into the image encoders of existing 3D perception models (Model A and Model B) for feature encoding to obtain image features. These image features are then input into their respective 3D encoders (3DEncoders) for feature encoding to obtain 3D features. Finally, the 3D features are input into their respective 3D decoders for feature decoding to obtain the corresponding 3D perception results. Model A is a pre-trained model, and each model has different pre-trained weights. The image encoders of all models jointly perform image initialization.

[0044] However, the training of existing 3D perception models often relies on manually labeled pre-training data and is limited by differences in model architecture. Due to the high cost of labeling, existing technologies, such as those in the field of autonomous driving, use a large amount of unlabeled data in autonomous driving datasets. These unlabeled data are used as pre-training data for self-supervised model training. For example, unlabeled pre-training data can be used to train a masked autoencoder (MAE) model to effectively utilize the large amount of unlabeled data, thereby leveraging a large amount of additional information to improve the performance of 3D perception models for downstream tasks.

[0045] Among them, the MAE model is a deep learning method based on self-supervised learning (SSL).

[0046] While existing technologies enable the training of 3D perception models based on unlabeled pre-training data, most self-supervised 3D perception model pre-training currently utilizes multimodal perception data as input. Compared to 2D image data captured by cameras, point cloud data acquired by LiDAR provides richer geometric and structural information about 3D scenes. Existing methods for training 3D perception models based on 2D images captured by cameras and point cloud data acquired by LiDAR can be broadly categorized into two types: one is based on contrastive learning, which improves the model's ability to extract 3D features by using newly generated point cloud data from different perspectives. Typical methods include PointContrast and DepthContrast. Figure 1As shown in section (b), section (b) is a self-supervised 2D pre-training method based on multimodal data. It utilizes unlabeled 2D images (Camera Input) and multimodal point cloud data (LIDAR Input) from radar input for pre-training. Specifically, it includes inputting the 2D image into an image encoder to obtain image features, inputting the image features into an image decoder for 3D box prediction and depth prediction, obtaining non-dense depth measurements from radar point cloud data, and calculating depth loss based on sparse depth ground truth (GT) and depth prediction results to update the 3D perception model based on the depth loss.

[0047] Another approach is volumetric rendering-based 3D reconstruction. This method explicitly reconstructs 3D features from a 2D image input, generating a depth map corresponding to the input image through volumetric rendering. The depth data from the point cloud input is then used as supervision. Typical methods include UniScene and UniPAD. Figure 1 As shown in section (c), this section presents a self-supervised 3D pre-training method based on multimodal data. It directly utilizes the self-supervised information from multimodal data to learn 3D geometric representations in 3D space without any annotation. Specifically, it involves inputting a 2D image into an image encoder to obtain image features, inputting these features into an image decoder to obtain bird's-eye view (BEV) features, performing occupancy prediction based on the BEV features, obtaining the ground truth (Occupancy GT) from radar point cloud data, and calculating a classification loss based on the Occupancy GT and Occupancy prediction. The 3D perception model is then updated based on the classification loss.

[0048] However, these methods all rely on multimodal data input and are thus limited by it. For example, LiDAR deployment is costly and susceptible to severe weather, and it also suffers from point irregularities and sparsity, typically requiring complex preprocessing to improve data quality. More importantly, processing LiDAR data increases network complexity and model-related computational overhead. These factors limit the applicability of existing self-supervised 3D pre-training frameworks that rely on LiDAR point cloud data as input.

[0049] Therefore, existing 3D perception model training methods usually rely on a large amount of labeled training data or multimodal data, which limits their versatility and efficiency in practical applications.

[0050] Based on this, this application proposes a three-dimensional perception model training method, which trains the three-dimensional perception model based on two-dimensional images as input. This method can achieve three-dimensional perception model pre-training without additional modalities or standard data, thereby improving the model's feature representation ability in various three-dimensional perception tasks.

[0051] Next, the training method for the three-dimensional perception model provided in this application will be described in detail.

[0052] like Figure 2 As shown, the 3D perception model training method provided in this application includes the following steps:

[0053] S100: Acquire two-dimensional images corresponding to different viewpoints.

[0054] S200 inputs a two-dimensional image into a three-dimensional perception model, enabling the three-dimensional perception model to obtain the two-dimensional feature information corresponding to the two-dimensional image, convert the two-dimensional feature information into three-dimensional space to obtain three-dimensional feature information, and render the two-dimensional image based on the three-dimensional feature information to obtain the depth map and color map corresponding to the two-dimensional image.

[0055] S300, obtains the pseudo-depth map corresponding to the two-dimensional image based on the two-dimensional image.

[0056] S400 determines the training loss value of the 3D perception model based on the 2D image and its corresponding depth map, color map, and pseudo-depth map.

[0057] S500 updates and trains the 3D perception model based on the training loss value until the 3D perception model meets the target requirements, and then ends the training of the 3D perception model.

[0058] By adopting the above technical solution, only a two-dimensional image is input, and three-dimensional feature information is generated from the two-dimensional image to obtain the corresponding depth map and color map. In this way, when training the three-dimensional perception model, only images are used as pre-training data, eliminating the need to acquire point cloud data, thus reducing the cost of acquiring pre-training data. Furthermore, by reducing the amount of pre-training data required for the three-dimensional perception model training, the complexity and computational load of the training are further reduced.

[0059] Furthermore, a pseudo-depth map is introduced. Based on the two-dimensional image and its corresponding depth map, color map, and pseudo-depth map, the training loss value of the three-dimensional perception model is determined, and the three-dimensional perception model is updated and trained, which effectively improves the perception accuracy of the three-dimensional perception model.

[0060] In one implementation of step S100, the two-dimensional image is an image from the autonomous driving dataset (nuScenes) and the navigation planning dataset (nuplan).

[0061] For example, keyframes from nuScenes and images from the nuplan dataset at a predetermined ratio are selected as pre-training data.

[0062] Specifically, the preset ratio can be... Of course, other proportions of data are also possible.

[0063] In one implementation of step S200, the two-dimensional feature information is obtained by extracting features from the two-dimensional image using the backbone network included in the three-dimensional perception model.

[0064] For example, two-dimensional images corresponding to different viewpoints (i.e., multi-view images) The input to the 2D backbone network of the 3D perception model is used for feature extraction to generate 2D feature information (i.e., 2D features F) of the 2D image. 2D ).

[0065] In one implementation, the backbone network is ConvNeXt, and the initial learning rate of the backbone network is 2×10. -4 .

[0066] Furthermore, the view transformation module based on the 3D perception model will convert the 2D feature information F 2D The information is converted to three-dimensional space (i.e., 3D space) to obtain three-dimensional feature information.

[0067] In one implementation, two-dimensional feature information is converted to three-dimensional space to obtain three-dimensional feature information, including first generating multi-scale two-dimensional image feature information based on the two-dimensional feature information using a feature pyramid network.

[0068] For example, using a Feature Pyramid Network (FPN) based on F... 2D Generate multi-scale two-dimensional image feature information To preserve both coarse and fine granular information in two-dimensional images.

[0069] Furthermore, a three-dimensional space is established, and the voxels in the three-dimensional space are projected onto the two-dimensional image to obtain the two-dimensional coordinate information corresponding to each voxel after projection.

[0070] For example, a three-dimensional space (i.e., 3D space, or 3D mesh space v) is modeled using three-dimensional grid occupancy. Based on the view transformation module, voxels in the three-dimensional space are projected onto a two-dimensional image plane, so that each voxel {v} i ∈V} is projected onto a multi-view image to obtain the two-dimensional coordinate information of the pixels corresponding to each voxel after projection.

[0071] In one implementation of this application, the method further includes: determining the camera intrinsic parameter matrix and camera extrinsic parameter matrix of the camera used to capture the two-dimensional image.

[0072] Furthermore, the two-dimensional coordinate information corresponding to each projected pixel after projection is obtained, including: obtaining the two-dimensional coordinate information of each projected pixel corresponding to each voxel after projection based on the camera intrinsic parameter matrix, the camera extrinsic parameter matrix and the three-dimensional coordinate information of each voxel in three-dimensional space.

[0073] The two-dimensional coordinate information is obtained in the following way:

[0074] p c,i =T in ·T e2c ·p e,i

[0075] Where, p c,i T represents the two-dimensional coordinate information of the projected pixels after projection. in Let T be the camera intrinsic parameter matrix. e2c p is the camera extrinsic parameter matrix. e,i Let be the three-dimensional coordinate information of the (i)th voxel in three-dimensional space.

[0076] Furthermore, based on the two-dimensional coordinate information, the feature information of the multi-scale two-dimensional image is sampled and processed to obtain the three-dimensional feature information.

[0077] For example, the projected two-dimensional coordinate information is used to sample and process the feature information of the multi-scale two-dimensional image, and finally the three-dimensional feature information f of the entire scene is obtained. v .

[0078] In one implementation, the three-dimensional feature information f v It is obtained in the following way:

[0079]

[0080] Among them, f v F is a set of three-dimensional feature information. k For multi-scale two-dimensional image feature information, p c,i Two-dimensional coordinate information of the projected pixels after projection.

[0081] Furthermore, in one implementation of this application, the depth map and color map are obtained by rendering the two-dimensional image based on the three-dimensional feature information by the rendering decoder included in the three-dimensional perception model, and the initial learning rate of the rendering decoder is 2×10. -5 .

[0082] For example, a differentiable rendering decoder (as an example of a rendering decoder) is used to render a two-dimensional image based on three-dimensional feature information to obtain a depth map and a color map corresponding to the two-dimensional image.

[0083] In the implementation of this application, firstly, multiple three-dimensional points on multiple rays corresponding to different scattering directions are determined in three-dimensional space with the ray origin as the origin, and the distance information of each three-dimensional point relative to the ray origin is determined.

[0084] For example, light rays are sampled along different scattering directions in three-dimensional space, where the camera's origin (i.e., the starting point of the light ray) is o, and the direction is d. i To facilitate calculation, several three-dimensional points are sampled from each ray to determine the distance information of each three-dimensional point relative to the starting point of the ray.

[0085] In the implementation of this application, the three-dimensional points in the two-dimensional image include a first preset number of three-dimensional points and a second preset number of three-dimensional points, wherein the first preset number of three-dimensional points are points evenly distributed in the two-dimensional image, and the second preset number of three-dimensional points are important points in the two-dimensional image.

[0086] For example, the number of sampling points in a single 2D image is set to 512, and the volume rendering process adopts a layered sampling strategy, in which 64 points (as an example of a first preset number of 3D points) are used for uniform sampling, and 32 points (as an example of a second preset number of 3D points) are used for importance sampling.

[0087] That is, when performing three-dimensional point sampling, a first preset number of points evenly distributed in the two-dimensional image and a second preset number of important points distributed in the two-dimensional image are sampled, as well as other numbers of scattering points, etc.

[0088] Furthermore, based on the coordinate information of the origin and the scattering direction and distance information corresponding to each three-dimensional point, the three-dimensional coordinate information of each three-dimensional point is obtained.

[0089] For example, the three-dimensional coordinate information of each three-dimensional point is obtained in the following way:

[0090]

[0091] Where, p k This provides the 3D coordinates of a point, where o represents the coordinates of the ray's origin, and s... kDistance information of three-dimensional points, d i This represents the scattering direction of the light rays from the three-dimensional point.

[0092] Furthermore, after obtaining the three-dimensional feature information and the three-dimensional coordinate information of each three-dimensional point, the depth map corresponding to the two-dimensional image is obtained based on the three-dimensional feature information and the three-dimensional coordinate information of each three-dimensional point.

[0093] First, the three-dimensional feature information of each three-dimensional point is obtained from the three-dimensional feature information by the nearest neighbor interpolation method.

[0094] For example, the nearest neighbor interpolation method is used to obtain the three-dimensional feature information (e.g., three-dimensional spatial information f) of three-dimensional points. k ).

[0095] Based on the first multilayer perceptron, a signed distance value for each three-dimensional point is generated according to the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point.

[0096] For example, the 3D feature information and 3D coordinate information of 3D points are concatenated and then used to generate a signed distance field (SDF) through a first multi-layer perceptron (MLP). The signed distance value is φ. SDF (cat(p k ,f k )), abbreviated as φ SDF (p k ).

[0097] Furthermore, the depth values ​​of the pixels corresponding to each ray are obtained through volume rendering.

[0098] First, the opacity of each 3D point is obtained based on the signed distance value of each 3D point.

[0099] For example, the opacity of each 3D point is calculated using the signed distance value (SDF) of each 3D point on the ray, wherein the opacity of each 3D point is calculated as follows:

[0100]

[0101] Where, α k Let φ be the opacity of the k-th 3D point. s () is the logic density distribution function, φ SDF (p k φ is the signed distance value of the k-th 3D point. SDF (p k+1 ) represents the signed distance value of the (k+1)th 3D point.

[0102] Where, φs () is the Sigmoid function Sig s (x)=(1+e -sx ) -1 The derivative of .

[0103] Furthermore, based on the opacity of each three-dimensional point on the light ray, the transmittance of the light ray reaching the corresponding three-dimensional point is obtained.

[0104] For example, the projection rate of light reaching a three-dimensional point is obtained based on the opacity of each three-dimensional point preceding the point in the direction of light scattering.

[0105] The transmittance of the three-dimensional points is obtained as follows:

[0106]

[0107] Among them, T k Let α be the transmittance of light reaching the k-th three-dimensional point. j The opacity of the light ray is the opacity of each three-dimensional point from the starting point of the light ray to the k-th three-dimensional point.

[0108] Furthermore, based on the distance information and opacity of each three-dimensional point on each ray, as well as the transmittance of the ray reaching the corresponding three-dimensional point, the depth value of the pixel corresponding to each ray is obtained.

[0109] For example, the depth weight of each 3D point is obtained based on its opacity and transmittance. Then, the depth value of pixel i corresponding to the ray is obtained by weighted summation based on the depth weights and distance information of all 3D points on the corresponding ray. i .

[0110] The depth weights of each 3D point are obtained as follows:

[0111] w k =T k ·α k

[0112] Among them, w k T represents the depth weight of the k-th 3D point on the ray corresponding to the i-th pixel. k Let α be the transmittance of light reaching the k-th three-dimensional point. k Let be the opacity of the k-th 3D point.

[0113] The depth value of each pixel is obtained as follows:

[0114]

[0115] Where, depth i w is the depth value of the i-th pixel.k s represents the depth weight of the k-th 3D point on the ray corresponding to the i-th pixel. k This represents the distance information for the k-th 3D point.

[0116] Furthermore, a two-dimensional image is rendered based on the depth value of each pixel to obtain the depth map corresponding to the two-dimensional image.

[0117] For example, a two-dimensional image is rendered based on the depth value of each pixel to obtain a depth map corresponding to the two-dimensional image.

[0118] Furthermore, in the implementation of this application, during the pre-training process of the three-dimensional perception model, the two-dimensional image is rendered based on the three-dimensional feature information to obtain the color map corresponding to the two-dimensional image.

[0119] For example, a two-dimensional image is rendered based on the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point on each ray of the two-dimensional image to obtain the color map corresponding to the two-dimensional image.

[0120] Specifically, based on the second multilayer perceptron, the first color value of each three-dimensional point is obtained from the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point on each ray corresponding to the two-dimensional image.

[0121] For example, using the three-dimensional coordinate information and three-dimensional feature information of each three-dimensional point on the aforementioned sampled light rays, the data is stitched together and then processed by another second multilayer perceptron to generate the first color value c of each three-dimensional point. k .

[0122] The first color value is obtained as follows:

[0123] c k =φ RGB (p k ,f k )

[0124] Among them, c k p is the first color value of the 3D point k. k f represents the three-dimensional coordinate information of point k. k This represents the three-dimensional feature information of point k.

[0125] Furthermore, the second color value of the pixel corresponding to each ray is obtained based on the first color value of each three-dimensional point on each ray.

[0126] For example, the second color value I′(p) of each pixel in the corresponding two-dimensional image is calculated by weighted summation of the first color values ​​of all three-dimensional points on the corresponding ray. i ).

[0127] Among them, the second color value I′(p iIt is obtained in the following way:

[0128]

[0129] Wherein, I′(p i ) represents the second color value of pixel i, w k c represents the depth weight corresponding to the 3D point k. k Let be the first color value of 3D point k, and n be the number of 3D points on the ray corresponding to pixel point i.

[0130] Then, a color map is obtained by rendering a two-dimensional image based on the second color value and depth value of each pixel.

[0131] For example, a color map (i.e., an RGB image) is obtained by rendering a two-dimensional image based on the second color value and depth value of each pixel.

[0132] In this implementation, a 2D backbone network is used to generate 2D feature information of a 2D image. Then, a view transformation module converts this 2D feature information into a 3D space modeled using a 3D raster, projecting voxels from the 3D space onto the 2D image plane. Based on the 2D feature information of the projected pixels, 3D feature information of the entire scene is generated. Further, based on the rasterized 3D feature information, a signed distance function (SDF) is used to represent the geometric information in the 3D space. Multiple rays are sampled in the 3D space, and multiple 3D spatial points are sampled from each ray. The spatial feature information (3D feature information) of each 3D point is extracted using the nearest neighbor interpolation method, and a multilayer perceptron is used to generate the SDF value of each 3D point. The depth value of the corresponding pixel is then obtained through volume rendering, thus generating a depth map. The entire volume rendering process is divided into four stages: determining the 3D feature information of the 3D points, determining the SDF value of each 3D point, determining the depth value of each pixel, and rendering to generate the depth map. Furthermore, the color value (i.e., the first color value) of each three-dimensional point is determined based on the three-dimensional feature information and three-dimensional coordinate information of each three-dimensional point. The color value of each pixel is obtained based on the color value of each three-dimensional point (i.e., the second color value). The two-dimensional image is rendered based on the color value of each pixel to generate a color map.

[0133] Further, for step S300, obtaining the pseudo depth map corresponding to the two-dimensional image based on the two-dimensional image includes: obtaining the pseudo depth map of the input two-dimensional image through a pre-trained depth estimation network using the method described above.

[0134] Furthermore, for step S400, based on the two-dimensional image and its corresponding depth map, color map, and pseudo-depth map, the training loss value of the three-dimensional perception model is determined, including the following steps:

[0135] S410, determine the photometric consistency loss value of the 3D perception model based on the 2D image and the corresponding depth map.

[0136] For example, when there are multiple frames of 2D images corresponding to different viewpoints, the photometric consistency between adjacent 2D images is used to supervise monocular depth estimation. For instance, the points of the current frame are projected into adjacent frames using depth values, camera intrinsic matrix, and camera extrinsic matrix, and then the L1 loss between the RGB values ​​of the sampled points and the RGB values ​​of the projected points is calculated.

[0137] Specifically, as mentioned above, the camera intrinsic and extrinsic parameter matrices of the camera used to capture 2D images are determined.

[0138] Furthermore, based on the depth map, camera intrinsic parameters, and camera extrinsic parameters corresponding to the first frame of the two-dimensional image at each viewpoint, each pixel of the first frame of the two-dimensional image is projected onto the second frame of the two-dimensional image. The first frame of the two-dimensional image and the second frame of the two-dimensional image are any two adjacent frames of two-dimensional images in the multi-frame two-dimensional image.

[0139] For example, the depth value D of each pixel in the depth map corresponding to the first frame of the two-dimensional image is used. t (p t The camera intrinsic parameter matrix and the camera intrinsic parameter matrix determine the three-dimensional coordinate information of each pixel in the camera coordinate system corresponding to the first frame of two-dimensional image.

[0140] The 3D coordinates of each pixel in the camera coordinate system corresponding to the first frame of the 2D image are determined as follows:

[0141]

[0142] in, D represents the 3D coordinates of pixel t in the camera coordinate system in the first frame of the 2D image. t (p t ) represents the depth value of pixel t in the depth map corresponding to the first frame of the 2D image, K is the camera intrinsic parameter matrix, and p t This refers to the three-dimensional coordinates of pixel t in the first frame of the two-dimensional image.

[0143] Furthermore, based on the camera extrinsic matrix and the three-dimensional coordinate information of each pixel in the first frame of the two-dimensional image in the camera coordinate system, the three-dimensional coordinate information of each pixel in the second frame of the two-dimensional image in the camera coordinate system is obtained.

[0144] The 3D coordinates of each pixel in the camera coordinate system in the second frame of the 2D image are obtained as follows:

[0145]

[0146] in, T represents the 2D coordinates of pixel s in the second frame of the 2D image, which corresponds to pixel t in the first frame of the 2D image, in the camera coordinate system. t→s This is the camera extrinsic parameter matrix.

[0147] It should be noted that the 3D coordinate information of the pixel is 3D homogeneous coordinates.

[0148] Furthermore, based on the two-dimensional coordinate information of each pixel in the second frame two-dimensional image in the camera coordinate system, the camera intrinsic parameter matrix, and the depth value corresponding to each pixel in the depth map, the projection coordinate information of each pixel in the second frame two-dimensional image is obtained.

[0149] The projection coordinates of pixels in the second frame of the 2D image are obtained as follows:

[0150]

[0151] Among them, P s P represents the projection coordinates of pixel s in the second frame of the two-dimensional image. s =(u s ,v s ), This represents the depth value of a pixel.

[0152] Furthermore, the third color value of each pixel in the first frame of the two-dimensional image and the fourth color value of each pixel in the projected second frame of the two-dimensional image are determined.

[0153] For example, the third color value of each pixel can be obtained directly from the first frame of the two-dimensional image:

[0154] I t (p t )=[R t (u t ,v t ),G t (u t ,v t ),B t (u t ,v t )] T ∈R 3

[0155] Furthermore, the fourth color value of each pixel in the second frame of the two-dimensional image is obtained as follows:

[0156]

[0157] Among them, I s (p s ) represents the fourth color value of each pixel in the second frame of the two-dimensional image.

[0158] Furthermore, a target sampling point is determined from multiple pixels, wherein the target sampling point is the same pixel in the first frame of the two-dimensional image and the second frame of the two-dimensional image (i.e., the same pixel), and there are one or more target sampling points.

[0159] Based on the third and fourth color values ​​of the target sampling points corresponding to each two-dimensional image, the photometric consistency loss value of the three-dimensional perception model is obtained.

[0160] For example, taking the same pixel t in the first frame of the two-dimensional image and pixel s in the second frame of the two-dimensional image as an example of target sampling points, the photometric consistency loss value of pixel t in the first frame of the two-dimensional image and pixel (i.e. projection point) s in the second frame of the two-dimensional image is calculated.

[0161] In one implementation of this application, the photometric consistency loss value of pixel t in the first frame of the two-dimensional image and pixel (i.e., projection point) s in the second frame of the two-dimensional image is obtained in the following way:

[0162] L p (p t ,s)=||I t (p t )-I s (p s )||1

[0163] Among them, L p (p t ,s) represents the luminous consistency loss value (i.e., the first loss value) between pixel t and pixel s.

[0164] Furthermore, in the implementation of this application, obtaining the photometric consistency loss value of the three-dimensional perception model based on the third and fourth color values ​​of the target sampling points includes: determining the first loss value corresponding to each two-dimensional image based on the third and fourth color values ​​of the target sampling points corresponding to each two-dimensional image; and determining the minimum loss value from the first loss values ​​as the photometric consistency loss value of the three-dimensional perception model.

[0165] The photometric consistency loss value of the 3D perception model is obtained as follows:

[0166]

[0167] Among them, L photo This represents the photometric consistency loss value of the 3D perception model.

[0168] It should be noted that L photo This is also known as the photometric consistency loss function.

[0169] For example, the minimum loss value among the photometric consistency loss values ​​of any adjacent two-dimensional images is determined as the photometric consistency loss value of the three-dimensional perception model.

[0170] S420: Determine the rendering loss value of the 3D perception model based on the 2D image and the corresponding color map.

[0171] The process of determining the rendering loss value of the 3D perception model based on the 2D image and its corresponding color map includes: determining the second loss value of each pixel between the color map and the 2D image based on the second color value of each pixel in the color map and the third color value of the corresponding pixel in the 2D image; and obtaining the rendering loss value of the 3D perception model based on the second loss value of each pixel.

[0172] For example, the second loss value for each pixel is obtained as follows:

[0173] L p (p i )=||I(p i )-I′(p i )||1

[0174] Among them, L p (p i I(p) represents the second loss value between pixel i in the two-dimensional image and its corresponding pixel i in the color map. i ) is the third color value of pixel i in the two-dimensional image, which is also the aforementioned I. i (p i ), I′(p i ) represents the second color value of pixel i in the color map.

[0175] Furthermore, the rendering loss value of the 3D perception model is obtained as follows:

[0176]

[0177] Among them, L render is the rendering loss value for the 3D perceptual model, and N is the number of pixels.

[0178] It should be noted that L render This is also known as the rendering loss function.

[0179] S430: Determine the relative proximity loss value of the 3D perception model based on the pseudo-depth map and depth map corresponding to the 2D image.

[0180] In one implementation of this application, multiple pixel pairs corresponding to the depth map and the pseudo-depth map are first determined.

[0181] For example, multiple pairs of pixels are determined in the depth map and the pseudo-depth map, each pair of pixels including a first pixel and a second pixel.

[0182] Furthermore, based on the depth value of each pixel pair in the depth map, the first relative proportion of each pixel pair corresponding to the depth map is determined.

[0183] For example, the first relative proportion of each pixel point corresponding to the depth map is obtained based on the depth value d(a) of the first pixel point a in the depth map and the depth value d(a′) of the second pixel point a′ in the depth map.

[0184] Furthermore, based on the depth values ​​of each pixel pair in the pseudo-depth map, a second relative proportion of each pixel pair corresponding to the pseudo-depth map is determined.

[0185] For example, the first relative proportion of each pixel point to the corresponding depth map is obtained based on the depth value d′(a) of the first pixel point a in the pseudo depth map and the depth value d′(a′) of the second pixel point a′ in the pseudo depth map.

[0186] Furthermore, the relative proximity loss value of the three-dimensional perception model is determined based on the first relative ratio and the second relative ratio.

[0187] For example, the Sigmoid function is used to apply a three-class constraint to the first relative proportion to obtain the first constraint result, and the Sigmoid function is used to apply a three-class constraint to the second relative proportion to obtain the second constraint result.

[0188] The first constraint result is obtained in the following way:

[0189]

[0190] in, This represents the first constraint result of the three-class classification constraint based on the first relative proportion using the Sigmoid function. Here, 'a' represents the first pixel, 'a′' represents the second pixel, 's' represents the scaling factor, and 'd(a)' represents the depth value of the first pixel in the depth map, and 'd(a′)' represents the depth value of the second pixel in the depth map. This represents the first relative scale of the pixel pair corresponding to the depth map.

[0191] The second constraint result is obtained in the following way:

[0192]

[0193] in, This is the second constraint result based on the sigmoid function to apply a three-class classification constraint to the second relative proportion. d′(a) is the depth value of the first pixel in the pseudo-depth map, and d′(a′) is the depth value of the second pixel in the pseudo-depth map. This represents the second relative proportion of the pixel pairs corresponding to the pseudo-depth map.

[0194] The relative proximity loss value of the 3D perception model is obtained as follows:

[0195]

[0196] Among them, L rank This represents the relative proximity loss value of the 3D perception model. This is the first constraint result of applying a three-class classification constraint to the first relative proportion based on the Sigmoid function. This is the second constraint result based on the Sigmoid function to perform a three-class classification constraint on the second relative proportion, where n′ is the number of pixel pairs.

[0197] It should be noted that L rank This is also known as the relative proximity loss function.

[0198] S440 determines the training loss value of the 3D perception model based on the photometric consistency loss value, rendering loss value, and relative proximity loss value.

[0199] For example, the training loss value of the 3D perception model is obtained by weighted summing of the photometric consistency loss value, the rendering loss value, and the relative proximity loss value.

[0200] The training loss value is obtained as follows:

[0201] L total =λ1L photo +λ2L render +λ3L rank

[0202] Among them, L total λ1, λ2, and λ3 are the training loss values ​​and the weight coefficients.

[0203] For example, λ1 = 0.8, λ2 = 0.5, and λ3 = 0.3.

[0204] It should be noted that L total This is also known as the total loss function.

[0205] For step S500, the backpropagation algorithm is used to update the network parameters of the 3D perception model according to the total loss function, and new pre-training data (i.e. pre-training samples) are randomly sampled. Steps S100-S300 are repeated to repeatedly train the 3D perception model until the 3D perception model meets the target requirements (i.e. training termination conditions), and the training of the 3D perception model ends.

[0206] The objective is that the training period of the 3D perception model reaches the preset training period, or the loss function converges.

[0207] For example, pre-training for 6 training cycles is performed, repeatedly executing steps S100 and S200, enabling the 3D perception model to establish a 3D perception process that obtains a depth map from a 2D image, thus generating 3D perception capability. After completing the 6 training cycles of pre-training, steps S300 and S400 are executed to calculate the training loss value of the trained 3D perception model. The network parameters in the 3D perception model are adjusted based on the training loss value, that is, the parameters of the backbone network and the differentiable rendering decoder are adjusted. Pre-training data is then re-acquired, and steps S100-S400 are executed again to verify the parameter adjustment results, until the maximum training cycle (as an example of a preset training cycle) is met, at which point training ends. Of course, the preset training cycle can be set according to actual conditions. Furthermore, in this implementation, the 3D perception model is also subjected to 12 cycles of perception adjustment for downstream tasks.

[0208] Of course, in the implementation of this application, it is also possible to determine whether the total loss function (i.e., the loss function) converges. If the total loss function converges, it means that the trained 3D perception model has reached the standard, and then the training ends.

[0209] It should be noted that in the implementation of this application, the 3D perception model is optimized based on the training loss value by the optimizer.

[0210] The optimizer can specifically be AdamW, which can stably handle multi-task losses, such as photometric consistency loss, rendering loss, and relative proximity loss, in order to update the parameters of the 3D perception model.

[0211] The optimizer updates parameters by adjusting the learning rate (LR) of the backbone network and the differentiable rendering decoder, and then updating the parameters of the 3D perception model based on the learning rate.

[0212] The 3D perception model training method provided in this application is a self-supervised 3D perception pre-training method based on pure visual input. It can pre-train the 3D perception model using only camera image data as input without additional modal data or labeled data, thereby improving the feature representation capability of the 3D perception model in various 3D perception tasks.

[0213] Furthermore, by introducing pseudo-depth maps to assist in supervising the image depth predicted by the 3D perception model, a relative proximity loss function is proposed to better utilize the prior information provided by the pseudo-depth maps to verify the performance of the 3D perception model in generating depth maps, thereby improving the stability of training and the performance of the 3D perception model.

[0214] Furthermore, in another implementation of this application, such as Figure 3 As shown, the method for pre-training and self-supervised 3D perception models with pure visual input using a 3D pre-training framework combining self-learning and pure visual input provided in this application includes the following steps.

[0215] First, the multi-view images captured by the camera are input into the image encoder to obtain two-dimensional feature information. The two-dimensional image features are then input into the view transformation module to transform the two-dimensional feature information into three-dimensional space, obtaining three-dimensional feature information (i.e., voxel features). Based on volume rendering, the two-dimensional image is rendered according to the voxel features to obtain an RGB rendered image (i.e., a color image) and a depth rendered image (i.e., a depth image).

[0216] Furthermore, self-supervised (No Ground Trial) processing is performed based on the RGB and depth rendering maps to obtain a rendering loss from the RGB rendering map and a photometric consistency loss from the depth map. A pseudo-depth map of the input image is then obtained through a pre-trained depth estimation network, and a relative proximity loss is calculated based on the pseudo-depth map and the depth map. Finally, training loss values ​​are obtained based on the rendering loss, photometric consistency loss, and relative proximity loss. These training loss values ​​are used to adjust the network parameters of the 3D perception model, and the 3D perception model is retrained and the loss is recalculated until the 3D perception model meets the target conditions, at which point training ends.

[0217] like Figure 4As shown, the pre-training process of the 3D perception model includes: inputting multi-view images (i.e., 2D images from different perspectives) into the image encoder of the image backbone network of the 3D perception model to obtain 2D feature information; inputting the 2D feature information into the view transformation module, which constructs a 3D space and transforms the 2D feature information into 3D space to obtain 3D feature information (Voxelated 3DFeatures); performing downstream tasks based on the 3D feature information; obtaining the 3D coordinate information and 3D feature information of each 3D point on each ray through volume rendering; inputting the 3D coordinate information and 3D feature information of each 3D point on each ray into the SDF network and the RGB network; the SDF network, based on the first multilayer perceptron, obtains the SDF value of each 3D point based on the 3D coordinate information and 3D feature information of each 3D point on each ray; obtaining the depth value of each pixel based on the SDF value of each 3D point; and then obtaining the depth map (Rendered) based on the depth value of each pixel. The depth map (Rendered Images, i.e., RGB images) is obtained by the RGB network based on the second multilayer perceptron using the 3D coordinate information and 3D feature information of each 3D point on the light. The RGB values ​​of each pixel are obtained based on the RGB values ​​of each 3D point. The color map (Rendered Images, i.e., RGB images) is obtained based on the depth value and RGB values ​​of each pixel.

[0218] Furthermore, the self-supervised training loss calculation process of the 3D perception model includes: constructing a model based on a pseudo-depth map (i.e., a depth estimation network); determining the pseudo-depth map based on multi-view images; determining the relative proximity loss (ranking loss) based on the pseudo-depth map and the depth map; determining the photometric consistency loss (photometric loss) based on multi-view images and the depth map; determining the rendering loss (rendering loss) based on multi-view images and the color map; finally, obtaining the total training loss based on the relative proximity loss, photometric consistency loss, and rendering loss; adjusting the network parameters of the 3D perception model based on the total training loss to update the model until the 3D perception model reaches the target conditions, and ending the training.

[0219] The 3D perception model training method provided in this application generates 2D feature information from image input and transforms it into 3D feature information. Volume rendering is then used to construct a corresponding depth map from the 3D feature information. Supervision is then performed using photometric consistency loss from adjacent frames. During pre-training, 3D spatial features are explicitly constructed and their rendered depth maps are supervised to better learn the geometric structure of the 3D scene, thereby enhancing the 3D representation capability of the 3D perception model. This method also adapts to different feature extraction backbone networks, making it suitable for various downstream 3D perception tasks. This process relies entirely on visual input and requires no additional LiDAR data or other modal information. Furthermore, an innovative loss function design is proposed, comprising three parts: photometric consistency loss, rendering loss, and relative proximity loss. The photometric consistency loss supervises monocular depth estimation, the rendering loss supervises depth rendering using the reconstructed input image, and the relative proximity loss uses a pseudo-depth map obtained from the pre-trained depth estimation network to assist in supervising the predicted depth. In addition, this application also employs specific training strategies, selects specific pre-training data, sets a reasonable number of sampling points and a hierarchical sampling strategy, and uses optimized learning rates and optimizers, etc., to further improve the training effect and stability of the model.

[0220] In summary, the main difference between the 3D perception model training method of this application and existing 3D perception model training methods is that this application is a self-supervised 3D perception model pre-training based on pure visual input of images, and innovatively proposes a new loss function and training strategy, which enables 3D perception model pre-training without additional modal or labeled data, and significantly improves the feature representation ability of the 3D perception model in various 3D perception tasks.

[0221] Experiments on the nuScenes and nuplan datasets demonstrate that the self-supervised update of the proposed 3D perception model significantly improves performance compared to baseline methods without additional supervision. Extensive experiments prove that it significantly enhances the baseline performance of models used directly without pre-training, surpassing the effectiveness of commonly used supervised initialization methods. Furthermore, pre-training using images allows for easy scaling to larger-scale unlabeled data, further enhancing 3D perception capabilities. Under the same baseline, the self-supervised training based on the unsupervised pre-training framework of 3D scene representation, compared to existing supervised pre-training methods based on point cloud data, not only maintains the final performance but also achieves improvements by utilizing more pre-training data, eliminating the need for pre-training data annotation. Simultaneously, the method pre-training based on camera-captured image data is more efficient than current unsupervised pre-training methods based on multimodal inputs.

[0222] See Table 1, which compares the 3D target detection performance of each model after training different baseline models using the 3D perception model training method provided in this application.

[0223] Table 1. Performance Comparison of 3D Object Detection Based on nuScenes

[0224] Baseline model backbone network Modal Pre-trained annotation types NDS↑ mAP↑ PGD

[30] ResNet-101 Visual images √ 39.3 31.7 DETR3D

[31] ResNet-101 Visual images √ 42.5 34.6 Spatial DETR[6] ResNet-101 Visual images √ 42.5 35.1 BEV model V2

[36] ResNet-50 Visual images √ 42.6 35.1 PETR

[17] VoVNet Visual images √ 44.2 37.0 BEV Model-S

[16] ResNet-101 Visual images √ 44.8 37.5 Ego

[19] ResNet-101 Visual images √ 45.0 37.5 CMT-C

[35] VoVNet Visual images √ 46.0 40.6 FCOS3D

[29] ResNet-101 Visual images √ 38.4 31.1 FCOS3D

[29] ResNet-101 Visual images × 32.2 26.1 FCOS3D-CSPP ResNet-101 Visual images × 39.2 31.9 UVTR-C

[14] ResNet-50 Visual images √ 40.1 31.3 UVTR-C

[14] ResNet-50 Visual images × 28.5 24.2 UVTR-C-CSRR ResNet-50 Visual images × 40.7 32.0 UVTR-C

[14] ResNet-101 Visual images √ 45.0 37.2 UVTR-C

[14] ResNet-101 Visual images × 35.1 30.2 UVTR-C-CSRR ResNet-101 Visual images × 45.4 37.8

[0225] In this study, single-modality visual images (Cameras) from the nuScenes dataset were input into different 3D perception models (Methods, such as PGD, DETR3D, ..., UVTR-C-CSPP). The image backbone networks (Backbone, such as ResNet-101, ResNet-50, VoVNet) included in these different 3D perception models were different.

[0226] Each 3D perception model performs 3D target detection based on visual images, and the NDS (NuScenes Detection Score, a comprehensive evaluation metric for the nuScenes dataset) and mAP (mean Average Precision) of each 3D perception model are significantly improved.

[0227] For example, when using ResNet-50 as the image backbone network, and using unlabeled pre-training data, the baseline model of UVTR-C-CSRR, which is unsupervised pre-trained based on the 3D perception model training method proposed in this application, improves the mAP by 7.8 (32.0-24.2=7.8) and the NDS by 12.2 (40.7-28.5=12.2) compared to the UVTR-C baseline model trained based on existing training methods. When using ResNet-101 as the backbone network, the FCOS3D-CSPP baseline model trained using the proposed 3D perception model training method achieves a 5.8 mAP improvement (31.9-26.1=5.8) and a 7.0 NDS improvement (39.2-32.2=7.0) compared to the FCOS3D baseline model. Similarly, the UVTR-C-CSRR baseline model trained using the proposed 3D perception model training method achieves a 7.6 mAP improvement (37.8-30.2=7.6) and a 10.3 NDS improvement (45.4-35.1=10.3) compared to the UVTR-C baseline model. Compared to existing supervised initialization methods, the self-supervised pre-training method proposed in this application not only does not sacrifice performance but also achieves improvements on multiple benchmarks. Furthermore, taking the FCOS3D baseline model as an example, based on the same image backbone network (ResNet-101), the FCOS3D-CSPP baseline model using pretrained annotation type (Pretrain.Label) with unpretrained annotation (i.e., pretrained annotation type is ×) performs better in 3D object detection (NDS is 39.2, mAP is 31.9) than the FCOS3D baseline model using pretrained annotation (i.e., pretrained annotation type is √) pretrained data (NDS is 38.4, mAP is 31.1).

[0228] Furthermore, see Table 2, which compares the performance of each model in predicting 3D grid occupancy after 3D perception training of different models based on the 3D perception model training method provided in this application.

[0229] Table 2 Performance Comparison of 3D Raster Occupancy Prediction Methods Based on nuScenes

[0230]

[0231] As shown in Table 2, the experimental results demonstrate that the training method for the 3D perception model proposed in this application brings significant improvements compared to ImageNet pre-training. For example, for the BEVFormer baseline model, which is also trained using unlabeled pre-training data (pre-training label type is ×), the BEVFormer-COSP model, which is unsupervised pre-trained using the 3D perception model training method of this application, has a mean intersection over union (mIoU) that is 2.47 higher than the BEVFormer model trained using the ImageNet pre-training method (27.45-24.98=2.47). For the TPVFormer baseline model, which is also trained using unlabeled pre-training data (pre-training label type is ×), the TPVFormer-COSP model, which is unsupervised pre-trained using the 3D perception model training method of this application, has a mean intersection over union (mIoU) that is 2.85 higher than the TPVFormer model trained using the ImageNet pre-training method (28.46-25.61=2.85).

[0232] Furthermore, as shown in Table 2, the 3D perception model trained based on the 3D perception model training method provided in this application exhibits better performance in detecting targets such as barriers, bicycles, buses, cars, construction vehicles, motorcycles, pedestrians, traffic cones, trailers, trucks, drivable surfaces, other flat areas, sidewalks, terrain, man-made objects, vegetation, and others.

[0233] The 3D perception model trained in this application utilizes a larger-scale pre-training data, which, although unsupervised, not only avoids performance degradation but also brings certain improvements.

[0234] Furthermore, compared with existing methods based on multimodal input and labeled pre-training data, the self-supervised pre-training method for 3D perception models based on pure visual input in this application has significantly improved effectiveness and scale for using larger pre-training data.

[0235] As shown in Table 3, Table 3 compares the performance of the 3D perception model pre-trained based on nuScenes and nuplan with that pre-trained based on multimodal data.

[0236] Table 3. Performance comparison of multimodal pre-training methods for 3D object detection (Det3D) based on nuScenes.

[0237]

[0238] As shown in Table 3, the 3D perception model trained by the self-supervised pre-training method based on pure visual input proposed in this application (+CSPP) can approach or even surpass the 3D perception model trained by other 3D perception model pre-training methods based on multimodal input (UVTR). Compared with the benchmark method, the 3D perception model trained in this application can improve mAP by 5.2 by performing unsupervised pre-training on downstream task datasets. In addition, the amount of pre-training data (Dataset) also has a significant impact on the results; by using a larger dataset, mAP can be further improved by 2.1.

[0239] Furthermore, compared with existing supervised pre-training methods, the self-supervised pre-training method for 3D perception models based on pure visual input in this application can achieve better results by using more pre-training data.

[0240] As shown in Table 4, Table 4 compares the performance of the 3D perception model pre-trained based on nuScenes and nuplan with that based on supervised pre-training.

[0241] Table 4. Performance comparison of supervised pre-training methods for 3D object detection (Det3D) based on nuScenes.

[0242]

[0243]

[0244] As shown in Table 4, compared with the most common method of initializing the current model using FCOS3D model weights, the mAP of the 3D perception model trained in this application is further improved by 1.3 (7.3-6.0=1.3).

[0245] Furthermore, such as Figure 5A As shown, Figure 5A This is a graph showing the relationship between the amount of pre-trained data (Frames) and the metric value of the downstream task.

[0246] like Figure 5AAs shown, both mAP and NDS increase with the increase of the amount of unlabeled pre-training data, which means that the increase of the amount of unlabeled pre-training data brings greater performance improvement to the 3D perception model.

[0247] Furthermore, this application evaluates the depth map re-rendered by the pre-trained task and compares it with current self-supervised depth estimation methods to verify the effectiveness of the 3D perception model obtained by the pre-training method in depth estimation.

[0248] As shown in Table 5, Table 5 compares the depth estimation performance based on nuScenes.

[0249] Table 5 shows the performance comparison of depth estimation based on nuScenes.

[0250]

[0251] As shown in Table 5, the 3D perception model obtained in this application still achieves performance comparable to state-of-the-art methods (SOTA, such as LiteMono, Monodepth2, PackNet-SfM, FSM, and SurroundDepth) in depth estimation. The 3D perception model obtained in this application matches or even outperforms SOTA methods in four metrics: absolute relative error (AbsRel), squared relative error (SqRel), root mean square error (RMSE), and logarithmic space RMSE (RMSE log). It also demonstrates competitive performance in metrics such as the percentage of pixels with a predicted depth-to-ground value ratio within 1.25 (δ1, i.e., relaxed precision), moderate precision (δ2), and relaxed precision (δ3, covering the vast majority of correct predictions).

[0252] Furthermore, this application also explores the performance of the self-supervised pre-training method based on this application on downstream tasks by visualizing the results of downstream tasks.

[0253] See Figure 5B , Figure 5B Visualize the 3D raster occupancy prediction results on Occ3D-nuScenes.

[0254] according to Figure 5B As can be seen, the perception prediction results (Prediction(COSP)) of the 3D perception model pre-trained using the method proposed in this application for the same 3D scene are compared with the perception prediction results (Prediction(Baseline)) based on the existing baseline model for the same 3D scene, and then compared with the ground truth. The accuracy of scene perception of the 3D perception model pre-trained using the method proposed in this application is higher than that of scene perception based on the existing baseline model.

[0255] Pre-training using the method proposed in this application significantly improves the overall scene understanding ability of the 3D perception model and enhances the accuracy of predicting large objects in the scene. This is because deep pre-training enables the 3D perception model to perceive the front-back relationships in the scene, thereby more accurately representing the object outlines.

[0256] This application also discloses a three-dimensional perception model, which is obtained based on the aforementioned three-dimensional perception model training method.

[0257] The 3D perception model training method provided in this application is applied to electronic devices. By setting computer program instructions, the electronic device pre-trains a 3D perception model based on a 2D image, and performs training loss value calculation and 3D perception model update.

[0258] Please see Figure 6 , Figure 6 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device may include: transceiver 121, processor 122, and memory 123.

[0259] The processor 122 executes computer execution instructions stored in the memory, causing the processor 122 to perform the technical solution of the three-dimensional perception model training method in the above embodiments. The processor 122 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital data processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0260] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0261] For example, and not as a limitation, memory 123 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 123 may include removable or non-removable (or fixed) media. Where appropriate, memory 123 may be internal or external to the integrated gateway device. In a particular embodiment, memory 123 is non-volatile solid-state memory. In a particular embodiment, memory 123 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only ROM (PROM), an erasable programmable read-only ROM (EPROM), an electrically erasable programmable read-only ROM (EEPROM), an electrically alterable read-only ROM (EAROM), or flash memory, or a combination of two or more of these. Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0262] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0263] Furthermore, the electronic device can be, for example, a computer, a server, or other electronic equipment.

[0264] This application also provides a chip for executing instructions, which is used to execute the technical solution of the three-dimensional perception model training method in the above embodiments.

[0265] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on the processor of an electronic device, the processor of the electronic device executes the technical solution of the three-dimensional perception model training method described in the above embodiments.

[0266] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product, which includes program code. When the program product is run on the processor of an electronic device, the program code is used to cause the processor of the electronic device to perform the steps in the methods of the various exemplary implementations of this application described above. For example, the electronic device can perform the three-dimensional perception model training method described in the embodiments of this application.

[0267] The program product may take the form of any combination of one or more readable media. A readable medium may be a readable data medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CDROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0268] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the three-dimensional perception model training method in the above embodiments.

[0269] It should be noted that, in addition to the specific embodiments described above, those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of describing the invention in conjunction with the implementation is to cover other options or modifications that may be derived from this application. To provide a thorough understanding of this application, many specific details are included in the above description, and this application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0270] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0271] It should be noted that the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0272] It should be noted that some structural or methodological features may be shown in the accompanying drawings in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0273] Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art should understand that the above description is a further detailed explanation of the application in conjunction with specific implementations, and should not be construed as limiting the specific implementation of the application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. A method for training a three-dimensional perception model, characterized in that, The method comprises: acquiring a two-dimensional image corresponding to different viewing angles; inputting the two-dimensional image into a three-dimensional perception model, so that the three-dimensional perception model obtains two-dimensional feature information corresponding to the two-dimensional image according to the two-dimensional image, converts the two-dimensional feature information to a three-dimensional space to obtain three-dimensional feature information, and renders the two-dimensional image according to the three-dimensional feature information to obtain a depth map and a color map corresponding to the two-dimensional image; obtaining a pseudo-depth map corresponding to the two-dimensional image according to the two-dimensional image; determining a training loss value of the three-dimensional perception model according to the two-dimensional image and the depth map, the color map and the pseudo-depth map corresponding to the two-dimensional image; training the three-dimensional perception model according to the training loss value until the three-dimensional perception model meets a target requirement, and ending the training of the three-dimensional perception model.

2. The three-dimensional perception model training method according to claim 1, wherein the method further comprises: determining a plurality of three-dimensional points on a plurality of light rays corresponding to different scattering directions with a light ray starting point as an origin in the three-dimensional space, and determining distance information of each three-dimensional point relative to the light ray starting point; obtaining three-dimensional coordinate information of each three-dimensional point according to coordinate information of the origin and scattering directions and the distance information corresponding to each three-dimensional point; then, rendering the two-dimensional image according to the three-dimensional feature information to obtain a depth map corresponding to the two-dimensional image comprises: obtaining three-dimensional feature information of each three-dimensional point from the three-dimensional feature information by a nearest neighbor interpolation method; determining a signed distance value of each three-dimensional point based on a first multi-layer perception machine according to the three-dimensional feature information and the three-dimensional coordinate information of each three-dimensional point; determining an opacity of each three-dimensional point according to the signed distance value of each three-dimensional point; determining a transmittance of each light ray reaching a corresponding three-dimensional point according to the opacity of each three-dimensional point on each light ray; determining a depth value of a pixel point corresponding to each light ray according to the distance information and the opacity of each three-dimensional point on each light ray and the transmittance of the light ray reaching the corresponding three-dimensional point; rendering the two-dimensional image according to the depth value of each pixel point to obtain the depth map corresponding to the two-dimensional image. The depth value of the pixel point is obtained by: 3.The three-dimensional perception model training method of claim 2, wherein, rendering the two-dimensional image according to the three-dimensional feature information to obtain a color map corresponding to the two-dimensional image comprises: Where, depth i w is the depth value of the i-th pixel. k s represents the depth weight of the k-th 3D point on the ray corresponding to the i-th pixel. k For the distance information of the k-th 3D point, T k α is the transmittance of the light reaching the k-th three-dimensional point. j α is the opacity of the ray from the ray's origin to the k-th 3D point. k For the opacity of the k-th 3D point, φ s () is the logic density distribution function, φ SDF (p k φ is the signed distance value of the k-th 3D point. SDF (p k+1 ) is the signed distance value of the (k+1)th three-dimensional point. 4.The three-dimensional perception model training method of claim 2 or 3, characterized in that, determining a first color value of each three-dimensional point on each light ray of the two-dimensional image based on a second multi-layer perception machine according to the three-dimensional feature information and the three-dimensional coordinate information of each three-dimensional point; determining a second color value of the pixel point corresponding to each light ray according to the first color value of each three-dimensional point on each light ray; rendering the two-dimensional image according to the second color value and the depth value of each pixel point to generate the color map. determining a training loss value of the three-dimensional perception model according to the two-dimensional image and the depth map, the color map and the pseudo-depth map corresponding to the two-dimensional image comprises: 5.The three-dimensional perception model training method of claim 4, wherein, ​ determine a photometric consistency loss value of the three-dimensional perception model according to the two-dimensional image and the depth map corresponding to the two-dimensional image; determine a rendering loss value of the three-dimensional perception model according to the two-dimensional image and the color map corresponding to the two-dimensional image; determine a relative proximity loss value of the three-dimensional perception model according to the pseudo-depth map corresponding to the two-dimensional image and the depth map; determine the training loss value of the three-dimensional perception model according to the photometric consistency loss value, the rendering loss value and the relative proximity loss value. 6.The three-dimensional perception model training method of claim 5, wherein, In the case that there are multiple frames of the two-dimensional images corresponding to each view angle, The method further comprises: determining a camera intrinsic parameter matrix and a camera extrinsic parameter matrix of a camera used for shooting the two-dimensional images; Then, the determination of the photometric consistency loss value of the three-dimensional perception model according to the two-dimensional image and the depth map corresponding to the two-dimensional image comprises: projecting each pixel point of a first frame of the two-dimensional image to a second frame of the two-dimensional image according to the depth map corresponding to the first frame of the two-dimensional image, the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, the first frame of the two-dimensional image and the second frame of the two-dimensional image being any two adjacent frames of the multiple frames of the two-dimensional images corresponding to each view angle; determining a third color value of each pixel point of the first frame of the two-dimensional image and a fourth color value of each pixel point of the second frame of the two-dimensional image after the projection; determining a target sampling point from the multiple pixel points, determining the photometric consistency loss value of the three-dimensional perception model according to the third color value and the fourth color value of the target sampling point. 7.The three-dimensional perception model training method of claim 6, wherein, The determination of the photometric consistency loss value of the three-dimensional perception model according to the third color value and the fourth color value of the target sampling point corresponding to the two-dimensional image comprises: determining a first loss value corresponding to each two-dimensional image according to the third color value and the fourth color value of the target sampling point corresponding to each two-dimensional image; determining a minimum loss value from the first loss values as the photometric consistency loss value of the three-dimensional perception model.

8. The three-dimensional perception model training method according to claim 7, wherein The determination of the rendering loss value of the three-dimensional perception model according to the two-dimensional image and the color map corresponding to the two-dimensional image comprises: determining a second loss value of each pixel point between the color map and the two-dimensional image according to the second color value of each pixel point in the color map and a third color value of the pixel point in the two-dimensional image; determining the rendering loss value of the three-dimensional perception model according to the second loss values of each pixel point.

9. The three-dimensional perception model training method according to claim 8, wherein The generation of the pseudo-depth map corresponding to the two-dimensional image according to the two-dimensional image comprises: generating the pseudo-depth map of the two-dimensional image according to the two-dimensional image based on a depth estimation network; The determination of the relative proximity loss value of the three-dimensional perception model according to the pseudo-depth map corresponding to the two-dimensional image and the depth map comprises: determine a plurality of corresponding pixel pairs in the depth map and the pseudo-depth map; determine a first relative proportion of each of the pixel pairs corresponding to the depth map according to a depth value of each of the pixel pairs in the depth map; determine a second relative proportion of each of the pixel pairs corresponding to the pseudo-depth map according to a depth value of each of the pixel pairs in the pseudo-depth map; determine the relative proximity loss value of the three-dimensional perception model according to the first relative proportion and the second relative proportion. 10.The three-dimensional perception model training method of claim 9, wherein, Each of the pixel pairs includes a first pixel and a second pixel, and the relative proximity loss value of the three-dimensional perception model is obtained by the following way: wherein a is the first pixel point, a' is the second pixel point, d(a) is a depth value of the first pixel point in the depth map, d(a') is a depth value of the second pixel point in the depth map, d'(a) is a depth value of the first pixel point in the pseudo depth map, d'(a') is a depth value of the second pixel point in the pseudo depth map, is the first relative proportion corresponding to the depth map for the pixel point pair, is the second relative proportion corresponding to the pseudo depth map for the pixel point pair, and s is a scaling factor, is a first constraint result of tri-classification constraint on the first relative proportion based on a Sigmoid function, is a second constraint result of tri-classification constraint on the second relative proportion based on the Sigmoid function. 11.The three-dimensional perception model training method of claim 10, wherein, convert the two-dimensional feature information to a three-dimensional space to obtain three-dimensional feature information, including: generating multi-scale two-dimensional image feature information according to the two-dimensional feature information based on a feature pyramid network; establishing a three-dimensional space, projecting voxels of the three-dimensional space to the two-dimensional image to obtain two-dimensional coordinate information corresponding to each of the projected pixels after projection; sampling and processing the multi-scale two-dimensional image feature information according to the two-dimensional coordinate information to obtain the three-dimensional feature information.

12. The three-dimensional perception model training method of claim 11, wherein: the two-dimensional image is an image in an autonomous driving data set and a navigation planning data set; The backbone network included in the three-dimensional perception model extracts features of the two-dimensional image to obtain the two-dimensional feature information, and an initial learning rate of the backbone network is 2x10 -4 ; The depth map is obtained by rendering the two-dimensional image according to the three-dimensional feature information by a rendering decoder included in the three-dimensional perception model, and the initial learning rate of the rendering decoder is 2x10 -5 ; the three-dimensional points in the two-dimensional image include a first preset number of the three-dimensional points and a second preset number of the three-dimensional points, wherein the first preset number of the three-dimensional points are points uniformly distributed in the two-dimensional image, and the second preset number of the three-dimensional points are important points in the two-dimensional image; the target requirement is that a training period of the three-dimensional perception model reaches a preset training period or a loss function converges.

13. An electronic device, comprising: The electronic device includes: a processor, and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to enable the electronic device to implement the three-dimensional perception model training method of any one of claims 1-12.

Images