Automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling

Abstract

The invention discloses an automatic driving scene three-dimensional reconstruction method based on multiple view angles and continuous depth modeling. The method comprises the following steps: realizing high-precision and real-time three-dimensional scene modeling by designing an LMV-NeRF model; according to the method, multi-view image data and a continuous depth representation technology are fused, camera parameters are utilized to align feature maps, and global consistency is enhanced through a cross-view attention mechanism; generating a group of depth values by adopting a hierarchical sampling strategy, predicting color and volume density in combination with a volume rendering formula, and finally generating a target image and a depth map under a new view angle; and obtaining a data set of the multi-view image in the automatic driving scene to train the model, and selecting an optimal model to obtain the target image in the new view. By fusing multi-view image data and a continuous depth representation technology, high-precision and real-time three-dimensional reconstruction of the automatic driving scene is realized, and the method is particularly suitable for dynamic environment perception and decision support in the automatic driving scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and three-dimensional reconstruction technology, and particularly relates to a three-dimensional reconstruction method for an automatic driving scene based on multi-view and continuous depth modeling. BACKGROUND

[0002] In recent years, automatic driving technology has developed rapidly worldwide. As one of the key technologies for automatic driving, environmental perception and scene reconstruction has always been a hot research topic. Environmental perception not only requires the system to recognize and understand the surrounding static objects (such as roads, traffic signs, buildings, etc.), but also needs to monitor the state of dynamic objects (such as other vehicles, pedestrians, etc.) in real time. Therefore, accurate and real-time three-dimensional scene reconstruction has become one of the core tasks in the automatic driving system.

[0003] NeRF (Neural Radiance Field) is a new type of three-dimensional modeling technology based on deep learning, which can generate high-quality scene rendering from different perspectives through neural networks. At present, three-dimensional reconstruction technology based on NeRF has made some progress. Researchers have proposed several variants of NeRF, such as fast NeRF, multi-view based NeRF, and NeRF-W (NeRF with Wild Scenes), aiming to improve its computational efficiency and adaptability in dynamic scenes. However, NeRF still faces high computational burden in practical applications, and has certain limitations in modeling complex dynamic scenes.

[0004] MPI (Multi-Scale Perception Information) technology has also received extensive attention, especially in the fusion of image and point cloud data. Existing MPI technology is mainly applied to three-dimensional reconstruction and lighting modeling of static scenes, but it relies on discrete depth representation and is difficult to handle weak texture areas, occlusions and dynamic changes in scenes, so its application in dynamic road traffic scenes is still limited. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provide a three-dimensional reconstruction method for an automatic driving scene based on multi-view and continuous depth modeling. By fusing multi-view image data and continuous depth representation technology, the present application realizes high-precision and real-time three-dimensional reconstruction of the automatic driving scene, and is particularly suitable for dynamic environmental perception and decision support in the automatic driving scene.

[0006] The purpose of the present application is achieved by the following technical solution: a three-dimensional reconstruction method for an automatic driving scene based on multi-view and continuous depth modeling, comprising the following steps: Step 1: design an LMV-NeRF model for three-dimensional reconstruction of the automatic driving scene; Step 2, use camera intrinsic and extrinsic parameters to perform geometric transformation to align multi-view feature maps to the same 3D space, and use cross-view attention mechanism to fuse feature maps of different views to obtain fused feature maps; Step 3, define the depth range of the view cone, and use a hierarchical sampling strategy to generate a set of arbitrary depth values; Step 4, generate target images and depth maps under new views according to the fused feature maps and depth values; Step 5, obtain a dataset of multi-view images in an autonomous driving scene, and divide the dataset into training set, validation set and test set; use the training set to train the LMV-NeRF model; use the validation set to select the optimal LMV-NeRF model; input the multi-view images in the test set into the optimal LMV-NeRF model to obtain target images under new views.

[0007] Further, the LMV-NeRF model is a model that reconstructs a continuous depth three-dimensional scene representation by inputting multi-view images, and generates target images and depth maps under new views; The LMV-NeRF model uses an encoder-decoder framework, wherein the encoder uses ResNet-50 to extract feature maps of each image, aligns feature maps of different views to the same 3D space using multi-view geometric principles, and then fuses multi-view feature maps through a cross-view attention mechanism; the input of the decoder includes fused feature maps and arbitrary depth values, and the decoder predicts color and volume density on each depth plane according to the fused feature maps and depth values to generate target images and depth maps under new views.

[0008] Further, in step 2, the multi-view feature maps are aligned to the same 3D space, and the process is as follows: wherein, is a camera intrinsic matrix composed of focal length and principal point position are focal lengths of the camera on the x-axis and y-axis, respectively; are coordinates of the principal point position on the x-axis and y-axis, respectively; is a homography matrix, used to describe the transformation relationship between two planes; is a rotation matrix, representing the rotation of the camera; is a plane vector representing the displacement of the camera; is a plane normal vector, and the superscript T is the transpose of the matrix or vector; ​Feature pixel coordinates representing source view, Feature pixel coordinates representing target view.

[0009] Further, in step 2, the features of different views are fused using a cross-view attention mechanism, the process is as follows: where, , and are learnable weight matrices, representing the weight matrices of queries, keys and values, respectively; represents the feature map of any view; , and represent the query vector, key vector and value vector, respectively; represents the query vector of the reference view ; represents the key vector of the view ; is the feature dimension; represents the attention weight between the view and the reference view; represents the softmax activation function; represents the value vector of the view ; represents the fused feature map; N represents the total number of views.

[0010] Further, in step 3, a set of arbitrary depth values is generated using a hierarchical sampling strategy, the process is as follows: where, near plane represents the closest visible depth to the camera, and far plane represents the farthest visible depth to the camera; is a Gaussian noise, , is the intensity of the controlled disturbance; represents the i-th depth value, represents the i-th disparity value.

[0011] Further, in step 4, the target image and depth map under the new view are generated, the process is as follows: wherein, denotes the color value of the pixel r, denotes the depth value of the pixel r; denotes the distance between neighboring sampling points, denotes the depth value the volume density at the corresponding position, denotes the depth value the color at the corresponding position; denotes the radiance transmittance of the sampling point at the i-th depth.

[0012] Further, the multi-view image dataset in the autonomous driving scene includes a KITTI dataset, a RealEstate10K dataset, and a nuScenes dataset.

[0013] Further, the training of the LMV-NeRF model using the training set specifically includes: inputting multiple multi-view images in the training set into the LMV-NeRF model, outputting a target image and a depth map in a new view, calculating a pixel-level difference between the target image and a real target view image corresponding to the target image as a loss function, taking minimization of the loss function as an optimization objective, adjusting parameters of the LMV-NeRF model until a preset training round is reached, and obtaining a trained LMV-NeRF model.

[0014] Compared with the prior art, the present application has the following advantages: (1) The present application proposes a three-dimensional reconstruction method (LMV-NeRF) for an autonomous driving scene based on multi-view input and continuous depth modeling. Compared with traditional discrete depth representation (such as MPI), the present application significantly improves the adaptability to weak texture areas, occluded areas and dynamic scenes through a continuous depth modeling method, and realizes higher-precision three-dimensional reconstruction.

[0015] (2) The present application adopts a hierarchical sampling strategy to optimize the calculation efficiency, adopts hierarchical sampling combined with a random disturbance mechanism, significantly reduces the calculation burden while ensuring high-quality rendering, and makes the model more suitable for the application scenario of autonomous driving environment perception which requires high real-time performance.

[0016] (3) The present application innovatively introduces a cross-view attention mechanism, enhances multi-view consistency through cross-view feature fusion, effectively solves the alignment problem of geometric and texture information under multi-view input, and further improves the robustness and precision of global scene modeling. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flow chart of the automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling of the present application; Figure 2 is a structural schematic diagram of the LMV-NeRF model of the present application. DETAILED DESCRIPTION

[0018] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers in the attached drawings refer to the same or similar elements. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims. It is understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0020] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish one from another. For example, a first information can be termed a second information, and similarly, a second information can also be termed a first information, without departing from the scope of the present application. As used herein, the word "if' can be construed to mean "when" or "in response to determining" depending on the context. Also, the term "example" is used herein to mean "serving as an example, instance, or illustration," and not to imply or warrant that it is necessarily the best. Additionally, the term "in response to" is used to indicate a determination by one or more processing units of a computing device, rather than a direct mechanical response to a stimulus. The term "include" and derivatives thereof, mean "comprising," "including," "containing," or "involving," and are not intended to be construed as exclusive or as limiting. Moreover, the term "or" is used in the inclusive sense, unless otherwise indicated.

[0021] The present application will be described in detail below with reference to the attached drawings. The features of the embodiments and implementation described below can be combined with each other, provided that there is no conflict.

[0022] Reference is made to Figure 1and Figure 2 The method for 3D reconstruction of autonomous driving scenes based on multi-view and continuous depth modeling of the present invention specifically includes the following steps: Step 1: Design an LMV-NeRF model for 3D reconstruction of autonomous driving scenarios.

[0023] In this embodiment, the LMV-NeRF model is a model that reconstructs a continuous-depth 3D scene representation from input multi-view images and generates target images and depth maps from the new viewpoints. The LMV-NeRF model is implemented using an encoder-decoder framework, such as... Figure 2 As shown, the encoder uses ResNet-50 to extract feature maps for each image, aligns feature maps from different perspectives to the same 3D space using multi-view geometry, and then fuses the feature maps from multiple perspectives through a cross-view attention mechanism to enhance global consistency. The decoder predicts the color and volume density on each depth plane based on the fused feature map and depth value. The decoder's input includes the fused feature map and arbitrary depth value, and finally generates the target image and depth map under the new perspective.

[0024] Step 2: Use camera intrinsic and extrinsic parameters to perform geometric transformations to align feature maps from multiple perspectives to the same 3D space, and use a cross-view attention mechanism to fuse feature maps from different perspectives to obtain the fused feature map.

[0025] Furthermore, the feature maps from multiple perspectives are aligned to the same 3D space, as follows: In the formula, It is determined by focal length and the location of the main point The camera intrinsic parameter matrix is ​​composed of These are the camera's focal lengths on the x and y axes, respectively. These are the coordinates of the principal point on the x-axis and y-axis, respectively. It is the calculated homography matrix, used to describe the transformation relationship between two planes; It is a rotation matrix that represents the camera's rotation; It is a planar vector representing the camera's displacement; It is a plane normal vector, usually set as The superscript T represents the transpose of a matrix or vector; The feature map pixel coordinates represent the source viewpoint. The pixel coordinates of the feature map representing the target's viewpoint are used. The pixel coordinates of the feature map from the source view can be... Mapping feature image pixel coordinates to target view .

[0026] Further, the feature maps of different views are fused using a cross-view attention mechanism, the process of which is as follows: wherein, , and are learnable weight matrices, representing the weight matrices of queries, keys and values, respectively; represents the feature map of any view; , and represent the query vector, key vector and value vector, respectively; represents the query vector of the reference view ; represents the key vector of the view ; is the feature dimension; represents the attention weight between the view and the reference view; represents the softmax activation function; represents the value vector of the view ; represents the fused feature map; N represents the total number of views. Specifically, for the feature map of each view , it is mapped to the query vector , the key vector and the value vector using linear transformation; a view is selected as the reference view, and the dot product is calculated between the query vector of the reference view and the key vector of other views to obtain the attention weight, wherein is the feature dimension, used to scale the dot product result to avoid gradient explosion. The softmax activation function normalizes the similarity into a probability distribution, which is used to weight the feature maps of different views.

[0027] Step 3, define the depth range of the view pyramid, and adopt a hierarchical sampling strategy to generate a set of arbitrary depth values.

[0028] Further, a set of arbitrary depth values are generated by using a hierarchical sampling strategy, which is as follows: where near plane represents the closest visible depth to the camera, far plane represents the farthest visible depth to the camera. is a Gaussian noise, , is the strength of the control disturbance. represents the i-th depth value, represents the i-th disparity value. The hierarchical sampling strategy divides the depth range into multiple intervals and adds random disturbance in each interval to calculate the corresponding depth value , and then converts the depth value to the disparity value .

[0029] Step 4, generating the target image and depth map in the new view according to the fused feature map and depth value.

[0030] Further, the target image and depth map in the new view are generated, which is as follows: where, represents the color value of pixel r, represents the depth value of pixel r. represents the distance between adjacent sampling points, represents the volume density at the corresponding position of the depth value , represents the color at the corresponding position of the depth value . represents the light transmittance of the sampling point at the i-th depth. Specifically, first, the depth plane of the source view is converted to the target view using the homography matrix ; then for each depth value , get N sampling points on each light ray; then according to each disparity value , predict the color and volume density at the corresponding position, and calculate the light transmittance of the sampling point at each depth Finally, according to the volume density and the light transmittance, the contribution of each sampling point to the final color and depth value is obtained, the final pixel color and depth value are obtained by weighted summation of the contributions of these sampling points, and then the target image and depth map are constructed.

[0031] Step 5, obtain the multi-view image dataset in the autonomous driving scene, and divide the dataset into a training set, a validation set and a test set; use the training set to train the LMV-NeRF model; use the validation set to select the optimal LMV-NeRF model; input the multi-view images in the test set into the optimal LMV-NeRF model to obtain the target image in the new view.

[0032] Further, the multi-view image dataset in the autonomous driving scene includes the KITTI dataset, the RealEstate10K dataset, the nuScenes dataset, etc.

[0033] Further, taking the KITTI dataset as an example, the KITTI dataset is used to train the LMV-NeRF model. Specifically, the KITTI dataset contains 7481 training images and 7518 test images, and provides binocular stereo images for stereo matching and depth estimation. Among them, 7481 images constitute a training set for training; 5000 images constitute a validation set for selecting the optimal LMV-NeRF model; 2518 images constitute a test set for offline evaluation, and output the target image in the new view. When the training set is used to train the LMV-NeRF model, the training process specifically includes: inputting multiple multi-view images into the LMV-NeRF model, using the LMV-NeRF model to predict the 4-channel image (RGB and volume density) of multiple depth planes in the new view, and then obtaining the target image and depth map in the new view through volume rendering; the pixel-level difference between the target image and the real target view image is calculated as a loss function, and the calculation formula is: In the formula, is the loss function, H and W are the height and width of the image respectively, represents the pixel of the target image in the new view output by the LMV-NeRF model in the coordinate, represents the pixel of the real target view image in the coordinate; the optimization objective is to minimize the loss function, the AdamW optimizer is used to update the parameters of the LMV-NeRF model, the learning rate is set to 0.0002, and a total of 500000 steps are trained.

[0034] Of course, the application can also use the RealEstate10K, nuScenes and other multi-view automatic driving scene commonly used data sets to train the LMV-NeRF model.

[0035] The final effect of the application is to input multi-view images into the optimal LMV-NeRF model, so as to obtain the neural implicit representation of the scene and generate target images under new perspectives, as shown in the effect of Figure 2 It should be noted that after the multi-view images are input, the feature map extraction and synthesis of the encoder, the LMV-NeRF model can predict the images and depth maps under new perspectives. This capability is the neural implicit representation of the scene, and it is suitable for any perspective.

[0036] The above embodiments are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1.A method for three-dimensional reconstruction of an autonomous driving scene based on multi-view and continuous depth modeling, characterized in that, The method comprises the following steps: Step 1, designing an LMV-NeRF model for 3D reconstruction of an autonomous driving scene; Step 2, using camera intrinsic and extrinsic parameters for geometric transformation to align multi-view feature maps to the same 3D space, and using a cross-view attention mechanism to fuse feature maps of different views to obtain fused feature maps; Step 3, defining the depth range of the view pyramid and using a hierarchical sampling strategy to generate a set of arbitrary depth values; Step 4, generating target images and depth maps under new views according to the fused feature maps and depth values; Step 5, obtaining a multi-view image dataset in an autonomous driving scene, and dividing the dataset into a training set, a validation set and a test set; training the LMV-NeRF model using the training set, selecting the optimal LMV-NeRF model using the validation set, and inputting multi-view images in the test set into the optimal LMV-NeRF model to obtain target images under new views. 2.The automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling according to claim 1, characterized in that, The LMV-NeRF model is a model that reconstructs a continuous depth 3D scene representation from input multi-view images and generates target images and depth maps under new views; The LMV-NeRF model uses an encoder-decoder framework, wherein the encoder uses ResNet-50 to extract feature maps of each image, aligns feature maps of different views to the same 3D space using multi-view geometric principles, and fuses multi-view feature maps through a cross-view attention mechanism; the input of the decoder includes fused feature maps and arbitrary depth values, and the decoder predicts the color and volume density on each depth plane according to the fused feature maps and depth values to generate target images and depth maps under new views. 3.The automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling according to claim 1, characterized in that, In step 2, the multi-view feature maps are aligned to the same 3D space as follows: In the formula, is the camera intrinsic matrix composed of focal length and principal point position ; are the focal lengths of the camera on the x-axis and y-axis respectively; are the coordinates of the principal point position on the x-axis and y-axis respectively; is a homography matrix, used to describe the transformation relationship between two planes; is a rotation matrix, representing the rotation of the camera; is a plane vector representing the displacement of the camera; is a plane normal vector, and the superscript T is the transpose of the matrix or vector; represents the feature image pixel coordinates of the source view, represents the feature image pixel coordinates of the target view. 4.The automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling according to claim 1, characterized in that, In step 2, the cross-view attention mechanism is used to fuse feature maps of different views as follows: wherein, , and are learnable weight matrices, representing the weight matrices of queries, keys and values, respectively; represents the feature map of any view; , and represent the query vector, key vector and value vector, respectively; represents the query vector of the reference view ; represents the key vector of the view ; is the feature dimension; represents the attention weight between the view and the reference view; represents the softmax activation function; represents the value vector of the view ; represents the fused feature map; N represents the total number of views. 5.The automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling according to claim 1, characterized in that, In step 3, a set of arbitrary depth values is generated using a hierarchical sampling strategy as follows: where near plane represents the closest visible depth to the camera, far plane represents the farthest visible depth to the camera; is a Gaussian noise, , controls the strength of the perturbation; represents the i-th depth value, represents the i-th disparity value. 6.The automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling according to claim 1, characterized in that, In step 4, target images and depth maps under new views are generated as follows: wherein denotes the color value of pixel r, denotes the depth value of pixel r; denotes the distance between neighboring sample points, denotes the depth value the volume density at the corresponding position, denotes the depth value the color at the corresponding position; denotes the radiance of the sample point at the i-th depth. 7.The automatic driving scene three-dimensional reconstruction method based on multi-view and continuous depth modeling according to claim 1, characterized in that, The multi-view image dataset in the autonomous driving scene includes KITTI dataset, RealEstate10K dataset and nuScenes dataset. 8.The method of claim 1, wherein, Training the LMV-NeRF model using the training set specifically includes: inputting multiple multi-view images in the training set into the LMV-NeRF model to output target images and depth maps under new views; calculating the pixel-level difference between the target images and their corresponding real target view images as a loss function, taking the minimization of the loss function as the optimization goal, adjusting the parameters of the LMV-NeRF model until a preset training round is reached to obtain the trained LMV-NeRF model.

Images