A NeRF 3D reconstruction method and system based on 3D point cloud

Abstract

The present invention proposes a NeRF three-dimensional reconstruction method and system based on 3D point cloud. The system integrates 3D point cloud data and image information collected by laser radar and high-definition camera, and inputs them into the improved NeRF model for training to build a high-precision three-dimensional model of the scene. The improved NeRF model integrates explicit and implicit scene representations, uses hash grid coding to independently model volume density and color attributes, and uses a multi-layer perceptron (MLP) to refine the color attributes, thereby improving the model construction efficiency and rendering quality. During the automatic parking operation stage, the vehicle control system performs accurate path planning based on the 3D model output by the improved NeRF model to ensure the safety and accuracy of automatic parking. The implementation of the intelligent parking lot assisted automatic parking system optimizes the automatic parking process and improves operational efficiency and user experience.

Description

Technical Field

[0001] The present invention relates to the field of 3D modeling, and in particular to a NeRF three-dimensional reconstruction method and system based on 3D point cloud. Background Art

[0002] In the field of 3D scene reconstruction, although technologies such as stereo vision and laser scanning have made progress in obtaining accurate 3D information, these methods face challenges in processing large-scale data sets and dynamic scenes. They usually require complex pre-processing and post-processing steps, as well as a large amount of computing resources, which not only increases the deployment cost of the system, but also limits the real-time nature of 3D scene updates. As a new 3D reconstruction method based on deep learning, NeRF (Neural Radiance Fields) technology models the volume density and color of the scene through a neural network, and synthesizes high-quality images of new perspectives from a sparse collection of images. However, NeRF technology still faces challenges in processing large-scale scenes and complex lighting conditions. Its training process requires a large amount of image data and computing resources, and complex neural network inference needs to be performed for each new perspective rendering, which takes dozens of hours, which limits its feasibility in real-time or near-real-time applications.

[0003] In the process of urbanization, the effective management of parking facilities has become one of the key factors in improving the quality of urban life. However, traditional parking management systems are mostly based on manual operations, which have significant limitations in processing large-scale data and real-time monitoring. Manual management is not only inefficient, but also prone to errors in the process of data recording and information transmission, which reduces the overall operational efficiency of the parking lot and affects the user's parking experience.

[0004] In terms of automatic parking technology, although on-board sensors such as ultrasonic radar and optical cameras have played a role in assisting driving and parking, their perception and accuracy in complex environments still need to be improved. These sensors are usually limited by their working principles and physical characteristics, such as performance degradation in bad weather conditions or detection blind spots in obstructed vision, which limit the application of automatic parking systems in diverse parking scenarios.

[0005] Therefore, current research and technology development needs focus on how to improve the intelligence level of parking lots and how to achieve accurate modeling and real-time updates of parking lot scenes through advanced sensor fusion and 3D reconstruction technology. This requires not only solving the limitations of traditional technologies, but also developing new algorithms and system architectures to support efficient and reliable automatic parking solutions.

[0006] In addition, if the time for 3D reconstruction using NeRF can be greatly reduced, and the NeRF-based 3D reconstruction is applied to the simulation modeling of laparoscopic surgical environments, the degree of image simulation can be greatly improved, reducing the difficulty of surgical implementation; when the NeRF-based 3D reconstruction is applied to the simulation modeling of unknown spatial scenes, it can assist operators in obtaining the fine features of unknown spatial scenes under unmanned conditions, and assist robots in performing dangerous operations. Summary of the invention

[0007] Technical Problems to be Solved by the Invention The traditional 3D reconstruction technology faces challenges such as long training time and slow rendering speed when processing complex scenes. The present invention aims to improve the speed and quality of 3D reconstruction of complex scenes by introducing an improved NeRF model, so as to improve accuracy and efficiency in scenes such as automatic parking of vehicles.

[0008] Technical solution introduction: The present invention provides a NeRF three-dimensional reconstruction method and system based on 3D point cloud, which mainly includes the following steps:

[0009] The grid space of the scene, and the sampling points along the rays extending from the camera are used to jointly train a volume density multilayer perceptron and a color multilayer perceptron to obtain pixel colors rendered by all rays to model the scene;

[0010] During the joint training, the grid space is segmented at a first resolution and consists of voxel grid units of the first resolution, or is segmented at a second resolution and consists of voxel grid units of the second resolution, wherein the second resolution is a resolution selected from a plurality of resolutions including the first resolution;

[0011] The volume density multilayer perceptron receives the number of point clouds mapped to each voxel grid unit of the first resolution as input, and generates a volume density at the sampling point;

[0012] The color multi-layer perceptron receives the mean point coordinates of the point cloud clusters mapped to each voxel grid unit of the second resolution and the index of the sampling point in the grid space composed of the voxel grid of the second resolution as input, generates a color feature vector at the sampling point in the voxel grid of the second resolution, and the color multi-layer perceptron also splices the color feature vectors of all the second resolution levels to obtain the color embedding feature of the sampling point;

[0013] generating a final color of the sampling point using the color embedding features of the sampling point and the camera data during the joint training;

[0014] During the joint training, the final color of the sampling point, the volume density at the sampling point, and the distance between adjacent sampling points on the ray are also used to obtain the pixel color rendered by the ray.

[0015] Optionally, the sampling points include first sampling points and second sampling points, the first sampling points are evenly spaced along the ray, the linear density of the second sampling points in each unit segment of the ray along the ray is positively correlated with the number of point clouds mapped from the point cloud data to each voxel grid unit of the first resolution close to each unit segment of the ray, the number of voxel grid units of the first resolution passed through each unit segment of the ray is equal, and the second sampling points in each unit segment of the ray are evenly spaced along the unit segment.

[0016] Optionally, the sampling points include first sampling points and second sampling points, the first sampling points are evenly spaced along the ray, the linear density of the second sampling points in each unit segment of the ray along the ray is positively correlated with the number of point clouds mapped from the point cloud data to each voxel grid unit of the first resolution close to each unit segment of the ray, the number of voxel grid units of the first resolution passed through each unit segment of the ray is equal, and the second sampling points in each unit segment of the ray are evenly spaced along the unit segment.

[0017] Optionally, it also includes generating the volume density at the sampling point by interpolating the encoded values ​​at nearby grid points in the grid space composed of voxel grids of the first resolution, wherein the encoded values ​​are the sum of 1 / 8 of the number of point clouds in the voxel grid cells of the first resolution mapped to the nearby grid point in the grid space composed of voxel grid cells of the first resolution.

[0018] Optionally, it also includes generating a color feature vector at the sampling point by interpolating the encoded values ​​at nearby grid points in the grid space composed of voxel grids of the second resolution, wherein the encoded values ​​are the sum of 1 / 8 of the mean point coordinates of the point cloud cluster in the voxel grid cells of the second resolution mapped to the nearby grid points in the grid space composed of voxel grid cells of the second resolution.

[0019] Optionally, the method further includes updating the volume density multilayer perceptron and the color multilayer perceptron based on a total loss function.

[0020] Optionally, generating the volume density at the sampling point comprises the following steps:

[0021] Step L1. Using the point cloud data obtained from the LiDAR sensor after performing noise filtering, each point cloud point is mapped to the nearest voxel grid v in the voxel grid unit at the first resolution:

[0022] σ 0 (v) = count(P pc ∩v);

[0023] Among them, σ 0 (v) is the initial density value at voxel grid v, P pc is the point cloud set after noise filtering, count() is a counting function used to calculate the number of point cloud points mapped to the voxel grid v;

[0024] Step L2. Set the above initial density value σ 0 (v) using a volume density encoder based on hash grid coding as input, dividing the encoded volume density feature vector into eight equal parts and distributing them to each vertex of the voxel grid v as the volume density feature vector component, wherein the eight vertices of each voxel grid v respectively store the sum of the volume density feature vector components of the voxel grid v and the surrounding adjacent voxel grids;

[0025]

[0026] in, is the volume density feature vector set component of each vertex of the voxel grid v, σ 0 (v) is the voxel density value of the voxel grid v, It is a multi-layer perceptron that uses hash grid encoding volume density features;

[0027] Step L3. The voxel grid where the sampling point is located is calculated based on the coordinates of the sampling point. Based on the sum of the volume density feature vector components stored at the vertices of the voxel grid obtained in step L2, the volume density σ(x) of each sampling point is calculated using trilinear interpolation:

[0028] index = h(x);

[0029]

[0030] In the above formula, h is the hash function, index is the index of the voxel grid unit where the sampling point is located, σ(x) represents the volume density of the sampling point, and x = (x, y, z) represents the position coordinates of a given sampling point. The sum of the volume density feature vector components of the voxel grids around each vertex of the voxel grid is determined by the voxel grid unit index index where the sampling point is located.

[0031] Optionally, the color multilayer perceptron receives the mean point coordinates of the clusters mapped to each voxel grid unit of the second resolution and the index of the sampling point in the grid space composed of the voxel grid units of the second resolution as input, and generates the color feature vector at the sampling point in the voxel grid unit of the second resolution, including:

[0032] Step T1. Define L different voxel grids of the second resolution Point Cloud Collection pc Mapping to the voxel grid unit at the second resolution, and taking the mean point coordinates of the point cloud cluster in the voxel grid unit as the input of the color embedding feature encoder at different second resolutions l, the color embedding feature encoder encodes the mean point coordinates of the point cloud cluster mapped to the voxel grid v based on the encoding method of the hash grid, and divides the color embedding feature vector obtained after encoding into eight parts and distributes them to each vertex of the voxel grid as the color embedding feature vector component, and the eight vertices of each voxel grid v respectively store the sum of the color embedding feature vector components of the main voxel grid v and the surrounding adjacent voxel grids;

[0033]

[0034]

[0035] in, is the mean point coordinate of the point cloud cluster in the voxel grid, average() is the function for solving the mean point coordinate of the point cloud cluster, is the color embedding feature vector component of each vertex of the voxel grid v, Multilayer Perceptron for color embedding features using hash grid encoding;

[0036] Step T2. Calculate the voxel grid where the sampling point is located at the second resolution l through a hash function based on the sampling point coordinates, and obtain its index index l , index l =h(x / s l );

[0037] index l is the sampling point index, h is the hash function, s l is the grid size scaling factor at the second resolution l;

[0038] Step T3. Find the sum of the color embedding feature vector components at each vertex of the voxel grid in the hash table, and use trilinear interpolation to obtain the color feature vector f of the sampling point x at resolution l l (x);

[0039]

[0040] The index of the voxel grid where the sampling point is located l Determine the sum of color embedding feature vector components of the voxel grid around each vertex of the voxel grid;

[0041] Step T4. Concatenating all color feature vectors of the second resolution level to obtain the color embedding features of the sampling points includes:

[0042]

[0043] concat is a concatenation operation that merges the color embedding feature vectors from different second-resolution voxel grids to obtain the color embedding feature c f .

[0044] Optionally, the camera data from the camera includes an image of the scene and camera pose information: the camera pose information includes the position of the camera and the viewing direction of the camera: (x c ,y c ,z c ,d), where d = (θ, φ) is the camera’s viewing angle, θ is the horizontal direction, φ is the vertical direction, (x c ,y c ,z c ) is the position of the camera;

[0045] Generating a final color of the sampling point using the color embedding feature of the sampling point and the camera data during the joint training includes:

[0046] Based on the obtained color embedding feature c of the sampling point f , two independent multi-layer perceptrons are used to obtain the viewpoint-dependent color c vd and the viewpoint-independent color c vi , merge c vd and c vi Get the final color value c of a given sampling point:

[0047] c=M(c f ,d);

[0048] c=c vd +c vi ;

[0049] C vd =M vd (c f ,d);

[0050] c vi =M vi (c f );

[0051] Among them, c represents the final color value of a given sampling point, M is a multi-layer perceptron including M vd 、M vi ;

[0052] The NeRF model randomly selects a subset of pixels in the training image at each iteration of training, generates a 3D ray r for each pixel, and calculates the volume density σ at the position of the i-th sampling point for the N sampling points on the ray r. i , the final color value c i .

[0053] Optionally, during the joint training, further using the final color of the sampling point, the volume density at the sampling point, and the distance between adjacent sampling points on the ray to obtain the pixel color rendered by the ray includes:

[0054] The pixel color for ray r is calculated using the following function:

[0055]

[0056]

[0057] δ i =(x i+1 -x i );

[0058] in, represents the pixel color rendered along ray r, N is the number of sampling points on the ray, T i represents the transmittance of the i-th sampling point, σ i represents the volume density of the i-th sampling point, δ i represents the distance between the i-th sampling point and the adjacent sampling point, c i Represents the final color of the i-th sampling point.

[0059] Optionally, the overall loss function during the NeRF network training is:

[0060] L=L p +λL r ;

[0061]

[0062]

[0063]

[0064]

[0065] Among them, L represents the total loss function, L p Represents the weighted luminosity loss, Lr represents the regularization term, λ represents the hyperparameter, Q is the number of ray samples, r j represents the jth ray, represents the dynamic weight based on the color difference of the sampling point on the jth ray, τ j is the weight coefficient, c vd,k represents the view-dependent color at the kth sampling point, H is the number of all sampling points on all rays; the weighted photometric loss is used to minimize the predicted color along ray r j The rendered pixel color The real color C(r j ), and the dynamic weight Through the clamp() function, the weight coefficient τ j Restricted to a specific range (usually between [1,10]) to amplify the loss of sampling points with large color differences; L r The regularization term is used to minimize the sum of the absolute values ​​of the view-dependent colors.

[0066] Optionally, updating the volume density multilayer perceptron and the color multilayer perceptron based on the total loss function includes:

[0067] Calculate L p Weighted photometric loss and L r The regularization term is about the volume density encoder parameters θ and the color embedding feature encoder parameters The loss gradient of the volume density encoder parameter θ is obtained and color embedding feature encoder parameters The total loss gradient The RAdam optimization algorithm is used to update the volume density encoder parameters θ and the color embedding feature encoder parameters Then the volume density and color embedding features at the sampling points are dynamically optimized.

[0068] Compared with the prior art, the present invention can significantly improve the speed and quality of three-dimensional reconstruction of parking lot scenes through the improved NeRF model, which is helpful for fast and accurate vehicle automatic parking control, and optimizes the operational efficiency and user experience of parking lots. In addition, the system of the present invention can adapt to different parking lot environments and vehicle types, and has good versatility and scalability. Through the real-time updated three-dimensional model, the system can provide accurate environmental information for the vehicle, reducing the uncertainty and time consumption in the parking process. In general, the present invention provides an innovative solution for the field of intelligent transportation systems, which helps to promote the development and application of intelligent parking lot technology. By introducing the YOLOv4 network, higher accuracy and real-time performance are achieved in vehicle detection and classification. Lidar point cloud data provides prior information about the geometric structure of the scene, which helps the model to reconstruct the scene faster and more accurately. At the same time, the introduction of the improved NeRF model makes the three-dimensional scene reconstruction of the parking lot more efficient, and provides accurate three-dimensional environmental information for automatic parking. The combination of these technologies makes the present invention have significant advantages and application prospects in the field of intelligent parking lot management. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of the three-dimensional reconstruction principle of the improved NeRF of the present invention. Specific implementation plan

[0070] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0071] Conventional Neural Radiance Fields (NeRF) are neural networks trained to implicitly represent a particular scene. Once a NeRF is trained on a scene, it can be used to generate new views of the scene from perspectives not included in the training data. Conventionally, the training data is simply multiple images of the scene, and the trained NeRF takes an image of the scene, the 3D position of the camera, and the camera's viewing direction: (x c ,y c ,z c ,d), as input, generates a 3D ray r extending from the camera for each pixel of the scene image, and outputs the scalar volume density σ at the sampling point on the ray and the color c emitted toward the viewing direction at the sampling point, and projects the output color and density onto the image to achieve view synthesis. Since stereo rendering is differentiable, the optimization of the neural network requires a series of scene images that determine the camera pose. Train NeRF for specific scenes. If you are interested in different scenes, you need to train a new NeRF. Where d = (θ, φ) is the viewing direction of the camera, θ is the horizontal direction, φ is the vertical direction, (x c ,y c ,z c ) is the position of the camera.

[0072] Training NeRF is to optimize a deep fully connected neural network, which is also called a multilayer perceptron (MLP). MLP is a fully connected feedforward artificial neural network. Each MLP includes an input layer, at least one hidden layer and an output layer. The layer is composed of nodes. The nodes in each layer receive the output from the nodes in the previous layer as input, and the nodes in the previous layer start from the input layer and end at the output layer. Each connection between nodes in adjacent layers has a weight. Each node has an activation function, which takes the weighted input of the node as its independent variable. MLP is fully connected because each node in a layer is connected to each node in the adjacent layer. Training MLP results in changing the weights via backpropagation.

[0073] The present invention provides a method for assisting automatic parking in an intelligent parking lot based on improved NeRF, comprising:

[0074] S101, using the YOLOv4 target detection and classification network to detect and classify vehicles in the monitoring image sequence at the entrance of the parking lot in real time, and allocate parking spaces, send instructions containing parking space location information to the vehicles through the wireless communication module, and use the path planning algorithm to plan a driving route for the vehicles;

[0075] S102, calling a laser radar device to obtain 3D point cloud data of the parking area, and using a high-definition camera to take a set of pictures of the area to achieve data acquisition and preprocessing;

[0076] S103, inputting the 3D point cloud data and the image set into the improved NeRF model for training to generate a detailed 3D model of the parking area, wherein the model adopts an efficient explicit and implicit hybrid voxel representation method combined with hash grid coding to improve the generation speed and quality of the 3D model;

[0077] S104, combining the 3D model constructed by the improved NeRF, using the existing vision-based automatic parking technology to plan the path of automatic parking for the vehicle, thereby realizing automatic parking of the vehicle.

[0078] S101: Use the YOLOv4 algorithm to perform real-time detection and classification tasks on vehicles in the surveillance image sequence.

[0079] The YOLOv4 algorithm is an efficient target detection framework designed to maintain high detection accuracy while improving processing speed. The core architecture of the algorithm includes the following key parts:

[0080] a. Backbone network: YOLOv4 uses CSPDarknet53 as its backbone network for feature extraction. The network optimizes computational efficiency through the CSPNet structure while maintaining the depth of the network to extract rich image features. CSPDarknet53 consists of continuous convolutional layers, depth-separable convolutions, and residual connections, which can effectively extract spatial features from input images.

[0081] b. Feature pyramid: In order to effectively detect vehicles of different sizes, YOLOv4 introduces multi-scale feature maps. These feature maps are obtained by applying convolution operations at different levels of the network and can capture vehicle features of different spatial scales.

[0082] c. Anchor boxes: In each cell of the feature map, YOLOv4 predicts a set of anchor boxes. These anchor boxes are generated according to predefined sizes and ratios, aiming to match vehicles of different sizes, thereby enhancing the accuracy of vehicle positioning.

[0083] d. Prediction layer: At multiple levels of the network, YOLOv4 performs classification and location prediction of vehicles. Each prediction layer is responsible for predicting vehicles within a specific scale range, and the output includes bounding box coordinates, size, category probability, and probability of vehicle existence.

[0084] e. Loss function: YOLOv4 uses a comprehensive loss function that combines bounding box coordinate loss, category loss, and confidence loss to optimize the overall performance of vehicle detection. The loss function can be expressed as:

[0085] L=∑ i [L coord (x i ,y i ,w i ,h i )+L class (c i )+L conf (p i )]

[0086] Among them, L coord , L class and L conf They represent coordinate loss, category loss, and confidence loss respectively.

[0087] f. Training strategy: During the training process, YOLOv4 adopts a variety of strategies including data enhancement, network structure adjustment, gradient accumulation and adaptive learning rate, aiming to further improve the performance and generalization ability of the model. The adaptive learning rate adjustment strategy can be described by the following formula:

[0088]

[0089] Among them, LR is the learning rate, LR 0 is the initial learning rate, t is the current number of training steps, and T is the scheduled number of training steps.

[0090] Step 1.1: Use the trained YOLOv4 object detection and classification network to perform real-time analysis on the surveillance image sequence to identify and classify vehicles entering the parking lot. YOLOv4 extracts image features through its backbone network CSPDarknet53 and uses the feature pyramid structure to detect vehicles at different scales. The anchor boxes predicted by the network are adjusted for different models such as sedans, SUVs (Sport Utility Vehicles), and commercial vehicles (MPV / Multi-Purpose Vehicles) to ensure accuracy.

[0091] Step 1.2: The classification result is used to assign an appropriate parking space to the vehicle, and the parking space location information is transmitted to the vehicle through the network communication module.

[0092] Step 1.3: Plan the optimal driving route based on the vehicle's current position and target parking space based on the path planning algorithm.

[0093] S102: This step is based on the vehicle detection classification and path planning results, and aims to conduct detailed mapping of the three-dimensional spatial data of the parking area to ensure that the parking assistance system can accurately understand the parking environment. This step covers the following main contents:

[0094] Step 2.1: In this step, the parking area determined in step S101 is fully scanned by a laser radar device to capture high-precision three-dimensional point cloud data. This data constructs a set of three-dimensional coordinate points reflecting the geometric structure of the parking space and its surrounding environment by accurately measuring the distance from the laser radar to the surrounding objects.

[0095] At the same time, this step uses a high-resolution camera to take full-scale photos of the parking area from multiple angles to obtain a series of high-quality images; and uses colmap to estimate the camera pose based on the images to obtain the camera pose information in each image. The camera pose information includes the camera position and the camera viewing direction: (x c ,y c ,z c ,d), where d = (θ, φ) is the camera’s viewing angle, θ is the horizontal direction, φ is the vertical direction, (x c ,y c ,z c ) is the position of the camera.

[0096] Step 2.2: Based on the obtained 3D point cloud data, noise filtering is first performed. The mean filtering algorithm is used to identify and remove noise points caused by equipment errors or external environmental factors to enhance the accuracy and overall quality of the data.

[0097] Point cloud mean filtering can be expressed as performing the following operations on each point in the point cloud set:

[0098]

[0099]

[0100] Where P is any point in the point cloud data, and its coordinates can be (x p ,y p ,z p ) indicates that a filtering window W is defined, which is centered on P. All adjacent points in the window W are recorded as a set {P i}, N is the number of points in the set, w i Assigned to point P i The weight of P′ is the new coordinate point calculated by weighted average of all the point coordinates in the window.

[0101] For the collected image data, basic preprocessing operations such as color correction and contrast adjustment are performed to ensure the consistency of data quality, and the processed image data is divided into training sets, validation sets, and test sets according to the established ratio. The division of the data set follows the principles of randomization and stratified sampling to ensure that the samples in each set can represent the overall data distribution and avoid bias.

[0102] The detailed 3D model of the parking area is constructed by using the NeRF 3D reconstruction method based on 3D point cloud. The specific implementation of the NeRF 3D reconstruction method based on 3D point cloud is further described below.

[0103] The NeRF 3D reconstruction method based on 3D point cloud includes the following steps:

[0104] S103: The 3D point cloud data preprocessed in step 2.2 and the training set images are input into the improved NeRF model for training to generate a detailed 3D model of the parking area.

[0105] During training, the volume density at the sampling point is obtained by the following steps:

[0106] Step L1: Using the processed point cloud data obtained from the LiDAR sensor, map each point cloud point to the nearest voxel grid v in the voxel grid unit at the first resolution. The voxel grid unit represents the three-dimensional scene as a grid, and the grid vertices are evenly distributed along three mutually orthogonal axes. For these mapped voxel grids v, set an initial density value σ0 =0, whenever a point cloud point is mapped to a voxel grid, the initial density value of the voxel grid v is increased by a fixed increment to reflect the distribution of the point cloud data in the scene. Based on the size of the calculated initial density gradient of the voxel grid, the distribution of the sampling points on the ray r is dynamically adjusted. Specifically, the setting of the sampling points is divided into two stages. In the first stage, the first sampling points are evenly spaced on the ray r. In the second stage, the ray r is divided into a plurality of unit segments according to the length, and the number of point cloud points mapped in the voxel grid near the unit segment is counted. In the unit segment near the voxel grid with a higher point cloud point density, the interval of the second sampling points is reduced to capture more geometric details; and in the unit segment near the voxel grid with a lower point cloud point density, the interval of the second sampling points is increased to avoid oversampling in open areas.

[0107] σ 0 (v) = count(P pc ∩v);

[0108] where σ 0 (v) is the initial density value at voxel grid v, R pc is the point cloud set preprocessed in S102, and count() is a counting function used to calculate the number of point cloud points mapped to the voxel grid v.

[0109] Step L2. Set the above initial density value σ 0 (v) is used as input and a volume density encoder based on hash grid coding is used to divide the encoded volume density feature vector into eight equal parts and distribute it to each vertex of the voxel grid v. The volume density feature vector components of up to eight voxel grids v around each vertex are merged at this common vertex and stored in this common vertex. In this way, the eight vertices of each voxel grid v in the grid of the three-dimensional scene respectively store the sum of the volume density feature vector components of the main voxel grid v and the surrounding adjacent voxel grids.

[0110]

[0111] In the above formula, is the sum of the volume density eigenvector components of each vertex of the voxel grid, σ 0 (v) is the voxel density value of voxel grid cell v. It is a multi-layer perceptron that uses hash grid encoding volume density features.

[0112] Step L3. The voxel grid where the sampling point is located is calculated based on the coordinates of the sampling point. Based on the sum of the volume density feature vector components stored at the vertices of the voxel grid obtained in step L2, the volume density σ(x) of each sampling point is calculated using trilinear interpolation:

[0113] index = h(x);

[0114]

[0115] In the above formula, h is a hash function, index is the index of the voxel grid where the sampling point is located, σ(x) represents the volume density of the sampling point, which is calculated based on the trilinear interpolation method, and x = (x, y, z) represents the position coordinates of a given sampling point. Since the spatial size of the scene is a known quantity, the viewing direction of the camera is known, the length of the ray r extending from the camera is a known quantity, and the position coordinates of the sampling point on the ray r are also known. The sum of the volume density feature vector components of the voxel grid around each vertex of the voxel grid determined by the voxel grid index index where the sampling point is located. The volume density σ(x) of the sampling point can be calculated as the weighted mean of the sum of the volume density feature vector components of the eight surrounding vertices of the voxel grid, for example, weighted by the proximity of the eight surrounding vertices of the voxel grid to the sampling point.

[0116] During training, the color feature vector at the sampling point is obtained by the following steps:

[0117] Step T1: construct a three-dimensional voxel grid unit of a second resolution, the grid covers the entire scene, the second resolution is a resolution selected from a plurality of resolutions including the first resolution, and L different voxel grids of the second resolution are defined. Each voxel grid G l Capture color embedding features c at different scales for a specific resolution level l f .

[0118] Point Cloud Collection pv Mapped to the voxel grid unit at the second resolution, and the mean point coordinates of the point cloud cluster in the voxel grid unit are used as the input of the color embedding feature encoder at different second resolutions l. The color embedding feature encoder encodes the mean point coordinates of the point cloud cluster mapped to the voxel grid v based on the encoding method of the hash grid, and divides the encoded color embedding feature vector into eight equal parts and distributes it to each vertex of the voxel grid. The color embedding feature vector components of up to eight voxel grids around each vertex are merged at this common vertex and displayed and stored in this common vertex.

[0119] Step T2: Based on the coordinates of the sampling point, the voxel grid where the sampling point is located at resolution l is calculated through the hash function to obtain its index index l ;

[0120] Step T3: Find the sum of the color embedding feature vector components at each vertex of the voxel grid in the hash table. Since the sampling point is located inside the voxel grid, the color feature vector f of the sampling point x can be obtained by trilinear interpolation. l (x);

[0121] Step T4: Obtain the color embedding feature c of the sampling point by concatenating the color feature vectors of all second resolution levels f :

[0122] Steps T1 to T4 sample the following functions for execution:

[0123]

[0124]

[0125] index l =h(x / s l );

[0126]

[0127]

[0128] in, is the color embedding feature vector component of each vertex of the voxel grid, P pc is the processed point cloud set, v is the voxel grid unit, is a multi-layer perceptron that uses hash grid coding to embed color features, index l is the sampling point index, h is the hash function, s l is the grid size scaling factor at resolution l, f l (x) is the color feature vector of sampling point x at resolution l, The index of the voxel grid where the sampling point is located l Determine the sum of the color embedding feature vector components of the voxel grid around each vertex of the voxel grid. Concat is a connection operation that combines the color embedding feature vectors from voxel grids of different resolutions to obtain the color embedding feature c of the sampling point. f .

[0129] The color embedding feature c based on the sample points obtained during training f , two independent multi-layer perceptrons (MLPs) are used to obtain the viewpoint-dependent color c vd and the viewpoint-independent color c vi , merge c vd and c vi Get the final color value c of the given sampling point.

[0130] In order to improve the model rendering efficiency and the synthesis quality of new views, the color of a given sampling point position is divided into viewpoint-dependent colors c by color decomposition. vd and the viewpoint-independent color c vi , and the final color c is composed of these two parts:

[0131] c=M(c f ,d);

[0132] c=c vd +c vi ;

[0133] c vd =M vd (c f ,d);

[0134] c vi =M vi (c f );

[0135] In the above formula, c is the final color value of the given sampling point position, d = (θ, φ) is the camera viewing direction, θ is the horizontal direction, φ is the vertical direction, and c vd is the viewpoint-dependent color, c vi is the viewpoint-independent color, M is a multi-layer perceptron (MLP) including M vd 、M vi , by vd Input the given camera viewing direction and sampling point color embedding feature c f Get the viewpoint dependent color c vd , by vi The color embedding feature c of the input sampling point f Get the viewpoint-independent color c vi , according to c vd and c vi Implicitly modeled scene colors.

[0136] During training, the volume density and final color value of each sampling point are used for volume rendering:

[0137]

[0138]

[0139] δ i =(x i+1 -x i );

[0140] In the above formula, represents the pixel color rendered along ray r, N is the number of sampling points on the ray r, T irepresents the transmittance of the i-th sampling point, σ i represents the volume density of the i-th sampling point, δ i represents the distance between adjacent sampling points, c i Represents the color value of the i-th sampling point, and n represents the sampling point before the i-th sampling point.

[0141] During training, the color of the sampling point is embedded into the feature c f The overall loss function is determined by taking the camera pose as an additional supervisory signal during NeRF training, and the NeRF network is trained based on the overall loss function to obtain the NeRF model.

[0142] L=L p +λL r ;

[0143]

[0144]

[0145]

[0146]

[0147] In the above formula, L represents the total loss function, L p Represents the weighted luminosity loss, L r represents the regularization term, λ represents the hyperparameter, Q is the number of ray samples, r j represents the jth ray, represents the dynamic weight based on the color difference of the sampling point on the jth ray, τ j is the weight coefficient, c vd,k represents the view-dependent color at the kth sampling point, H is the number of all sampling points on all rays; the weighted photometric loss is used to minimize the predicted color along ray r j The rendered pixel color The real color C(r j ), and the dynamic weight Through the clamp() function, the weight coefficient τ j Restricted to a specific range (usually between [1, 10]) to amplify the loss of sampling points with large color differences, so that the model pays more attention to these difficult-to-render samples during training; L r The regularization term encourages the model to learn the view-dependent color c by minimizing the sum of the absolute values ​​of the view-dependent colors. vd The changes are as smooth as possible.

[0148] The scalar volume density σ at the N sampling points on the ray r and the color c emitted toward the viewing direction at each sampling point are obtained above, and the output color and density are projected onto the image in the direction of the ray viewing angle, thereby realizing view synthesis.

[0149] During the training of the NeRF network, the parameters of the volume density encoder and the color embedding feature encoder in steps L1 to L3 and steps T1 to T4 are optimized by minimizing the overall loss function.

[0150] Calculate L p Weighted photometric loss and L r The regularization term is about the volume density encoder parameters θ and the color embedding feature encoder parameters The loss gradient of the volume density encoder parameter θ is obtained and color embedding feature encoder parameters The total loss gradient The RAdam optimization algorithm is used to update the volume density encoder parameters θ and the color embedding feature encoder parameters Then the volume density and color embedding features are dynamically optimized. The RAdam optimization algorithm can be found at https: / / blog.csdn.net / zhaoyin654 / article / details / 100057752.

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167] In the above formula, m t,θ 、v t,θ are the first-order moment estimate and second-order moment estimate of the volume density encoder parameter θ at time step t, β 1,θ , β 2,θ are the decay rates of the corresponding first-order moment and second-order moment, is the total loss gradient of the volume density encoder parameter θ, are the first-order moment estimate and second-order moment estimate after correction of the volume density encoder parameter θ, α t,θ is the learning rate of the volume density encoder parameter θ, α θ is the initial learning rate of the volume density encoder parameter θ, ∈ is a very small positive number, ρ ∞,θ For the rectifier r t,θ The judgment threshold of are the volume density encoder parameters at time step t First-order moment estimation and second-order moment estimation, are the decay rates of the corresponding first-order moment and second-order moment, Color embedding feature encoder parameters The total loss gradient, They are the volume density encoder parameters The corrected first-order moment estimate and second-order moment estimate, is the volume density encoder parameter The learning rate, is the volume density encoder parameter The initial learning rate, For rectifier The judgment threshold.

[0168] S104: The three-dimensional reconstructed model obtained by training in S103 is sent to the vehicle terminal by wireless transmission. When the vehicle arrives at the designated parking area, the automatic parking path planning is implemented based on the existing automatic parking algorithm under the finely reconstructed three-dimensional model.

[0169] The method of initializing volume density with Lidar point cloud data introduced in step S103, the method of storing color features by multi-resolution hash voxel grid encoding, and the method of separating volume density and color information for prediction all speed up the model training speed; the color is decomposed into perspective-dependent highlights and reflections related to perspective, and perspective-independent diffuse reflections that are independent of perspective. The color decomposition method is closer to physical phenomena and can more realistically simulate the lighting and reflection characteristics of the real world.

[0170] Compared with traditional point cloud reconstruction, the 3D reconstruction based on NeRF has specific advantages in the following aspects:

[0171] 1. Continuous volume representation: NeRF provides continuous volume density and color information, overcoming the discontinuity of point cloud data in sparse areas.

[0172] 2. Computational efficiency: NeRF’s volume rendering technology is computationally more efficient, especially for large-scale scenes.

[0173] 3. Generalization ability: NeRF’s implicit field representation enables it to generalize to unseen viewpoints and new scenes.

[0174] 4. Lighting simulation: NeRF can simulate complex lighting effects, while traditional point cloud reconstruction usually does not consider lighting information.

[0175] 5. Multi-sensor fusion: NeRF can more naturally fuse data from multiple sensors such as lidar and cameras.

[0176] 6. Real-time rendering: NeRF supports real-time rendering, which is suitable for instant decision-making in autonomous driving systems.

[0177] 7. Reduce data dependence: NeRF learns implicit representations of scenes, reducing dependence on large amounts of labeled data.

[0178] Technical effect:

[0179] 1. Accurate obstacle identification: Using a detailed 3D model, the path planning system can more accurately identify and locate obstacles, including pedestrians, vehicles, and static objects.

[0180] 2. Improve decision transparency: Detailed three-dimensional scene representation makes the decision process of path planning clearer and easier to analyze and understand.

[0181] 3. Reduce dependence on high-precision maps: Autonomous driving systems can reduce dependence on pre-made high-precision maps and improve the system's adaptability to new environments.

[0182] Technical issues solved:

[0183] 1. Continuity and completeness of 3D models: NeRF provides a continuous volume representation that can generate a complete scene representation even in areas where point cloud model data is sparse or missing.

[0184] 2. View-dependency: NeRF is able to simulate view-dependent illumination and occlusion effects, while point cloud models usually do not contain this view-specific information.

[0185] 3. Lighting and Shadows: NeRF can render realistic lighting effects, including shadows and reflections, while point cloud models usually cannot directly represent these lighting characteristics.

[0186] 4. Capturing complex details: NeRF is able to capture subtle surface details and texture information, while point cloud models may lose these details due to resolution limitations.

[0187] The 3D model rendered by NeRF is realistic enough, and the present invention can significantly improve the speed and quality of 3D reconstruction of parking lot scenes through the improved NeRF model, which contributes to fast and accurate vehicle automatic parking control.

[0188] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in the field.

Claims

1. A NeRF 3D reconstruction method based on 3D point cloud, characterized in that: The following steps are involved: Modeling the scene using camera data of the scene from a camera, point cloud data of the scene from a lidar, a grid space of the scene, and sampling points along rays extending from the camera to jointly train a volume density multilayer perceptron and a color multilayer perceptron to obtain pixel colors rendered by all rays; During the joint training, the grid space is segmented at a first resolution and consists of a voxel grid of the first resolution, or is segmented at a second resolution and consists of a voxel grid of the second resolution, wherein the second resolution is a resolution selected from a plurality of resolutions including the first resolution; The volume density multilayer perceptron receives the point cloud number of the nearest voxel grid in each voxel grid unit of the first resolution mapped to the point cloud data as input, and generates the volume density at the sampling point; The color multi-layer perceptron receives the mean point coordinates of the point cloud clusters mapped to each voxel grid unit of the second resolution and the index of the sampling point in the grid space composed of the voxel grid of the second resolution as input, generates a color feature vector at the sampling point in the voxel grid of the second resolution, and the color multi-layer perceptron also splices the color feature vectors of all the second resolution levels to obtain the color embedding feature of the sampling point; generating a final color of the sampling point using the color embedding features of the sampling point and the camera data during the joint training; During the joint training, the final color of the sampling point, the volume density at the sampling point, and the distance between adjacent sampling points on the ray are also used to obtain the pixel color rendered by the ray; Generating the volume density at the sampling point comprises the following steps: Step L1. Using the point cloud data obtained from the LiDAR sensor after performing noise filtering, each point cloud point is mapped to the nearest voxel grid v in the voxel grid unit at the first resolution: σ0(v)=count(P pc ∩v); Where σ0(v) is the initial density value at the voxel grid v, P pc is the point cloud set after noise filtering, count() is a counting function used to calculate the number of point cloud points mapped to the voxel grid v; Step L2. Taking the above initial density value σ0(v) as input and using a volume density encoder based on hash grid coding, the volume density feature vector obtained after encoding is evenly divided into eight parts and distributed to each vertex of the voxel grid v as a volume density feature vector component, and the eight vertices of each voxel grid v respectively store the sum of the volume density feature vector components of the main voxel grid v and the surrounding adjacent voxel grids; in, is the volume density feature vector set component of each vertex of the voxel grid v, σ0(v) is the voxel density value of the voxel grid v, It is a multi-layer perceptron that uses hash grid encoding volume density features; Step L3. The voxel grid where the sampling point is located is calculated based on the coordinates of the sampling point. Based on the sum of the volume density feature vector components stored at the vertices of the voxel grid obtained in step L2, the volume density σ(x) of each sampling point is calculated using trilinear interpolation: index = h(x); In the above formula, h is the hash function, index is the index of the voxel grid unit where the sampling point is located, σ(x) represents the volume density of the sampling point, and x = (x, y, z) represents the position coordinates of a given sampling point. The sum of the volume density feature vector components of the voxel grid around each vertex of the voxel grid determined by the voxel grid unit index index where the sampling point is located; The color multilayer perceptron receives the mean point coordinates of the point cloud clusters mapped to each voxel grid unit of the second resolution and the index of the sampling point in the grid space composed of the voxel grid units of the second resolution as input, and generates the color feature vector at the sampling point in the voxel grid unit of the second resolution, including: Step T1. Define L different voxel grids of the second resolution Point Cloud Collection pc Mapping to the voxel grid unit at the second resolution, and taking the mean point coordinates of the point cloud cluster in the voxel grid unit as the input of the color embedding feature encoder at different second resolutions l, the color embedding feature encoder encodes the mean point coordinates of the point cloud cluster mapped to the voxel grid v based on the encoding method of the hash grid, and divides the color embedding feature vector obtained after encoding into eight parts and distributes them to each vertex of the voxel grid as the color embedding feature vector component, and the eight vertices of each voxel grid v respectively store the sum of the color embedding feature vector components of the main voxel grid v and the surrounding adjacent voxel grids; in, is the mean point coordinate of the point cloud cluster in the voxel grid, average() is the function for solving the mean point coordinate of the point cloud cluster, is the color embedding feature vector component of each vertex of the voxel grid v, Multilayer Perceptron for color embedding features using hash grid encoding; Step T2. Calculate the voxel grid where the sampling point is located at the second resolution l through a hash function based on the sampling point coordinates, and obtain its index index l , index l =h(x / s l ); index l is the sampling point index, h is the hash function, s l is the grid size scaling factor at the second resolution l; Step T3. Find the sum of the color embedding feature vector components at each vertex of the voxel grid in the hash table, and use trilinear interpolation to obtain the color feature vector f of the sampling point x at resolution l l (x); The index of the voxel grid where the sampling point is located l Determine the sum of color embedding feature vector components of the voxel grid around each vertex of the voxel grid; Step T4. Concatenating all color feature vectors of the second resolution level to obtain the color embedding features of the sampling points includes: concat is a concatenation operation that merges the color embedding feature vectors from different second-resolution voxel grids to obtain the color embedding feature c f .

2. The method according to claim 1 further comprises, after the joint training, a component actuated based on the modeled scenario.

3. The method according to claim 1, characterized in that The sampling points include first sampling points and second sampling points, the first sampling points are evenly spaced along the ray, the linear density of the second sampling points in each unit segment of the ray along the ray is positively correlated to the number of point clouds mapped from the point cloud data to each voxel grid of the first resolution close to each unit segment of the ray, the number of voxel grid cells of the first resolution passed through each unit segment of the ray is equal, and the second sampling points in each unit segment of the ray are evenly spaced along the unit segment.

4. The method according to claim 1, characterized in that: It also includes generating the volume density at the sampling point by interpolating the encoded values ​​at the nearby grid points in the grid space composed of the voxel grids of the first resolution, where the encoded values ​​are the sum of 1 / 8 of the number of point clouds in the voxel grid units of the first resolution mapped to the nearby grid points in the grid space composed of the voxel grid units of the first resolution.

5. The method according to claim 1, characterized in that It also includes generating a color feature vector at the sampling point by interpolating the encoded values ​​at nearby grid points in the grid space composed of voxel grids of the second resolution, wherein the encoded values ​​are the sum of 1 / 8 of the mean point coordinates of the point cloud cluster in the voxel grid units of the second resolution mapped to the nearby grid points in the grid space composed of voxel grid units of the second resolution.

6. The method according to claim 1 further comprises updating the volume density multilayer perceptron and the color multilayer perceptron based on a total loss function.

7. The method according to claim 1, characterized in that The camera data from the camera includes the image of the scene and the camera pose information: the camera pose information includes the position of the camera and the viewing direction of the camera: (x c ,y c ,z c ,d), where d = (θ, φ) is the camera’s viewing angle, θ is the horizontal direction, φ is the vertical direction, (x c ,y c ,z c ) is the position of the camera; Generating a final color of the sampling point using the color embedding feature of the sampling point and the camera data during the joint training includes: Based on the obtained color embedding feature c of the sampling point f , two independent multi-layer perceptrons are used to obtain the viewpoint-dependent color c vd and the viewpoint-independent color c vi , merge c vd and c vi Get the final color value c of a given sampling point: c=M(c f ,d); c=c vd +C vi ; c vd =M vd (c f ,d); C vi =M vi (c f ); Among them, c represents the final color value of a given sampling point, M is a multi-layer perceptron including M vd 、M vi ; The NeRF model randomly selects a subset of pixels in the training image at each iteration of training, generates a 3D ray r for each pixel, and calculates the volume density σ at the position of the i-th sampling point for the N sampling points on the ray r. i , the final color value c i .

8. The method according to claim 1, characterized in that Also using the final color of the sampling point, the volume density at the sampling point, and the distance between adjacent sampling points on the ray during the joint training to obtain the pixel color rendered by the ray includes: The pixel color for ray r is calculated using the following function: δ i =(x i+1 -x i ); in, represents the pixel color rendered along ray r, N is the number of sampling points on the ray, T i represents the transmittance of the i-th sampling point, σ i represents the volume density of the ith sampling point, δ represents the distance between the ith sampling point and the adjacent sampling points, c i Represents the final color of the i-th sampling point, and n represents the sampling point before the i-th sampling point.

9. The method according to claim 6, characterized in that The overall loss function during NeRF network training is: L=L p +λL r ; Among them, L represents the total loss function, L p represents the weighted luminosity loss, L r represents the regularization term, λ represents the hyperparameter, Q is the number of ray samples, r j represents the jth ray, represents the dynamic weight based on the color difference of the sampling point on the jth ray, τ j is the weight coefficient, C vd,k represents the view-dependent color at the kth sampling point, H is the number of all sampling points on all rays; the weighted photometric loss is used to minimize the predicted color along ray r j The rendered pixel color The real color C(r j ), and the dynamic weight Through the clamp() function, the weight coefficient τ j Restricted to [1,10] to amplify the loss of sampling points with large color differences; L r The regularization term is used to minimize the sum of the absolute values ​​of the view-dependent colors.

10. The method according to claim 6, characterized in that Updating the volume density multilayer perceptron and the color multilayer perceptron based on the total loss function includes: Calculate L p Weighted photometric loss and L r The regularization term is about the volume density encoder parameters θ and the color embedding feature encoder parameters The loss gradient of the volume density encoder parameter θ is obtained and color embedding feature encoder parameters The total loss gradient The RAdam optimization algorithm is used to update the volume density encoder parameters θ and the color embedding feature encoder parameters Then the volume density and color embedding features at the sampling points are dynamically optimized.

11. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method according to one of claims 1 to 10.

Images