Neural radiation field-based three-dimensional reconstruction method and device, and storage medium

Abstract

The invention provides a three-dimensional reconstruction method and device based on a neural radiation field and a storage medium, and the method comprises the steps: obtaining a plurality of pieces of sample data, the sample data comprising multi-view image data; inputting the plurality of sample data into a preset nerve radiation field for training to obtain a trained initial nerve radiation field; extracting a three-dimensional grid from a three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on the plurality of surface point coordinates in the three-dimensional grid; projecting the plurality of target light rays to the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the plurality of projected image data; re-training the initial neural radiation field based on the multiple pieces of sample data and the pseudo labels of the multiple pieces of target light to obtain a trained target neural radiation field; and performing three-dimensional reconstruction on the to-be-processed image data by using the target neural radiation field. The invention aims to improve the three-dimensional reconstruction precision of multi-view image data.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional data processing, and in particular to a three-dimensional reconstruction method, device and storage medium based on neural radiation field. Background Art

[0002] With the rapid development of autonomous driving technology, achieving high-precision 3D reconstruction of the driving environment is crucial for ensuring safe driving and effective navigation. Neural Radiance Fields (NeRF) technology provides a new approach for creating detailed models of complex environments. NeRF utilizes neural networks to model objects, enabling high-quality 3D reconstruction. Specifically, objects can be treated as radiation fields, and 3D reconstruction is achieved by modeling these radiation fields.

[0003] However, in actual autonomous driving scenarios, data collection is typically performed along a predetermined path, resulting in a very limited viewpoint and a path-distributed distribution. Therefore, neural radiance field technology faces challenges such as sparse viewpoints and data distribution along a predetermined path. This linear, path-constrained data collection method results in a severe lack of scene information, leading to insufficient 3D reconstruction accuracy. Summary of the Invention

[0004] The main purpose of this application is to provide a three-dimensional reconstruction method, device and storage medium based on neural radiation field, aiming to solve the problem of insufficient three-dimensional reconstruction accuracy of current technology.

[0005] In a first aspect, the present application provides a three-dimensional reconstruction method based on neural radiation fields, comprising:

[0006] Acquire a plurality of sample data, wherein the sample data includes image data from multiple perspectives;

[0007] Inputting the plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field;

[0008] extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on a plurality of surface point coordinates in the three-dimensional grid;

[0009] Projecting the plurality of target light rays onto the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data;

[0010] retraining the initial neural radiation field based on the plurality of sample data and the pseudo labels of the plurality of target light rays to obtain a trained target neural radiation field;

[0011] The target nerve radiation field is used to perform three-dimensional reconstruction on the image data to be processed.

[0012] In a second aspect, the present application also provides a neural radiation field model training method, comprising:

[0013] Acquire a plurality of sample data, wherein the sample data includes image data from multiple perspectives;

[0014] Inputting the plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field;

[0015] extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on a plurality of surface point coordinates in the three-dimensional grid;

[0016] Projecting the plurality of target light rays onto the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data;

[0017] Based on the plurality of sample data and the pseudo labels of the plurality of target light rays, the initial neural radiation field is retrained to obtain a trained target neural radiation field.

[0018] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the three-dimensional reconstruction method or model training method as described in the embodiments of the present application.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the three-dimensional reconstruction method or model training method as described in the embodiments of the present application is implemented.

[0020] The present application provides a three-dimensional reconstruction method, device and storage medium based on neural radiation field. The present application trains the initial neural radiation field through multi-perspective image data, extracts a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and then generates multiple target rays based on the surface point coordinates of the three-dimensional grid. Afterwards, the multiple target rays are projected onto multiple image data, and pseudo-labels of the multiple target rays are obtained based on the projected multiple image data. Finally, based on multiple sample data and the pseudo-labels of the multiple target rays, the initial neural radiation field is retrained to obtain a trained target neural radiation field, so that the target neural radiation field can be used to perform three-dimensional reconstruction on the image data to be processed. In this way, the pseudo-labels of multiple target rays can be effectively utilized to achieve high-quality view synthesis and surface reconstruction, with finer surface details, which can significantly improve the three-dimensional reconstruction accuracy of multi-perspective image data and effectively solve the problem of insufficient reconstruction accuracy caused by sparse views and path constraint data acquisition in autonomous driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A schematic diagram of a three-dimensional reconstruction method according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a scenario for implementing the 3D reconstruction method provided in this embodiment;

[0024] Figure 3 A schematic diagram of the steps of a model training method provided in an embodiment of the present application;

[0025] Figure 4 A schematic block diagram of a three-dimensional reconstruction system provided in an embodiment of the present application;

[0026] Figure 5 A schematic block diagram of another three-dimensional reconstruction system provided in an embodiment of the present application;

[0027] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present application.

[0028] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0031] With the rapid development of autonomous driving technology, achieving high-precision 3D reconstruction of the driving environment is crucial for ensuring safe driving and effective navigation. Neural radiance field technology offers a new approach for creating detailed models of complex environments. However, in real-world autonomous driving scenarios, it faces challenges such as sparse viewports and data distribution along predetermined paths. This results in suboptimal reconstruction, particularly in areas with rich detail or complex geometry.

[0032] For example, autonomous driving data collection typically occurs along predefined routes, resulting in a very limited viewpoint distribution and a path-based distribution. Unlike traditional multi-view acquisition methods around objects, this linear, path-constrained data collection approach severely lacks scene information, leading to reduced reconstruction quality. This manifests itself in numerous floating-point artifacts, inaccurate geometry, and a loss of surface detail.

[0033] Based on this, the embodiment of the present application provides a 3D reconstruction method, device and storage medium based on neural radiation field. The view enhancement method guided by geometric information significantly improves the 3D reconstruction accuracy of autonomous driving scenes and solves the problem of insufficient reconstruction accuracy of traditional neural radiation field under sparse viewing angle and path constraint acquisition conditions. It is particularly suitable for sparse viewing angle in autonomous driving scenes. Figure 3 Dimensional reconstruction can provide technical support for improving the simulation test of autonomous driving systems.

[0034] Among them, the three-dimensional reconstruction method can be applied to computer equipment, and the computer equipment can include terminal equipment or servers. The terminal equipment can be electronic devices such as mobile phones, tablet computers, laptops, desktop computers, personal digital assistants and wearable devices; the server can be a single server or a server cluster composed of multiple servers.

[0035] The embodiments of this application are applicable to the following industries: environmental perception in autonomous driving, high-precision map construction and navigation systems, urban planning, and digital twins. Specifically, they can be used for digital reconstruction of urban road environments, virtual reality (VR) and augmented reality (AR), and the creation of immersive driving scenario simulations.

[0036] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0037] Please refer to Figure 1 , Figure 1 A schematic flow chart of the steps of a three-dimensional reconstruction method provided in an embodiment of the present application.

[0038] like Figure 1 As shown, the three-dimensional reconstruction method includes steps S101 to S106.

[0039] Step S101: Acquire a plurality of sample data, where the sample data includes image data from multiple perspectives.

[0040] The multiple sample data can be used as training sample data for the neural radiation field for iterative training of the neural radiation field. The multiple sample data can be collected in an autonomous driving scenario. For example, the multi-view image data can be collected by a multi-view camera array set up on the autonomous driving platform, thereby providing scene image data from different perspectives as the basic input source for 3D reconstruction.

[0041] It should be noted that sample data can include multi-view image data and other synchronized data, such as the autonomous driving platform's pose data and depth information. The autonomous driving platform can be an autonomous vehicle, but currently, it can also be a platform such as a drone or prototype vehicle used for simulation testing. The pose data can include the autonomous driving platform's position and posture data, and the depth information can be direct depth information of the scene collected by a depth sensor.

[0042] In one embodiment, obtaining a plurality of sample data includes: obtaining multi-perspective image data collected by an autonomous driving platform; performing distortion correction and color balance on the multi-perspective image data to obtain multi-perspective sample image data; and removing abnormal image data from the multi-perspective sample image data to obtain sample data.

[0043] It should be noted that multi-view image data is calibrated by performing distortion correction and color balancing on the multi-view image data. Multi-view sample image data is cleaned by removing abnormal image data from the multi-view sample image data. Abnormal image data refers to sample image data with anomalies, which can be identified through methods such as deep learning. Performing distortion correction and color balancing before filtering can improve the quality of the sample data, thereby improving the accuracy of the neural radiation field model.

[0044] The sample data also includes the pose data and depth information of the autonomous driving platform; these pose data, depth information, and the multi-view image data in the sample data are acquired in the same time series. It should be noted that the pose data, depth information, and multi-view image data are acquired in the same time series to ensure temporal consistency of all data, which together constitute the basic input for 3D reconstruction.

[0045] Step S102: input a plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field.

[0046] Multiple sample data are input into a preset neural radiation field for iterative training until the neural radiation field converges, thereby obtaining a converged neural radiation field as the trained initial neural radiation field. The convergence condition of the neural radiation field can be set according to actual conditions, for example, based on model parameters, loss function value, number of training times, training time, and other parameters.

[0047] In one embodiment, the preset neural radiation field can adopt the NeuS architecture, which includes two continuous functions: a signed distance function f□(x)→s and a view-dependent color function f□(x,d)→c. The training process minimizes the mean square error loss through the volume rendering method:

[0048]

[0049] Where: L MSE is the mean square error loss function, r is the camera light, is the predicted light color, and C(r) is the true color of the light (ground truth value). The color of the volume rendering is calculated as:

[0050]

[0051] Where N is the number of points sampled along the ray; Ti is the cumulative transmittance, which represents the probability that the ray does not hit any particle from t1 to ti; α i is the discrete opacity value, which is a function of the probability density function of the signed distance function f1; C i is the color at sampling point i. The cumulative transmittance is calculated as:

[0052]

[0053] Step S103: extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating multiple target rays based on the coordinates of multiple surface points in the three-dimensional grid.

[0054] After obtaining the initial neural radiation field, it can be used to generate a 3D radiation field, thereby extracting a 3D mesh of the 3D radiation field. The 3D mesh can be composed of multiple surface point coordinates. For some surface point coordinates, multiple target rays can be generated.

[0055] For example, by randomly selecting a preset number of surface point coordinates and using each selected surface point coordinate as a starting point, one or more rays in different directions can be generated as target rays, thereby obtaining multiple target rays. Generating multiple target rays based on multiple surface point coordinates in a 3D mesh converts the implicit neural representation into an explicit geometric representation, preparing for subsequent ray enhancement.

[0056] In one embodiment, a 3D mesh is extracted from the 3D radiance field generated by the trained initial neural radiance field using a Marching Cubes algorithm. The algorithm identifies isosurfaces in the density field and generates a 3D mesh representation of the scene as a geometric proxy volume.

[0057] In one embodiment, the surface point coordinates include vertex coordinates, and generating multiple target rays based on the multiple surface point coordinates in the three-dimensional grid includes: obtaining multiple vertex coordinates in the three-dimensional grid; and generating multiple target rays in random directions corresponding to each vertex coordinate with each vertex coordinate as a starting point.

[0058] It should be noted that a 3D mesh can include vertex coordinates for multiple vertices. Based on each vertex coordinate, multiple target rays with random directions can be generated to prepare for subsequent light enhancement. For example, for each vertex coordinate, multiple rays with random directions are emitted from that point, thereby generating multiple target rays with random directions corresponding to each vertex coordinate.

[0059] In one embodiment, after generating multiple target rays based on multiple surface point coordinates in a three-dimensional grid, the method further includes: performing a visibility check on each target ray to determine whether each target ray is blocked by an object; and removing the target rays blocked by the object from the multiple target rays.

[0060] It should be noted that the ray marching algorithm can be used to check the visibility of each target ray to determine whether the target ray is blocked by other objects in the scene. If the ray is invisible (blocked by the scene), it is eliminated to prevent the subsequent model retraining from being ineffective and affecting the model accuracy.

[0061] Understandably, in autonomous driving scenarios, due to the complex occlusion relationships between objects, existing methods struggle to accurately infer the geometry and appearance of occluded areas, resulting in "ghost" artifacts or geometric discontinuities in the reconstruction results. Therefore, the embodiments of the present application utilize visibility checks to remove target rays blocked by objects from multiple target rays, effectively improving the quality of training samples for subsequent model retraining, thereby enhancing the effectiveness of model retraining.

[0062] In one embodiment, the angle between the target light and the surface normal is less than a predetermined angle, wherein the predetermined angle includes π / 2. It should be noted that ensuring that the visible viewing angle between the target light and the surface normal is less than π / 2 ensures that the target light is directed toward the outside of the scene, thereby enhancing the viewing angle range and ensuring a correct viewing angle.

[0063] That is to say, during the projection process of the target light, a constraint mechanism can be included to ensure that the angle between the target light and the surface normal is less than π / 2, so as to ensure that the light is directed outside the scene.

[0064] In one embodiment, a three-dimensional mesh is extracted from a three-dimensional radiation field generated by an initial neural radiation field; multiple vertex coordinates in the three-dimensional mesh are obtained; with each vertex coordinate as a starting point, multiple target rays of random directions corresponding to each vertex coordinate are generated; a visibility check is performed on each target ray to determine whether each target ray is blocked by an object; and target rays blocked by the object are eliminated from the multiple target rays.

[0065] It should be noted that existing methods typically learn scene representations directly from images, lacking a clear understanding and utilization of the scene's geometric structure, making it difficult to perform reasonable reconstruction and extrapolation when information is insufficient. Therefore, there is a lack of geometric prior guidance. However, the embodiments of the present application, by extracting the 3D mesh of the geometric proxy, performing target raycasting and visibility checks, can increase the clear understanding and utilization of the geometric structure, thereby achieving high-quality view synthesis and effectively improving the accuracy of 3D reconstruction.

[0066] Step S104 : Projecting the plurality of target light rays onto a plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data.

[0067] The pseudo-labels of target rays can include RGB information from the image data, specifically as a set of vector points. These pseudo-labels can be used for light enhancement, thereby refining surface details and enabling high-quality view synthesis, effectively improving 3D reconstruction accuracy.

[0068] In one embodiment, projecting multiple target rays onto multiple image data can be expressed as:

[0069]

[0070] Where Sv is the set of all rays at surface point v; (c j , d j ) is the pair of light color and corresponding light direction; c v，j is the color of the surface point v when viewed from direction dj; Norm(o j -v) is the normalized direction vector from the camera center oj to the surface point v; N vis is the number of visible or target rays for the surface point v. Each target ray can be regarded as a visible angle, so the number of target rays can also be called the number of visible angles.

[0071] In one embodiment, pseudo labels of multiple target light rays are obtained based on multiple image data after projection, including: when the number of multiple target light rays is less than a preset number threshold, using linear interpolation to obtain the radiation value pseudo labels of the multiple target light rays from the multiple image data after projection; when the number of multiple target light rays is greater than or equal to the preset number threshold, using spherical harmonic functions to obtain the radiation distribution of the multiple target light rays as pseudo labels from the multiple image data after projection.

[0072] It should be noted that different methods can be used to obtain pseudo-labels for each target light, depending on the number of target rays. For example, when the number of target rays is less than a preset threshold value N, linear interpolation can be used to obtain the radiance value of each target light, and the radiance value of each target light can be used as the pseudo-label for each target light. When the number of target rays is greater than or equal to the preset threshold value N, spherical harmonics can be used to obtain the radiance distribution of each target light, and the radiance distribution of each target light can be used as the pseudo-label for each target light.

[0073] In other words, an adaptive radiance calculation strategy can be used to automatically select a method based on the number of visible viewpoints N (each target ray can be considered a visible viewpoint): spherical harmonics fitting is used when there are sufficient visible viewpoints; when there are insufficient visible viewpoints, interpolation based on geodesic distance is used. This improves the accuracy of the pseudo-labels obtained for target rays.

[0074] For example, if the visible viewing angle is less than the threshold N, the radiance value of the nearest viewing angle is selected and linear interpolation is applied:

[0075]

[0076] in, is the interpolated radiation value; c0 and c1 are the radiation values ​​of the two nearest viewing angles; w is the weight based on the geodesic distance. The weight is calculated as:

[0077]

[0078] in, is the direction vector of the new perspective, v1 and v2 are the direction vectors of the two most recent known perspectives; d(·,·) is the geodesic distance on the sphere. The geodesic distance is:

[0079]

[0080] For example, if the number of visible view angles (the number of vertex coordinates) exceeds a threshold N, the system uses spherical harmonics to fit the radiation distribution of the surface points:

[0081]

[0082] Among them, c v (d) is the radiation value of point v in direction d; l max is the maximum order of spherical harmonics; is the spherical harmonic coefficient, which is a three-dimensional vector for RGB color; is the spherical harmonic basis function; e is the spherical harmonic series (order); m: is the index of the spherical harmonic series, ranging from -l to +l. The spherical harmonic coefficients are solved by the least squares method:

[0083]

[0084] Understandably, autonomous driving scenes contain a large number of moving objects (such as pedestrians and vehicles). These dynamic objects cause the scene's lighting and geometry to change over time. The original NeRF model, designed for static scenes, cannot effectively handle this time-varying behavior, resulting in artifacts in the synthesized images and degraded reconstruction quality.

[0085] Therefore, the embodiment of the present application uses the spherical harmonic function radiation distribution fitting method to perform spherical harmonic basis function expansion on all light sets Sv of the surface point v, and uses the least squares method to solve the spherical harmonic coefficients Through the spherical harmonics c v (d) Predicting the radiation value in any direction d can accurately obtain the radiation distribution of each target light as a pseudo-label of each target light, thereby being able to handle the time-varying illumination characteristics of the scene caused by dynamic objects, thereby improving the 3D reconstruction effect.

[0086] Step S105: retrain the initial neural radiation field based on the multiple sample data and the pseudo labels of the multiple target light rays to obtain a trained target neural radiation field.

[0087] Understandably, existing NeRF methods are weak at synthesizing views outside the training data, especially in scenarios with limited data collection, such as autonomous driving. Due to the lack of an effective radiometric value prediction mechanism, synthesized results from unobserved viewpoints often exhibit distortion and blur.

[0088] Based on this, the embodiments of the present application combine multiple sample data and pseudo-labels of multiple target rays to retrain the initial neural radiation field until it converges, thereby obtaining a trained target neural radiation field, thereby improving the understanding of the target neural radiation field. The original training data is combined with the pseudo-label data generated by geometric guidance to optimize the training for the characteristics of autonomous driving scenarios.

[0089] In one embodiment, to handle the uncertainty in the projection process, the loss function of the initial neural radiance field can apply a self-supervised inverse deep smoothing loss:

[0090]

[0091] in, is the depth smoothing instant; di is the depth map; is the image Laplacian operator at pixel xi; and are the second-order partial derivatives of depth respectively; It is an image structure weighting factor that reduces the smoothness constraint at the edges.

[0092] Experimental results show that on the KITTI dataset, the PSNR of this method reaches 19.04, which is 2.11 higher than the best comparison method MipNeRF-360; on the NuScenes dataset, the PSNR of this method reaches 26.92, which is 8.88 higher than the comparison method; on the Argoverse dataset, the PSNR is 29.50 and the SSIM is 0.874, demonstrating the superior performance of this method in sparse view reconstruction of autonomous driving scenes.

[0093] In one embodiment, the training samples of the initial neural radiation field are different from those of the aforementioned training process of the preset neural radiation field. The loss function of the initial neural radiation field may be different from the loss function of the preset neural radiation field. For example, the loss function of the preset neural radiation field may use a mean square error loss function, while the loss function of the initial neural radiation field may use a self-supervised inverse deep smoothing loss function, thereby effectively optimizing the model parameters of the neural radiation field and improving model performance.

[0094] Step S106: Perform three-dimensional reconstruction on the image data to be processed using the target nerve radiation field.

[0095] Among them, the image data to be processed may include multi-perspective image data in the sample data, and may also include multi-perspective image data from other data sources. For example, the image data to be processed may be multi-perspective image data newly acquired by the autonomous driving platform. This embodiment does not make specific limitations on this.

[0096] It should be noted that using the target neural radiance field to perform 3D reconstruction on the processed image data essentially converts the optimized neural radiance field into a high-quality 3D scene representation. This process generates a detailed scene mesh model with realistic textures, which can be directly applied to scene understanding and simulation testing for autonomous driving.

[0097] For example, Figure 2 As shown, images 11, 12, and 13 are the multi-view image data to be processed. Correspondingly, images 21, 22, and 23 are the multi-view image data after 3D reconstruction. Images 21, 22, and 23 exhibit finer surface details, effectively addressing the issue of insufficient reconstruction accuracy caused by sparse views and path-constrained data acquisition in autonomous driving scenarios.

[0098] The three-dimensional reconstruction method provided in the above embodiment trains the initial neural radiation field through multi-perspective image data, extracts a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and then generates multiple target rays based on the surface point coordinates of the three-dimensional grid. Afterwards, the multiple target rays are projected onto multiple image data, and pseudo-labels of the multiple target rays are obtained based on the projected multiple image data. Finally, based on multiple sample data and the pseudo-labels of the multiple target rays, the initial neural radiation field is retrained to obtain a trained target neural radiation field, so that the target neural radiation field can be used to perform three-dimensional reconstruction on the image data to be processed. In this way, the pseudo-labels of multiple target rays can be effectively utilized to achieve high-quality view synthesis and surface reconstruction, with finer surface details, which can significantly improve the three-dimensional reconstruction accuracy of multi-perspective image data and effectively solve the problem of insufficient reconstruction accuracy caused by sparse views and path constraint data acquisition in autonomous driving scenarios.

[0099] Specifically, by training an initial neural radiance field from multi-view image data, extracting a rough geometric proxy volume, performing raycasting and visibility checks based on surface points, fitting the radiance distribution of surface points using spherical harmonics, generating geometrically consistent radiance pseudo-labels, and retraining the neural radiance field with the original and augmented data, high-quality view synthesis and surface reconstruction are achieved, significantly improving the accuracy of 3D reconstruction in autonomous driving environments. This system integrates the entire process from data acquisition to simulation testing, providing reliable technical support for environmental perception, scene understanding, and algorithm verification in autonomous driving technology.

[0100] Please refer to Figure 3 , Figure 3 A schematic flow chart of the steps of a neural radiation field model training method provided in an embodiment of the present application.

[0101] like Figure 3As shown, the model training method includes steps S201 to S205.

[0102] Step S201: Acquire a plurality of sample data, where the sample data includes multi-view image data.

[0103] Step S202: input a plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field.

[0104] Step S203: extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating multiple target rays based on the coordinates of multiple surface points in the three-dimensional grid.

[0105] Step S204 : Projecting the plurality of target light rays onto a plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data.

[0106] Step S205: retrain the initial neural radiation field based on the multiple sample data and the pseudo labels of the multiple target light rays to obtain a trained target neural radiation field.

[0107] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific process of the above-mentioned neural radiation field model training method can refer to the corresponding process in the aforementioned three-dimensional reconstruction method embodiment, and will not be repeated here.

[0108] The model training method provided in the above embodiment combines multiple sample data and pseudo-labels of multiple target rays to retrain the initial neural radiance field, enhancing the ability to synthesize views outside the coverage area and effectively optimizing the model parameters of the neural radiance field, thereby improving model performance. This helps avoid distortion and blurring in scenarios such as autonomous driving, where data collection is limited.

[0109] See also Figure 4 , Figure 4 A schematic block diagram of a three-dimensional reconstruction system provided in an embodiment of the present application.

[0110] like Figure 4 As shown, the 3D reconstruction system includes a data acquisition module 1, a perception data optimization module 2, a geometry-guided neural field processing module 3, a scene reconstruction module 4, and a simulation test module 5. The 3D reconstruction system can also be called a 3D reconstruction system based on geometric information synthesis view enhancement.

[0111] The data acquisition module 1 is used to collect scene images from different angles through multiple cameras to obtain multi-view appearance information of the environment, which constitutes the main input source for 3D reconstruction.

[0112] The multi-view camera array 1.1 is used to provide scene image data from different viewpoints as a basic input source for three-dimensional reconstruction.

[0113] The depth sensor 1.2 is used to collect direct depth information of the scene, provide geometric prior knowledge, and assist in resolving scale ambiguity when relying solely on images.

[0114] The vehicle positioning system 1.3 is used to: provide high-precision absolute position and attitude information to ensure that the collected data has accurate geographic reference and spatial consistency.

[0115] The time synchronization module 1.4 is used to ensure the time alignment of sensor data, solve the problem of asynchronous data fusion, and improve the overall consistency of the system.

[0116] The perception data optimization module 2 is used to calibrate, filter and extract features of the collected data to improve data quality and consistency, laying the foundation for subsequent reconstruction.

[0117] The image calibration unit 2.1 is used to eliminate camera distortion, correct color distortion, calibrate camera parameters, and improve image data quality.

[0118] The pose estimation unit 2.2 is used to accurately calculate the position and orientation of each camera in the global coordinate system, providing a geometric basis for multi-view fusion.

[0119] The depth map generation unit 2.3 is used to process the depth sensor data to generate a high-quality scene depth map to assist in 3D reconstruction.

[0120] The data cleaning and filtering unit 2.4 is used to identify and remove noise, outliers and redundant information in the collected data to improve data reliability.

[0121] The feature extraction unit 2.5 is used to extract key points and descriptors from the image, support scene matching and understanding, and provide semantic support for 3D reconstruction.

[0122] The timing alignment unit 2.6 is used to process data collected at different times, ensure timing consistency, and provide support for dynamic scene modeling.

[0123] The geometry-guided neural field processing module 3 is used to: use spherical harmonics and geometric proxy technology to realize surface radiation value estimation and visibility judgment, and solve the core problem of sparse view reconstruction.

[0124] The surface ray casting unit 3.1 is used to emit rays from surface points in multiple directions to create a geometric basis for multi-view observation.

[0125] The visibility check unit 3.2 is used to verify the visibility of each ray, exclude invalid rays blocked by the scene, and ensure the physical correctness of view synthesis.

[0126] The radiation pattern estimation unit 3.3 is used to reconstruct the radiation characteristic distribution of surface points based on the observation data of the existing perspective and support the appearance synthesis of the new perspective.

[0127] The spherical harmonic function fitting unit 3.4 is used to: use spherical harmonic functions to mathematically model the radiation distribution of surface points, and achieve high-precision interpolation and extrapolation of radiation values ​​in any direction.

[0128] The depth warping unit 3.5 is used to warp depth information from a known perspective to a new perspective, provide geometric constraints, and improve reconstruction consistency.

[0129] The geometry proxy generation unit 3.6 is used to: construct a rough geometric representation of the scene, assist in ray casting and occlusion judgment, and improve reconstruction efficiency.

[0130] The scene reconstruction module 4 is used to convert the neural radiation field into an explicit three-dimensional model representation, including voxel rendering, mesh extraction and texture mapping, and output high-quality three-dimensional reconstruction results.

[0131] The voxel rendering unit 4.1 is used to perform volume rendering based on the trained neural radiation field to generate high-quality new perspective images.

[0132] The mesh extraction unit 4.2 is used to extract an explicit three-dimensional mesh model from the neural radiation field to facilitate subsequent application and interaction.

[0133] The texture mapping unit 4.3 is used to generate fine textures for the extracted three-dimensional mesh to improve visual quality and realism.

[0134] See also Figure 6 , Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present application.

[0135] like Figure 6 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a storage medium and an internal memory, and the storage medium may be non-volatile or volatile.

[0136] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any three-dimensional reconstruction method.

[0137] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0138] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any three-dimensional reconstruction method.

[0139] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0140] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0141] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0142] Acquire a plurality of sample data, wherein the sample data includes image data from multiple perspectives;

[0143] Inputting the plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field;

[0144] extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on a plurality of surface point coordinates in the three-dimensional grid;

[0145] Projecting the plurality of target light rays onto the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data;

[0146] retraining the initial neural radiation field based on the plurality of sample data and the pseudo labels of the plurality of target light rays to obtain a trained target neural radiation field;

[0147] The target nerve radiation field is used to perform three-dimensional reconstruction on the image data to be processed.

[0148] In one embodiment, the surface point coordinates include vertex coordinates, and the processor, when implementing the generating of the plurality of target rays based on the plurality of surface point coordinates in the three-dimensional mesh, is configured to implement:

[0149] Obtaining coordinates of multiple vertices in the three-dimensional grid;

[0150] Taking each vertex coordinate as a starting point, a plurality of target rays in random directions corresponding to each vertex coordinate are generated.

[0151] In one embodiment, after generating a plurality of target rays based on the coordinates of a plurality of surface points in the three-dimensional grid, the processor is further configured to:

[0152] Performing a visibility check on each of the target rays to determine whether each of the target rays is blocked by an object;

[0153] Target rays blocked by the object are removed from the multiple target rays.

[0154] In one embodiment, the angle between the target light and the surface normal is smaller than a preset angle, and the preset angle includes π / 2.

[0155] In one embodiment, when the processor acquires the pseudo labels of the plurality of target light rays based on the plurality of projected image data, the processor is configured to implement:

[0156] When the number of the plurality of target light rays is less than a preset number threshold, using linear interpolation to obtain the pseudo labels of the radiation values ​​of the plurality of target light rays from the plurality of projected image data;

[0157] In a case where the number of the plurality of target light rays is greater than or equal to a preset number threshold, a spherical harmonic function is used to obtain the radiation distribution of the plurality of target light rays from the projected plurality of image data as the pseudo labels.

[0158] In one embodiment, when implementing the acquiring of the plurality of sample data, the processor is configured to implement:

[0159] Acquire multi-perspective image data collected by the autonomous driving platform;

[0160] Performing distortion correction and color balancing on the multi-view image data to obtain multi-view sample image data;

[0161] Abnormal image data in the multi-view sample image data is removed to obtain the sample data.

[0162] In one embodiment, the sample data further includes posture data and depth information of the autonomous driving platform;

[0163] The posture data, the depth information and the multi-view image data in the sample data are collected in the same time sequence.

[0164] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0165] Acquire a plurality of sample data, wherein the sample data includes image data from multiple perspectives;

[0166] Inputting the plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field;

[0167] extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on a plurality of surface point coordinates in the three-dimensional grid;

[0168] Projecting the plurality of target light rays onto the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data;

[0169] Based on the plurality of sample data and the pseudo labels of the plurality of target light rays, the initial neural radiation field is retrained to obtain a trained target neural radiation field.

[0170] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned three-dimensional reconstruction method embodiment, and will not be repeated here.

[0171] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0172] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the three-dimensional reconstruction method of the present application.

[0173] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0174] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0175] It should be understood that the term "and / or" used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0176] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A three-dimensional reconstruction method based on neural radiation field, characterized in that: include: Acquire a plurality of sample data, wherein the sample data includes image data from multiple perspectives; Inputting the plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field; extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on a plurality of surface point coordinates in the three-dimensional grid; Projecting the plurality of target light rays onto the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data; retraining the initial neural radiation field based on the plurality of sample data and the pseudo labels of the plurality of target light rays to obtain a trained target neural radiation field; The target nerve radiation field is used to perform three-dimensional reconstruction on the image data to be processed.

2. The three-dimensional reconstruction method according to claim 1, wherein: The surface point coordinates include vertex coordinates; and generating a plurality of target rays based on the plurality of surface point coordinates in the three-dimensional grid comprises: Obtaining coordinates of multiple vertices in the three-dimensional grid; Taking each vertex coordinate as a starting point, a plurality of target rays in random directions corresponding to each vertex coordinate are generated.

3. The three-dimensional reconstruction method according to claim 1, wherein: After generating a plurality of target rays based on the coordinates of a plurality of surface points in the three-dimensional grid, the method further includes: Performing a visibility check on each of the target rays to determine whether each of the target rays is blocked by an object; Target rays blocked by the object are removed from the multiple target rays.

4. The three-dimensional reconstruction method according to claim 1, wherein: The angle between the target light and the surface normal is smaller than a preset angle, and the preset angle includes π / 2.

5. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that: The acquiring of pseudo labels of the plurality of target light rays based on the plurality of projected image data comprises: When the number of the plurality of target light rays is less than a preset number threshold, using linear interpolation to obtain the pseudo labels of the radiation values ​​of the plurality of target light rays from the plurality of projected image data; In a case where the number of the plurality of target light rays is greater than or equal to a preset number threshold, a spherical harmonic function is used to obtain the radiation distribution of the plurality of target light rays from the projected plurality of image data as the pseudo labels.

6. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that: The obtaining of a plurality of sample data includes: Acquire multi-perspective image data collected by the autonomous driving platform; Performing distortion correction and color balancing on the multi-view image data to obtain multi-view sample image data; Abnormal image data in the multi-view sample image data is removed to obtain the sample data.

7. The three-dimensional reconstruction method according to claim 6, wherein: The sample data also includes the posture data and depth information of the autonomous driving platform; The posture data, the depth information and the multi-view image data in the sample data are collected in the same time sequence.

8. A neural radiation field model training method, characterized in that: include: Acquire a plurality of sample data, wherein the sample data includes image data from multiple perspectives; Inputting the plurality of sample data into a preset neural radiation field for training to obtain a trained initial neural radiation field; extracting a three-dimensional grid from the three-dimensional radiation field generated by the initial neural radiation field, and generating a plurality of target rays based on a plurality of surface point coordinates in the three-dimensional grid; Projecting the plurality of target light rays onto the plurality of image data, and acquiring pseudo labels of the plurality of target light rays based on the projected plurality of image data; Based on the plurality of sample data and the pseudo labels of the plurality of target light rays, the initial neural radiation field is retrained to obtain a trained target neural radiation field.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the three-dimensional reconstruction method as described in any one of claims 1 to 7, or implements the model training method as described in claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the three-dimensional reconstruction method according to any one of claims 1 to 7, or implements the model training method according to claim 8.

Images