A NeRF modeling method integrating time dimension and hierarchical sampling strategy

Abstract

The present invention discloses a NeRF modeling method integrating time dimension and hierarchical sampling strategy, including constructing a data set; sampling M coarse sampling points for each ray of an RGB image in the data set; encoding the position of the coarse sampling points, and then outputting the volume density and color of the coarse sampling points by MLP; calculating the weight and cumulative weight of the coarse sampling points; generating N fine sampling points based on the cumulative weight, calculating the volume density, color, opacity and cumulative transmittance of the fine sampling points; calculating the color of the ray by combining the coarse sampling points and the fine sampling points; constructing the loss function L of MLP; and training MLP to obtain a three-dimensional scene modeling model. The present invention improves rendering efficiency through a hierarchical sampling strategy, integrates explicit spatial structures during encoding to enhance the model's ability to capture complex geometry, and increases time encoding to efficiently cope with dynamic changes in the scene, thereby having better performance and efficiency when processing large-scale, complex, and dynamic three-dimensional scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a NeRF modeling method integrating a time dimension and a hierarchical sampling strategy. Background Art

[0002] In modern three-dimensional visual scenes such as virtual reality, film and television special effects, and digital heritage protection, high-precision three-dimensional reconstruction and rendering technology has become an indispensable and important means. Through three-dimensional modeling and reproduction of scenes, the digital world can reflect the structure and details of the real world in a realistic way, thereby bringing users a more immersive experience. However, traditional three-dimensional modeling and rendering technologies usually rely on cumbersome manual modeling or high-cost laser scanning. These methods are not only time-consuming and labor-intensive, but also difficult to accurately capture subtle changes in complex dynamic scenes. At the same time, in the process of three-dimensional reconstruction, problems such as dynamic scene modeling, multi-view image integration, and large-scale scene processing also bring huge challenges to traditional methods.

[0003] The NeRF (Neural Radiance Field) model is a novel view synthesis method that uses volume rendering with an implicit neural scene representation through Multi Layer Perceptrons (MLP). The basic form of the NeRF model is to represent a 3D scene as a radiance field approximated by a neural network. The radiance field describes the color and volume density of each point in the scene and each viewing direction. This is written , Represents the input of the NeRF model, which is a five-dimensional vector: including any camera pose x=(x,y,z) and camera direction d=(θ,Φ), where d=(θ,Φ) is a spherical coordinate representation, including the azimuth angle θ and the pitch angle Φ; Represents the output of the NeRF model, which are volume density (opacity σ) and color c. Volume density is only related to spatial position, while color is related to spatial position and observation angle. The implementation method is as follows: the five-dimensional vector is first input into the MLP network, and opacity σ and a 256-dimensional intermediate feature are output. The intermediate feature is then input into an additional fully connected layer (128 dimensions) together with d=(θ,Φ) to predict the color. Obtaining opacity σ and color c requires two MLP passes, which can be regarded as two branches of the same neural network or as two independent MLPs. Summary of the invention

[0004] The purpose of the present invention is to provide a NeRF modeling method that solves the above problems, can significantly improve the realism and real-time performance of three-dimensional scene rendering, shorten the rendering time, and make the modeling of large-scale complex scenes more efficient, integrating the time dimension and the hierarchical sampling strategy.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a NeRF modeling method integrating time dimension and hierarchical sampling strategy comprises the following steps;

[0006] S1, constructing a data set, the data set includes RGB images of each perspective of the three-dimensional scene captured by a camera, and for the i-th RGB image I i , the time label is t i , I i The camera origin O i The ray pointing to pixel p is r i,p , the viewing direction is d i,p ;

[0007] S2, in ray r i,p The coarse sampling obtains M coarse sampling points, among which the mth coarse sampling point In r i,p The position on , the three-dimensional coordinates in the three-dimensional scene are , 1≤m≤M;

[0008] S3, position encoding the coarse sampling points generates input feature vectors, feeds the input feature vectors into MLP, and outputs the volume density and color of the coarse sampling points, where The processing method includes steps S31 to S34;

[0009] S31, yes The three-dimensional coordinate x i,p,m , viewing direction d i,p and time tag t i Perform position encoding respectively and obtain the corresponding encoding γ(x i,p,m ),γ(d i,p ),γ(t i );

[0010] S32, dividing the three-dimensional space of the three-dimensional scene into a plurality of voxels, each voxel corresponds to an intermediate feature, and according to x i,p,m Find The corresponding intermediate feature f i,p,m ;

[0011] S33, according to the formula ,generate The input feature vector z i,p,m , where ‖ represents the concatenation operation of vectors;

[0012] S34, z i,p,m Send to MLP, output The volume density ,color ;

[0013] S4, calculate r i,p The weight of each coarse sampling point on Weight According to the following formula:

[0014] ,

[0015] ,

[0016] ,

[0017] In the formula, , , They are opacity, cumulative transmittance and sampling interval, exp is the natural exponential function, j is the jth coarse sampling point among the 1st to (m-1)th coarse sampling points;

[0018] S5, construct the cumulative weights of the coarse sampling points and normalize them, where The cumulative weight is CDF(m), and the normalized value obtained by normalization is CDF norm (m);

[0019] ,

[0020] S6, generating N fine sampling points based on the cumulative weights, and obtaining the volume density, color, opacity and cumulative transmittance of the fine sampling points, including steps S61-S63;

[0021] S61, generate N uniformly distributed random numbers u1~u in the interval [0,1] N , where the nth random number is u n , 1≤n≤N;

[0022] S62, for u n , find the CDF that satisfies norm (m)≥u n And the smallest m, generate u according to the following formula n The corresponding fine sampling points In r i,p Position on ;

[0023] ,

[0024] S63, according to steps S3 and S4, obtain The volume density ,color , Opacity and the cumulative transmittance ;

[0025] S7, combine the coarse sampling points and the fine sampling points to calculate the ray r i,p The color C(r i,p );

[0026] ,

[0027] S8, construct the loss function L of MLP;

[0028] ,

[0029] Where B is the total number of RGB images of the 3D scene, and P is I i The total number of pixels in C gt (r i,p ) is the true color of pixel p, is the L2 norm, L tc is the temporal consistency loss term, λ is L tc The weight of

[0030] S9, use the dataset to train the MLP to minimize L, and obtain a three-dimensional scene modeling model for reconstruction of new perspective RGB images in three-dimensional scenes.

[0031] As a preference: ray r i,p According to the following formula:

[0032] ,

[0033] Where, d i,p For O i The unit direction vector pointing to pixel p, t n ,t f are the near and far boundaries of the ray, respectively, t c is a point on the ray, r i,p (t c ) is r i,p Upper point c The three-dimensional coordinates of .

[0034] Preferably, in S31, the position code is obtained according to the following formula:

[0035] ,

[0036] Where v is the encoding parameter, v∈{x i,p,m ,di,p ,t i}, L is the number of position encoding layers.

[0037] As a preference: CDF(m) corresponds to the normalized value CDF norm (m) is obtained according to the following formula:

[0038] .

[0039] As a preference: In S32, search The corresponding intermediate feature f i,p,m The method is to use x i,p,m Sure The voxel where the voxel is located is used as the intermediate feature corresponding to the voxel. The corresponding intermediate feature f i,p,m .

[0040] Preferably: in S9, generating a new viewing angle RGB image in the three-dimensional scene includes steps S91 to S94;

[0041] S91, generating rays from the camera origin to each pixel in the new viewing angle RGB image;

[0042] S92, randomly select a ray, generate sampling points on it, and generate an input feature vector for each sampling point, wherein the sampling points include coarse sampling points and fine sampling points;

[0043] S93, inputting the input feature vectors of all sampling points into the three-dimensional scene modeling model, outputting the volume density and color of each sampling point, and calculating the color of the ray;

[0044] S94, process each ray in sequence according to S92 and S93 to obtain a new viewing angle RGB image.

[0045] Compared with the prior art, the advantages of the present invention are:

[0046] (1) Accelerated training and rendering based on hierarchical sampling: This paper divides the sampling process into two stages: coarse sampling and fine sampling. The results of coarse sampling are used to guide the location of fine sampling, thereby achieving fine sampling in important areas. This method enables the model to concentrate computing resources on the areas that are most important for the rendering results, greatly improving the rendering accuracy while maintaining computing efficiency. This ensures that NeRF has both high computing efficiency and fine visual effects when rendering high-quality complex and dynamically changing scenes.

[0047] (2) Introducing a dynamic scene modeling mechanism: Time tags are introduced when encoding sampling points, which expands the application scope of NeRF and enables it to handle scenes containing dynamic elements and capture information about time changes. Time information is added to the input features, and time consistency constraints are introduced in the loss function to ensure that the model can accurately reflect dynamic changes.

[0048] (3) Hybrid representation integrating explicit spatial structure: When encoding the positions of coarse sampling points and fine sampling points, an explicit spatial structure is introduced to divide the three-dimensional space into several voxels. Each voxel stores intermediate features. For each sampling point, the corresponding intermediate features are quickly retrieved through the spatial index and added to the input feature vector. This method of combining MLP implicit expression with explicit spatial structure can efficiently organize spatial data, improve memory utilization and the ability to capture geometric details, and increase query speed. It is particularly suitable for modeling needs of large-scale scenes.

[0049] In summary, the present invention can significantly improve rendering efficiency, enhance the model's ability to capture complex geometry, and efficiently cope with dynamic changes in the scene. These innovations enable the model to have better performance and efficiency when processing large-scale, complex, and dynamic three-dimensional scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the present invention;

[0051] Figure 2 A schematic diagram of generating coarse sampling points and fine sampling points under a new viewing angle in Example 3;

[0052] Figure 3 This is the effect diagram of the new perspective RGB image reconstructed in Example 3. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings.

[0054] Example 1: See Figure 1 ,A NeRF modeling method integrating the time dimension and the hierarchical sampling strategy includes the following steps;

[0055] S1, constructing a data set, the data set includes RGB images of each perspective of the three-dimensional scene captured by a camera, and for the i-th RGB image I i , the time label is t i , I i The camera origin O i The ray pointing to pixel p is r i,p , the viewing direction is d i,p ;

[0056] S2, in ray ri,p The coarse sampling obtains M coarse sampling points, among which the mth coarse sampling point In r i,p The position on , the three-dimensional coordinates in the three-dimensional scene are , 1≤m≤M;

[0057] S3, position encoding the coarse sampling points generates input feature vectors, feeds the input feature vectors into MLP, and outputs the volume density and color of the coarse sampling points, where The processing method includes steps S31 to S34;

[0058] S31, yes The three-dimensional coordinate x i,p,m , viewing direction d i,p and time tag t i Perform position encoding respectively and obtain the corresponding encoding γ(x i,p,m ),γ(d i,p ),γ(t i );

[0059] S32, dividing the three-dimensional space of the three-dimensional scene into a plurality of voxels, each voxel corresponds to an intermediate feature, and according to x i,p,m Find The corresponding intermediate feature f i,p,m ;

[0060] S33, according to the formula ,generate The input feature vector z i,p,m , where ‖ represents the concatenation operation of vectors;

[0061] S34, z i,p,m Send to MLP, output The volume density ,color ;

[0062] S4, calculate r i,p The weight of each coarse sampling point on Weight According to the following formula:

[0063] ,

[0064] ,

[0065] ,

[0066] In the formula, , , They are opacity, cumulative transmittance and sampling interval, exp is the natural exponential function, j is the jth coarse sampling point among the 1st to (m-1)th coarse sampling points;

[0067] S5, construct the cumulative weights of the coarse sampling points and normalize them, where The cumulative weight is CDF(m), and the normalized value obtained by normalization is CDF norm (m);

[0068] ,

[0069] S6, generating N fine sampling points based on the cumulative weights, and obtaining the volume density, color, opacity and cumulative transmittance of the fine sampling points, including steps S61-S63;

[0070] S61, generate N uniformly distributed random numbers u1~u in the interval [0,1] N , where the nth random number is u n , 1≤n≤N;

[0071] S62, for u n , find the CDF that satisfies norm (m)≥u n And the smallest m, generate u according to the following formula n The corresponding fine sampling points In r i,p Position on ;

[0072] ,

[0073] S63, according to steps S3 and S4, obtain The volume density ,color , Opacity and the cumulative transmittance ;

[0074] S7, combine the coarse sampling points and the fine sampling points to calculate the ray r i,p The color C(r i,p );

[0075] ,

[0076] S8, construct the loss function L of MLP;

[0077] ,

[0078] Where B is the total number of RGB images of the 3D scene, and P is I i The total number of pixels in C gt (ri,p ) is the true color of pixel p, is the L2 norm, L tc is the temporal consistency loss term, λ is L tc The weight of

[0079] S9, use the dataset to train the MLP to minimize L, and obtain a three-dimensional scene modeling model for reconstruction of new perspective RGB images in three-dimensional scenes.

[0080] Example 2: See Figure 1 , based on Example 1, ray r i,p According to the following formula:

[0081] ,

[0082] Where, d i,p For O i The unit direction vector pointing to pixel p, t n ,t f are the near and far boundaries of the ray, respectively, t c is a point on the ray, r i,p (t c ) is r i,p Upper point c The three-dimensional coordinates of .

[0083] In S31, the position code is obtained according to the following formula;

[0084] ,

[0085] Where v is the encoding parameter, v∈{x i,p,m ,d i,p ,t i}, L is the number of position encoding layers.

[0086] Normalized value CDF corresponding to CDF(m) norm (m) is obtained according to the following formula:

[0087] .

[0088] In S32, find The corresponding intermediate feature f i,p,m The method is to use x i,p,m Sure The voxel where the voxel is located is used as the intermediate feature corresponding to the voxel. The corresponding intermediate feature f i,p,m .

[0089] Example 3, see Figure 1 , Figure 2 and Figure 3, based on Example 1, in S9, generating a new viewing angle RGB image in a three-dimensional scene includes steps S91 to S94;

[0090] S91, generating rays from the camera origin to each pixel in the new viewing angle RGB image;

[0091] S92, randomly select a ray, generate sampling points on it, and generate an input feature vector for each sampling point, wherein the sampling points include coarse sampling points and fine sampling points;

[0092] S93, inputting the input feature vectors of all sampling points into the three-dimensional scene modeling model, outputting the volume density and color of each sampling point, and calculating the color of the ray;

[0093] S94, process each ray in sequence according to S92 and S93 to obtain a new viewing angle RGB image.

[0094] Taking a specific 3D scene modeling model as an example, after selecting a new perspective, we generate coarse sampling points and fine sampling points of all rays under this perspective, as shown in the schematic diagram Figure 2 After completing the ray sampling and model calculation, the reconstructed new perspective RGB image is generated by rendering, as shown in Figure 3 The results show that the images generated by this method have significant advantages in clarity and realism.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A NeRF modeling method integrating time dimension and hierarchical sampling strategy, characterized by: The steps include: S1, constructing a data set, the data set includes RGB images of each perspective of the three-dimensional scene captured by a camera, for the i-th RGB image I i , the time label is t i , I i The camera origin O i The ray pointing to pixel p is r i,p , the viewing direction is d i,p ; S2, in ray r i,p The coarse sampling obtains M coarse sampling points, among which the mth coarse sampling point In r i,p The position on , the three-dimensional coordinates in the three-dimensional scene are , 1≤m≤M; S3, position encoding the coarse sampling points generates input feature vectors, feeds the input feature vectors into MLP, and outputs the volume density and color of the coarse sampling points, where The processing method includes steps S31 to S34; S31, yes The three-dimensional coordinate x i,p,m , viewing direction d i,p and time tag t i Perform position encoding respectively and obtain the corresponding encoding γ(x i,p,m ),γ(d i,p ),γ(t i ); S32, dividing the three-dimensional space of the three-dimensional scene into a plurality of voxels, each voxel corresponds to an intermediate feature, and i,p,m Find The corresponding intermediate feature f i,p,m ; S33, according to the formula ,generate The input feature vector z i,p,m , where ‖ represents the concatenation operation of vectors; S34, z i,p,m Send to MLP, output The volume density ,color ; S4, calculate r i,p The weight of each coarse sampling point on Weight According to the following formula: , , , In the formula, , , They are opacity, cumulative transmittance and sampling interval, exp is the natural exponential function, j is the jth coarse sampling point among the 1st to (m-1)th coarse sampling points; S5, construct the cumulative weights of the coarse sampling points and normalize them, where The cumulative weight is CDF(m), and the normalized value obtained by normalization is CDF norm (m); , S6, generating N fine sampling points based on the cumulative weights, and obtaining the volume density, color, opacity and cumulative transmittance of the fine sampling points, including steps S61-S63; S61, generate N uniformly distributed random numbers u1~u in the interval [0,1] N , where the nth random number is u n , 1≤n≤N; S62, for u n , find the CDF that satisfies norm (m)≥u n And the smallest m, generate u according to the following formula n The corresponding fine sampling points In r i,p Position on ; , S63, according to steps S3 and S4, obtain The volume density ,color , Opacity and the cumulative transmittance ; S7, combine the coarse sampling points and the fine sampling points to calculate the ray r i,p The color C(r i,p ); , S8, construct the loss function L of MLP; , Where B is the total number of RGB images of the 3D scene, and P is I i The total number of pixels in C gt (r i,p ) is the true color of pixel p, is the L2 norm, L tc is the temporal consistency loss term, λ is L tc The weight of S9, use the dataset to train the MLP to minimize L, and obtain a three-dimensional scene modeling model for reconstruction of new perspective RGB images in three-dimensional scenes.

2. According to claim 1, a NeRF modeling method integrating time dimension and hierarchical sampling strategy is characterized in that: Ray i,p According to the following formula: , Where, d i,p For O i The unit direction vector pointing to pixel p, t n ,t f are the near and far boundaries of the ray, respectively, t c is a point on the ray, r i,p (t c ) is r i,p Upper point c The three-dimensional coordinates of .

3. The NeRF modeling method integrating the time dimension and the hierarchical sampling strategy according to claim 1 is characterized in that: In S31, the position code is obtained according to the following formula; , Where v is the encoding parameter, v∈{x i,p,m ,d i,p ,t i }, L is the number of position encoding layers.

4. The NeRF modeling method integrating the time dimension and the hierarchical sampling strategy according to claim 1 is characterized in that: Normalized value CDF corresponding to CDF(m) norm (m) is obtained according to the following formula: 。 5. The NeRF modeling method integrating the time dimension and the hierarchical sampling strategy according to claim 1 is characterized in that: In S32, find The corresponding intermediate feature f i,p,m The method is to use x i,p,m Sure The voxel where the voxel is located is used as the intermediate feature corresponding to the voxel. The corresponding intermediate feature f i,p,m .

6. The NeRF modeling method integrating the time dimension and the hierarchical sampling strategy according to claim 1 is characterized in that: In S9, generating a new perspective RGB image in the three-dimensional scene includes steps S91 to S94; S91, generating rays from the camera origin to each pixel in the new viewing angle RGB image; S92, randomly select a ray, generate sampling points on it, and generate an input feature vector for each sampling point, wherein the sampling points include coarse sampling points and fine sampling points; S93, inputting the input feature vectors of all sampling points into the three-dimensional scene modeling model, outputting the volume density and color of each sampling point, and calculating the color of the ray; S94, process each ray in sequence according to S92 and S93 to obtain a new viewing angle RGB image.

Images