Universal and controllable radiation field enhancement method based on space perception

Abstract

The invention provides a universal and controllable radiation field enhancement method based on spatial perception. The method comprises the following steps: step 1, obtaining training data; 2, light rays and sampling points are generated according to the training data; 3, constructing a neural radiation field hybrid model; 4, inputting the coordinates of the sampling points and the direction of the light into the neural radiation field hybrid model constructed in the step 3, and calculating the volume density and color of each sampling point; 5, calculating an RGB color value of a pixel corresponding to each light ray; step 6, according to the volume density and color of each sampling point and the RGB color value corresponding to each light, training the neural radiation field hybrid model constructed in the step 3 to obtain a trained neural radiation field hybrid model; and step 7, using the trained neural radiation field hybrid model to render a two-dimensional picture under the view angle according to the camera pose of the new view angle, and completing universal and controllable radiation field enhancement based on spatial perception.

Description

Technical Field

[0001] The invention relates to a radiation field enhancement method, in particular to a universal and controllable radiation field enhancement method based on space perception. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the emergence of concepts such as the metaverse, virtual reality, and digital twins, people have an increasing demand for digital reconstruction of complex scenes and the production of realistic and high-quality images. How to render more realistic virtual scenes from images is a topic that is currently being studied in the fields of computer graphics and computer vision. Among them, the synthesis of new perspectives is a task that is of particular concern. Reconstructing a continuous three-dimensional scene through a two-dimensional view with a limited perspective, accurately restoring the scene's geometric structure, surface texture, lighting and other information, in order to generate realistic new perspectives, is of great significance to application fields such as augmented reality, virtual reality, and video games.

[0004] Traditional scene expression based on image rendering synthesis requires the geometric shape and attributes of objects as a priori, and has large computational complexity and low rendering efficiency. With the rapid development of deep learning-driven image rendering technology, Neural Radiance Fields (NeRF) pioneered the use of neural networks to represent scenes, enabling it to have the ability to synthesize new views with photo-realistic effects.

[0005] NeRF was originally proposed by Mildenhall et al. at the 2020 ECCV conference. This method represents a three-dimensional scene as a radiation field approximated by a multi-layer perceptron (MLP), describes the color and volume density of each sampling point in the scene under a specific observation direction, and synthesizes images from a new perspective through volume rendering.

[0006] Previously, many solutions have been proposed to address the problems of slow NeRF speed, jagged rendering, and sampling efficiency, but they are all based on the basic assumptions of the independence of spatial points and rays, and there is a problem of insufficient modeling of spatial dependencies.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide a universal and controllable radiation field enhancement method based on spatial perception in view of the deficiencies in the prior art.

[0009] In order to solve the above technical problems, the present invention discloses a universal and controllable radiation field enhancement method based on space perception, comprising the following steps:

[0010] Step 1, obtain training data;

[0011] Step 2, generating light and sampling points according to the training data;

[0012] Step 3, constructing a neural radiation field hybrid model;

[0013] Step 4, inputting the coordinates of the sampling point and the direction of the light into the neural radiation field hybrid model constructed in step 3, and calculating the volume density and color of each sampling point;

[0014] Step 5, calculate the RGB color value of the pixel corresponding to each light ray;

[0015] Step 6, training the neural radiation field mixed model constructed in step 3 according to the volume density and color of each sampling point obtained in step 4 and step 5, and the RGB color value corresponding to each light ray, to obtain a trained neural radiation field mixed model;

[0016] Step 7: Use the trained neural radiation field hybrid model to render a two-dimensional image from the new perspective according to the camera pose of the new perspective, and complete the universal and controllable radiation field enhancement based on spatial perception.

[0017] Furthermore, the training data is obtained in step 1, that is, two-dimensional images under different viewing angles are obtained by taking pictures with a camera, and the camera posture corresponding to each image is recorded, and the two-dimensional images and the corresponding camera posture are used as training data.

[0018] Furthermore, the generation of light and sampling points in step 2, that is, using the neural radiation field to model the light, assumes that each pixel in the two-dimensional image emits a light ray, the starting point of the light ray is the camera position, the direction is the direction vector pointing from the camera position to the pixel position in the two-dimensional image plane, and each pixel corresponds to a light ray; the starting point coordinates and direction vector of the light ray are calculated according to the camera posture, and sampling is performed on the light ray to obtain the coordinates of the sampling point, specifically including:

[0019] Step 2-1, based on the camera pose and the pixel coordinates in the two-dimensional image plane, calculate and generate the starting point coordinates and direction vector of the light ray corresponding to the pixel in the world coordinate system, the specific method includes:

[0020] Randomly select an image block on each two-dimensional image, and calculate the starting coordinates and direction vector of the light corresponding to the pixel in the world coordinate system through the pixels of the image block; let the length and width of the image block be p h and pw , then the tensor corresponding to the RGB color of the image block pixel The coordinate tensor of the starting point of the light corresponding to the pixel of the image block Direction vector tensor

[0021] Step 2-2, use the uniform sampling method to sample the light to generate the initial sampling point, and calculate the coordinates of the initial sampling point in the world coordinate system;

[0022] The uniform sampling method described in step 2-2 is as follows:

[0023]

[0024] Among them, t n is the shortest distance between the sampling point and the camera, t f is the farthest distance between the sampling point and the camera, t i is the distance from the i-th sampling point to the camera obtained by uniform sampling method, N init is the number of initial sampling points, U represents uniform distribution, and ~ represents obedience to a certain probability distribution.

[0025] Furthermore, the construction of the neural radiation field hybrid model described in step 3 includes the following steps:

[0026] Step 3-1, construct two granularity models under the neural radiation field hybrid model, namely a coarse model and a fine model; wherein the coarse model includes: a sampling point coordinate feature encoder, a direction vector feature encoder, a volume density decoder and a color decoder; the output of the coarse model is the volume density σ of the sampling point coarse and color c coarse ;

[0027] The fine model includes: a sampling point coordinate feature encoder, a direction vector feature encoder, a volume density decoder, a color decoder and a mixing module; wherein the mixing module includes: a cross-ray cross-sampling point mixing module and a same-ray cross-sampling point mixing module; the output of the fine model is the volume density feature F of the sampling point σ 、Color feature F c , volume density σ fine 、Color c fine , the volume density after mixing across rays and sampling points σ inter_mixed , the color c after mixing across rays and sampling points rnter_mixed , the volume density after mixing cross-rays and same-rays σ intra_inter_mixed and the color c after mixing the cross-light and the same-light intra_inter_mixed ;

[0028] Step 3-2, constructing a cross-ray and cross-sampling point mixing module in the radiation field mixing model;

[0029] Step 3-3, construct a same-ray cross-sampling point mixing module in the radiation field mixing model.

[0030] Furthermore, the sampling point coordinate feature encoder described in step 3-1 is composed of eight layers of two-dimensional convolutional layers with a convolution kernel of 1*1, and encodes the input sampling point coordinate features to obtain volume density features;

[0031] The volume density decoder is composed of a two-dimensional convolution layer with a convolution kernel of 1*1, which decodes the volume density feature to obtain the volume density of each sampling point;

[0032] The directional vector feature encoder is composed of a two-dimensional convolution layer with a convolution kernel of 1*1. After concatenating the volume density feature with the input directional vector feature, the directional vector feature encoder is used to encode the feature again to obtain the color feature.

[0033] The color decoder is composed of a two-dimensional convolution layer with a convolution kernel of 1*1, which decodes the color features to obtain the color of each sampling point.

[0034] Furthermore, the cross-ray and cross-sampling point mixing module described in step 3-2 is composed of a volume density mixing sub-network and a color mixing sub-network, wherein each sub-network is composed of two layers of two-dimensional convolutional layers with a convolution kernel of k*k;

[0035] The cross-ray and cross-sampling point mixing module is used to predict the volume density weight and color weight of each sampling point at the same depth as the adjacent sampling points on different rays, and obtain the volume density σ of each sampling point after cross-ray and cross-sampling point mixing. inter_mixed and the color c after mixing across rays and sampling points inter_mixed , the specific prediction method is as follows:

[0036] Step 3-2-1: The volume density feature F obtained by the sampling point in the fine model σ Input the volume density hybrid subnetwork to predict the volume density weight W_inter of the adjacent sampling points at the same depth as the sampling point on different rays to the sampling point. σ , for the volume density σ fine Perform weighted summation to obtain the volume density σ after mixing across rays and sampling points inter_mixed , the specific method is as follows:

[0037]

[0038] Where k is the number of adjacent sampling points, is the volume density weight of the i-th adjacent sampling point to the sampling point, is the volume density of the i-th adjacent sampling point;

[0039] Step 3-2-2: The color feature F obtained by the sampling point in the fine model c Input into the color mixing subnetwork, and predict the color weight W_inter of the adjacent sampling points at the same depth as the sampling point on different rays to the sampling point. c , for the color c fine Perform weighted summation to obtain the color c after mixing across light and sampling points inter_mixed , the specific method is as follows:

[0040]

[0041] Where k is the number of adjacent sampling points, is the body color weight of the i-th adjacent sampling point to the sampling point, is the color of the i-th adjacent sampling point.

[0042] Furthermore, the same-ray cross-sampling point mixing module described in step 3-3 is composed of a volume density mixing sub-network and a color mixing sub-network, wherein each sub-network is composed of two layers of one-dimensional convolution neural networks with a convolution kernel of k;

[0043] The same-ray cross-sampling point mixing module is used to predict the volume density weight and color weight of each sampling point on the same ray to obtain the volume density σ of each sampling point after mixing across the ray and the same ray. intra_inter_mixed and the color c after mixing the cross-light and the same-light intra_inter_mixed , the specific prediction method is as follows:

[0044] Step 3-3-1: The volume density feature F obtained by the sampling point in the fine model σ Input the volume density hybrid subnetwork to predict the volume density weight W_intra of the adjacent sampling point on the same ray to the sampling point σ , for step 3-2, the volume density σ after blending across rays and sampling points inter_mixed Perform weighted summation to obtain the volume density σ after mixing both the cross-ray and the same-ray intra_inter_mixed , the specific method is as follows:

[0045]

[0046] Where k is the number of adjacent sampling points, is the volume density weight of the i-th adjacent sampling point to the sampling point, is the volume density of the i-th adjacent sampling point after mixing across rays and sampling points;

[0047] Step 3-3-2: The color feature F obtained by the sampling point in the fine model c Input into the color mixing subnetwork, and predict the color weight W_intra of the adjacent sampling point on the same ray to the sampling point c , for the color c after mixing across the light and across the sampling points in step 3-2 inter_mixed Perform weighted summation to obtain the color c after mixing the cross-light and the same-light intra_inter_mixed , the specific method is as follows:

[0048]

[0049] Where k is the number of adjacent sampling points, is the body color weight of the i-th adjacent sampling point to the sampling point, It is the color of the i-th adjacent sampling point after being mixed across the light and across the sampling point.

[0050] Furthermore, the calculation of the volume density and color of each sampling point described in step 4 includes the following steps:

[0051] Step 4-1, encoding the coordinates and direction vectors of the sampling points to obtain coordinate features and direction vector features. The specific encoding method is as follows:

[0052] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0053] Among them, p is the coordinate of the sampling point or the direction vector of the light where it is located, L is the encoding frequency, and γ(p) represents the encoded feature after encoding, where the coordinate feature is obtained after the coordinate encoding, and the direction vector feature is obtained after the direction vector encoding; the coordinate feature tensor is Where D x =3×(2×l x +1); the direction vector feature tensor is V∈ Where D v =3×(2×L v +1), N is the number of sampling points on each ray;

[0054] Step 4-2: The coordinate feature tensor of the initial sampling point described in step 2-2 With the direction vector feature tensor Input the coarse model in step 3 to obtain the volume density of the initial sampling point in the coarse model and color N init is the number of initial sampling points;

[0055] Step 4-3, using the hierarchical sampling method, based on the volume density σ of the sampling points in the coarse model coarse , and sample again to obtain fine sampling points, as follows:

[0056] Step 4-3-1, calculate the volume density σ of the sampling points on each ray in the coarse model coarse Normalize it and use it as the probability p that the sampling point is on the surface of the object in the current scene i , the specific method is as follows:

[0057]

[0058] Among them, N init is the number of initial sampling points on each ray;

[0059] Step 4-3-2: According to the probability p of each sampling point on each ray being on the surface of the object in the current scene, i Distribution, generate hierarchical sampling points through inverse transform sampling;

[0060] Step 4-3-3, merging the initial sampling point in step 2-2 and the hierarchical sampling point in step 4-3-2 into the final fine sampling point;

[0061] Step 4-4, encode the coordinates and direction vectors of the fine sampling points using the encoding method in step 4-1 to obtain the coordinate feature tensor and the direction vector feature tensor And input the fine model constructed in step 3 to obtain the volume density characteristics of the fine sampling points Color Features Body density color Volume density after blending across rays and sampling points The color after blending across rays and sampling points Volume density after mixing across rays and within rays The color after mixing both the cross-ray and the same-ray N fine is the number of fine sampling points, the specific steps are as follows:

[0062] Step 4-4-1, obtain the volume density feature F in the fine model through the coordinate feature encoder in the fine model σ ;

[0063] Step 4-4-2, obtain the color feature F in the fine model through the direction vector feature encoder in the fine model c ;

[0064] Step 4-4-3, obtain the volume density σ in the fine model through the volume density decoder in the fine model fine ;

[0065] Step 4-4-4, obtain the color c in the fine model through the color decoder in the fine model fine ;

[0066] Step 4-4-5, transform the above volume density feature F σ 、Color feature F c , volume density σ fine and color c fine , input the cross-ray and cross-sampling point mixing module to obtain the volume density σ after cross-ray and cross-sampling point mixing inter_mixed and color c inter_mixed ;

[0067] Step 4-4-6, the above volume density feature F σ 、Color feature F c , the volume density after mixing across rays and sampling points σ inter_mixed and the color c after mixing across rays and sampling points imter_mixed , input the same light cross sampling point mixing module to obtain the volume density σ after mixing the cross light and the same light intra_inter_mixed With color c intra_inter_mixed .

[0068] Furthermore, the specific method for calculating the RGB color value of the pixel corresponding to each light ray in step 5 includes:

[0069]

[0070] in, Represents the final RGB color value of the pixel corresponding to the light, σ i and c i is the volume density and color of the i-th sampling point, δ j =t j+1 -t j is the distance between adjacent sampling points on the same ray, T i is the cumulative penetration rate from the camera position to the i-th sampling point.

[0071] Further, the neural radiation field hybrid model constructed in step 3 is trained in step 6, that is, the parameters of the neural radiation field hybrid model are updated using the Adam optimizer by calculating the loss, wherein the method of calculating the loss includes:

[0072] Step 6-1, for the volume density σ obtained from the coarse model in step 4-2 coarse and color c coarse , calculate the pixel RGB value obtained by rendering the radiation field in the coarse model using the method in step 5

[0073] Step 6-2, for the volume density σ obtained from the fine model in step 4-2 fine and color c fine , calculate the pixel RGB value obtained by rendering the radiation field in the fine model using the method in step 5

[0074] Step 6-3, for the volume density σ mixed by the cross-ray and cross-sampling point mixing module in the fine model in step 4-4 inter_mixed and color c inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the cross-ray and cross-sampling point mixing module in the fine model using the method in step 5

[0075] Step 6-4, for the volume density σ mixed by the same light cross sampling point mixing module in step 4-5 intra_nter_mixed and color c intra_inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the same light cross-sampling point mixing module in the fine model using the method in step 5

[0076] Step 6-5, calculate the loss as follows:

[0077] L RF =L coarse +L fine

[0078]

[0079] Among them, L RF is the total loss of the neural radiation field hybrid model, R is the set of rays, and L coarse and L fine are the losses of the coarse model and the fine model respectively, It is the true RGB value of the pixel corresponding to the image block pixel.

[0080] Beneficial effects:

[0081] 1) This method proposes a radiance field mixing strategy, which eliminates the implicit assumption of spatial sampling point independence of neural radiance field-based methods and strengthens their ability to detect the structural structure of the scene.

[0082] 2) This method can significantly improve the quality of new view synthesis based on neural radiance field based methods while increasing the inference time by only 25%. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0084] Figure 1 The flowchart of the radiation field enhancement method based on space perception.

[0085] Figure 2 Flowchart for calculating the density and color of sampling points.

[0086] Figure 3 Schematic diagram of the process of calculating the initial sample point volume density and color for the coarse model.

[0087] Figure 4 Schematic diagram of the process of calculating the density and color of fine sample point volumes for the fine model. DETAILED DESCRIPTION

[0088] In view of the shortcomings of the existing neural radiation field-based methods, the present invention provides a universal and controllable radiation field enhancement method based on spatial perception, thereby improving the rendering quality when synthesizing new perspective images. The specific technical scheme is as follows: The present invention discloses a universal and controllable radiation field enhancement method based on spatial perception. The core of the method is to use a dynamic kernel to perform local fusion of adjacent sampling points of the radiation field, improve the spatial perception ability of the features, and thus improve the rendering quality of the synthesized image under a new perspective. Figure 1 As shown, the following steps are included:

[0089] Step 1: Read training data; read two-dimensional images from different perspectives and the camera pose corresponding to each image;

[0090] Step 2: Generate light and sampling points; the neural radiation field models light in the following way: each pixel emits a ray to simulate the process of light reaching the scene from the observer (or camera). The starting point of the ray is usually the camera position, and the direction is the direction vector from the camera position to a specific pixel on the image plane. Therefore, each pixel corresponds to a ray. The starting point coordinates and direction vector of the ray are calculated through the camera pose, and the coordinates of the sampling point are obtained by sampling on the ray. The process is divided into the following steps:

[0091] Step 2-1: Based on the camera pose and the pixel coordinates in the two-dimensional image plane, calculate and generate the starting point coordinates and direction vector of the light ray corresponding to the pixel in the world coordinate system. The specific method includes:

[0092] Randomly select an image block on each two-dimensional image, and calculate the starting coordinates and direction vector of the light corresponding to the pixel in the world coordinate system through the pixels of the image block; let the length and width of the image block be p h and p w , then the tensor corresponding to the RGB color of the image block pixel The coordinate tensor of the starting point of the light corresponding to the pixel of the image block Direction vector tensor

[0093] Step 2-2: Use uniform sampling to sample the light to generate the initial sampling point, and calculate the coordinates of the initial sampling point in the world coordinate system;

[0094] In step 2-2, uniform sampling is calculated as follows:

[0095]

[0096] Among them, t n is the shortest distance between the sampling point and the camera, t f is the farthest distance between the sampling point and the camera, t i is the distance from the i-th sampling point to the camera obtained by uniform sampling method, N init is the number of initial sampling points, U represents uniform distribution, and ~ represents obedience to a certain probability distribution.

[0097] Step 3: Construct a neural radiation field hybrid model, including the following steps:

[0098] Step 3-1, construct two granularity models under the neural radiation field hybrid model, namely a coarse model and a fine model; wherein the coarse model includes a coordinate feature encoder composed of eight layers of 1*1 two-dimensional convolutional layers, a volume density decoder composed of one layer of 1*1 two-dimensional convolutional layers, a direction vector feature encoder composed of one layer of 1*1 two-dimensional convolutional layers, and a color decoder composed of one layer of 1*1 two-dimensional convolutional layers.

[0099] Step 3-2, construct a detailed model, including a coordinate feature encoder composed of eight layers of two-dimensional convolutional layers with a convolution kernel of 1*1, a volume density decoder composed of one layer of two-dimensional convolutional layer with a convolution kernel of 1*1, a direction vector feature encoder composed of one layer of two-dimensional convolutional layer with a convolution kernel of 1*1, a color decoder composed of one layer of two-dimensional convolutional layer with a convolution kernel of 1*1, and a mixing module; wherein the mixing module includes: a cross-ray and cross-sampling point mixing module and a same-ray cross-sampling point mixing module;

[0100] Step 3-3, build a cross-ray and cross-sampling point mixing module, which consists of a volume density mixing sub-network and a color mixing sub-network. The volume density sub-network consists of two layers of two-dimensional convolutional layers with a convolution kernel of k*k, which is used to predict the volume density weight W_inter of each sampling point at the same depth as the adjacent sampling points on different rays. σ ; The color subnetwork consists of two layers of two-dimensional convolutional layers with convolution kernels of k*k, which are used to predict the color weight W_inter of each sampling point at the same depth as the adjacent sampling points on different rays. c ;

[0101] Step 3-4, build the same-ray cross-sampling point mixing module, which consists of a volume density mixing sub-network and a color mixing sub-network. The volume density sub-network consists of two one-dimensional convolution layers with a convolution kernel of k, which is used to predict the volume density weight W_intra of the adjacent sampling points on the same ray to each sampling point. σ ; The color subnetwork consists of two one-dimensional convolutional layers with a convolution kernel of k, which is used to predict the color weight W_intra of each sampling point on the same ray as the adjacent sampling point. c ;

[0102] Step 4: Input the coordinates and direction vector of the initial sampling point into the neural radiation field hybrid model in step 3 to calculate the volume density and color of each sampling point, such as Figure 2 As shown, it is divided into the following steps:

[0103] Step 4-1: The initial sampling point is obtained by uniform sampling in step 2-2. The coordinate information and direction vector information of the sampling point are encoded using the following encoding formula:

[0104] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0105] Among them, p is the coordinate of the sampling point or the direction vector of the light where it is located, L is the encoding frequency, and γ(p) represents the encoded feature after encoding, where the coordinate feature is obtained after the coordinate encoding, and the direction vector feature is obtained after the direction vector encoding; the coordinate feature tensor is Where D x =3×(2×L x +1); the direction vector feature tensor is Where D v =3×(2×L v +1), N is the number of sampling points on each ray;

[0106] Step 4-2: Convert the coordinate feature tensor of step 4-1 and the direction vector feature tensor Input the coarse model in the neural radiation field mixed model in step 3 to obtain the volume density σ of the initial sampling point coarse and color c coarse , this step is as follows Figure 3 As shown;

[0107] Step 4-3: Use the hierarchical sampling method based on the volume density σ of the sampling points in the coarse model coarse , and sample again to obtain fine sampling points to achieve the purpose of importance sampling, as follows:

[0108] Step 4-3-1, for each initial sampling point on the ray in step 4-2, the volume density σ coarse The specific method for normalization is as follows:

[0109]

[0110] Among them, p i Indicates the probability that the i-th sampling point is located on the surface of the object in the current scene, N init is the number of initial sampling points on each ray;

[0111] Step 4-3-2, generating hierarchical sampling points by inverse transform sampling according to the probability distribution in step 4-3-1;

[0112] Step 4-3-3, merging the initial sampling points in step 2-2 and the hierarchical sampling points in step 4-3-2 into the final fine sampling points;

[0113] Step 4-4: Encode the coordinates and direction vectors of the fine sampling points using the encoding method in step 4-1 to obtain the coordinate feature tensor and the direction vector feature tensor N fine is the number of fine sampling points, and the fine model constructed in step 3-2 is input, where the volume density feature F is obtained through the coordinate feature encoder σ , the direction vector feature encoder obtains the color feature F c , the volume density decoder obtains the volume density σ fine , the volume density decoder obtains the color c Fine , this step is as follows Figure 4 As shown;

[0114] Step 4-5: Convert the volume density feature F in step 4-4 σ With color feature F cInput the cross-ray and cross-sampling point blending module in step 3-3. For each sampling point, this module predicts the volume density weight W_inter of the adjacent sampling points at the same depth as the sampling point on different rays. σ W_inter with color weights c Finally, the volume density σ obtained in step 4-4 is fine With color c fine After weighted summation, we can get the volume density σ after mixing across light and sampling points. inter_mixed With color c inter_mixed , this step is as follows Figure 4 As shown. The weighted summation formula is as follows:

[0115]

[0116] Where k is the number of adjacent sampling points, is the volume density weight of the i-th adjacent sampling point to the sampling point, is the volume density of the ith adjacent sampling point, is the body color weight of the i-th adjacent sampling point to the sampling point, is the color of the i-th adjacent sampling point.

[0117] Step 4-6: Convert the volume density feature F in step 4-4 σ With color feature F c Input the same-ray cross-sampling point blending module in step 3-4. For each sampling point, this module predicts the volume density weight W_intra of the adjacent sampling points on the same ray. σ , with color weight W_intra c Finally, the volume density σ after mixing across light and sampling points obtained in step 4-4 is inter_mixed With color c inter_mixed After weighted summation, we can get the volume density σ after mixing the cross-ray and the same-ray. intra_inter_mixed and color c intra_inter_mixed , this step is as follows Figure 4 As shown. The weighted summation formula is as follows:

[0118]

[0119] Where k is the number of adjacent sampling points, is the volume density weight of the i-th adjacent sampling point to the sampling point, is the volume density of the i-th adjacent sampling point after mixing across rays and sampling points, is the body color weight of the i-th adjacent sampling point to the sampling point, It is the color of the i-th adjacent sampling point after being mixed across the light and across the sampling point.

[0120] Step 5: Calculate the RGB color value of each pixel corresponding to the light through the volume rendering equation;

[0121] The volume rendering equation is as follows:

[0122]

[0123] in, Represents the final RGB color value of the pixel corresponding to the light, σ i and c i is the volume density and color of the i-th sampling point, δ j =t j+1 -t j is the distance between adjacent sampling points on the same ray, T i is the cumulative penetration rate from the camera position to the i-th sampling point.

[0124] Step 6: Train the neural radiation field hybrid model in step 3; update the model parameters using the Adam optimizer by calculating the loss, where the loss calculation can be divided into the following steps:

[0125] Step 6-1: For the volume density σ obtained from the rough model in step 4-2 coarse Color c coarse , calculate the pixel RGB value obtained by rendering the radiation field in the coarse model using the method in step 5

[0126] Step 6-2: For the volume density σ obtained from the fine model in step 4-4 fine Color c fine , calculate the pixel RGB value obtained by rendering the radiation field in the fine model using the method in step 5

[0127] Step 6-3: For the volume density σ mixed by the cross-ray and cross-sample point mixing module in step 4-5 inter_mixed and color c inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the cross-ray and cross-sampling point mixing module in the fine model using the method in step 5

[0128] Step 6-4: For the volume density σ mixed by the same light cross sampling point mixing module in step 4-6 intra_nter_mixed and color c intra_inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the same light cross-sampling point mixing module in the fine model using the method in step 5

[0129] Step 6-5: For the neural radiation field hybrid model defined in step 3, the loss is calculated using the following formula:

[0130] L RF =L coarse +L fine

[0131]

[0132] Among them, L RF is the total loss of the neural radiation field hybrid model, R is the set of rays, and L coatse and L fine are the losses of the coarse model and the fine model respectively, is the true RGB value of the pixel corresponding to the pixel in the image block. Step 7: Render the two-dimensional image under the new perspective according to the camera pose of the new perspective.

[0133] Step 7 only involves the forward rendering process of the model. Given the camera pose at a specific viewing angle and the length and width of the synthesized image, the final synthesized RGB image is obtained according to steps 2 to 7.

[0134] Neural radiation field is a new perspective image generation method that has emerged in recent years with excellent results. This technology is based on deep learning, implicitly models three-dimensional scenes through neural networks, and uses volume rendering technology to achieve high-fidelity image synthesis. Traditional neural radiation field methods often assume that spatial sampling points and light are independent of each other when processing them, which limits the expressiveness of the synthesized image in terms of color continuity and spatial details. Therefore, the present invention proposes a universal and controllable radiation field enhancement method based on spatial perception, which reorganizes the sampling points and dynamically generates combination weights for the reorganized elements, and can efficiently extract and fuse feature information. The present invention can be universally deployed in most new view generation methods based on neural radiation fields, and can greatly improve the quality of synthesized images without increasing too much memory and computational overhead.

[0135] Example:

[0136] Step 1, obtain training data;

[0137] Step 2, generating light and sampling points according to the training data;

[0138] Step 3, constructing a neural radiation field hybrid model;

[0139] Step 4, inputting the coordinates of the sampling point and the direction of the light into the neural radiation field hybrid model constructed in step 3, and calculating the volume density and color of each sampling point;

[0140] Step 5, calculate the RGB color value of the pixel corresponding to each light ray;

[0141] Step 6, training the neural radiation field mixed model constructed in step 3 according to the volume density and color of each sampling point obtained in step 4 and step 5, and the RGB color value corresponding to each light ray, to obtain a trained neural radiation field mixed model;

[0142] Step 7: Use the trained neural radiation field hybrid model to render a two-dimensional image from the new perspective according to the camera pose of the new perspective, and complete the universal and controllable radiation field enhancement based on spatial perception.

[0143] In step 1, we take the classroom scene in the open source dataset DONeRF as an example. The dataset contains two-dimensional images of classrooms with fine texture features at different viewing angles, as well as the camera pose corresponding to each two-dimensional image. The length and width of the two-dimensional image of the classroom scene are both 800 pixels.

[0144] In step 2, light rays and sampling points are generated, the starting coordinates and direction vector of the light rays are calculated according to the camera pose, and sampling is performed on the light rays to obtain the coordinates of the sampling points, including:

[0145] Step 2-1: Randomly select image blocks on each two-dimensional image, and calculate the starting coordinates and direction vector of the light corresponding to the pixel in the world coordinate system through the pixels of the image block; in the dataset of the classroom scene, each two-dimensional image is 800 pixels long and wide, and randomly select image blocks on each image; in the experiment, the length and width of the image block are 40 pixels, so the tensor C corresponding to the RGB color of the image block pixel gt ∈R 3×40×40 , the coordinate tensor O∈R of the starting point of the light corresponding to the pixel of the image block 3×40×40 , the direction vector tensor Dir∈R 3×40×40 .

[0146] Step 2-2, use the uniform sampling method to sample the light to generate the initial sampling point, and calculate the coordinates of the initial sampling point in the world coordinate system;

[0147] The uniform sampling method described in step 2-2 is as follows:

[0148]

[0149] Among them, t n is the shortest distance between the sampling point and the camera, t f is the farthest distance between the sampling point and the camera, t i is the distance from the i-th sampling point to the camera obtained by uniform sampling method, N initis the number of sampling points, U represents uniform distribution, and ~ represents a probability distribution. In the dataset of the classroom scene, t n is 0.64, t f 8.80, N init is 64.

[0150] Step 3: Construct a neural radiation field hybrid model.

[0151] Step 4, calculating the volume density and color of each sampling point, includes the following steps:

[0152] Step 4-1, encode the coordinates and direction vectors of the 64 initial sampling points generated in step 2-2 to obtain coordinate features and direction vector features. The specific encoding method is as follows:

[0153] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0154] Among them, p is the coordinate of the sampling point or the direction vector of the light where it is located, L is the encoding frequency, and γ(p) represents the encoding feature after encoding. In the experiment, the encoding frequency of the coordinate is 10, and the encoding frequency of the direction vector is 4. After the coordinate is encoded, the coordinate feature tensor X is obtained init ∈R 64 × 63 × 40 × 40 , after encoding the direction vector, we get the direction vector feature tensor V init ∈R 64 × 27 × 40 × 40 ;

[0155] Step 4-2: The coordinate feature tensor X of the initial sampling point in step 4-1 is init and the direction vector feature tensor V init Input the coarse model in step 3 to obtain the volume density σ at the initial sampling point in the coarse model coarse ∈R 64×1×40×40 and color c coarse ∈R 64×3×40×40 ;

[0156] Step 4-3, using the hierarchical sampling method, based on the volume density σ of the sampling points in the coarse model coarse , re-sample to obtain hierarchical sampling points, and merge them with the initial sampling points to form the final fine sampling points;

[0157] Step 4-4: Encode the coordinates and direction vectors of the fine sampling points using the encoding method in step 4-1 and input them into the fine model to obtain the volume density σ of the fine sampling points fine ∈R 192×1×40×40 、Color c fine ∈R 192×3×40×40 , the volume density after mixing across rays and sampling points σ inter_mixed R 192×1×40×40 , the color c after mixing across rays and sampling points inter_mixed ∈R 192 ×3×40×40 , the volume density after mixing both cross-rays and same-rays σ intra_inter_mixed R 192×1×40×40 The color c after mixing both the cross-ray and the same-ray intra_inter_mixed ∈R 192×3×40×40 , where the number of fine sampling points is 192;

[0158] Step 5, calculating the RGB color value corresponding to each light ray, the specific method includes:

[0159]

[0160] in, Represents the final RGB color value of the pixel corresponding to the light, σ i and c i is the volume density and color of the i-th sampling point, δ j is the distance between adjacent sampling points on the same ray, T i is the cumulative penetration rate from the camera position to the i-th sampling point, and N is the number of sampling points on the ray;

[0161] Step 6: training the neural radiation field hybrid model constructed in step 3, that is, updating the parameters of the neural radiation field hybrid model using an Adam optimizer by calculating the loss, wherein the method for calculating the loss includes:

[0162] Step 6-1, for the volume density σ obtained from the coarse model in step 4-3-2 coarse and color c coarse , calculate the pixel RGB value obtained by rendering the radiation field in the coarse model using the method in step 5

[0163] Step 6-2, for the volume density σ obtained from the fine model in step 4-3-3 fine and the color c obtained by the fine model in step 4-3-4 fine , calculate the pixel RGB value obtained by rendering the radiation field in the fine model using the method in step 5

[0164] Step 6-3, for the volume density σ mixed by the cross-ray-cross-sampling point mixing module in the fine model in step 4-3-5 inter_mixed and color c inter_mixed , the pixel RGB value obtained by mixing the radiation field in the fine model with the cross-ray cross-sampling point mixing module after rendering is calculated by the method in step 5

[0165] Step 6-4, for the volume density σ mixed by the same light cross sampling point mixing module in step 4-3-6 intra_nter_mixed and color c intra_inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the same light cross-sampling point mixing module in the fine model using the method in step 5

[0166] Step 6-5, calculate the loss as follows:

[0167] L RF =L coarse +L fine

[0168]

[0169] Among them, L RF is the total loss of the neural radiation field hybrid model, R is the set of rays, and L coarse and L fine are the losses of the coarse model and the fine model respectively. In the experiment, C gt ∈R 3 × 40 × 40 is the true RGB value of the pixel corresponding to the pixel in the image block;

[0170] The present invention proposes a universal and controllable radiation field enhancement method based on spatial perception, which solves the limitation of the basic assumption of independence between sampling points in the neural radiation field-based method, which leads to the lack of fine texture in the synthesized image. The hybrid strategy proposed in the present invention significantly improves the quality of the synthesized image with only an additional 25% of rendering time. At the same time, the present invention proposes a universal radiation field enhancement module, which can be integrated into a variety of neural radiation field-based methods to further improve the quality of the synthesized image.

[0171] Table 1 Experimental results of classroom scene dataset

[0172]

[0173] Among them, the evaluation indicators of the quality of the synthetic image include the peak signal-to-noise ratio, the structural similarity index and the perceived image block similarity. The larger the ↑ value, the higher the quality of the image, and the smaller the ↓ value, the higher the quality of the image. It can be seen from the data in the table that the present invention is superior to the original neural radiation field method in all three indicators. In addition, the time for rendering a picture by the method proposed by Fang Ming is increased from 28.235 seconds of the original neural radiation field method to 35.088 seconds, which can significantly improve the image quality with only an additional 24.24% of the time.

[0174] The present invention provides a general controllable radiation field enhancement method based on spatial perception. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A universal and controllable radiation field enhancement method based on space perception, characterized in that: The steps include: Step 1, obtain training data; Step 2, generating light and sampling points according to the training data; Step 3, constructing a neural radiation field hybrid model; Step 4, inputting the coordinates of the sampling point and the direction of the light into the neural radiation field hybrid model constructed in step 3, and calculating the volume density and color of each sampling point; Step 5, calculate the RGB color value of the pixel corresponding to each light ray; Step 6, training the neural radiation field mixed model constructed in step 3 according to the volume density and color of each sampling point obtained in step 4 and step 5, and the RGB color value corresponding to each light ray, to obtain a trained neural radiation field mixed model; Step 7: Use the trained neural radiation field hybrid model to render a two-dimensional image from the new perspective according to the camera pose of the new perspective, and complete the universal and controllable radiation field enhancement based on spatial perception.

2. A universal and controllable radiation field enhancement method based on space perception according to claim 1, characterized in that: The step 1 of obtaining training data is to use a camera to obtain two-dimensional images at different viewing angles, and record the camera pose corresponding to each image, and use the two-dimensional images and the corresponding camera pose as training data.

3. The universal and controllable radiation field enhancement method based on space perception according to claim 2, characterized in that: The step 2 of generating light and sampling points is to use the neural radiation field to model the light. It is assumed that each pixel in the two-dimensional image emits a light ray, the starting point of the light ray is the camera position, the direction is the direction vector pointing from the camera position to the pixel position in the two-dimensional image plane, and each pixel corresponds to a light ray; the starting point coordinates and direction vector of the light ray are calculated according to the camera posture, and the coordinates of the sampling point are obtained by sampling on the light ray, which specifically includes: Step 2-1, based on the camera pose and the pixel coordinates in the two-dimensional image plane, calculate and generate the starting point coordinates and direction vector of the light ray corresponding to the pixel in the world coordinate system, the specific method includes: Randomly select an image block on each two-dimensional image, and calculate the starting coordinates and direction vector of the light corresponding to the pixel in the world coordinate system through the pixels of the image block; let the length and width of the image block be p h and p w , then the tensor corresponding to the RGB color of the image block pixel The coordinate tensor of the starting point of the light corresponding to the pixel of the image block Direction vector tensor Step 2-2, use the uniform sampling method to sample the light to generate the initial sampling point, and calculate the coordinates of the initial sampling point in the world coordinate system; The uniform sampling method described in step 2-2 is as follows: Among them, t n is the shortest distance between the sampling point and the camera, t f is the farthest distance between the sampling point and the camera, t i is the distance from the i-th sampling point to the camera obtained by uniform sampling method, N init is the number of initial sampling points, U represents uniform distribution, and ~ represents obedience to a certain probability distribution.

4. The universal and controllable radiation field enhancement method based on space perception according to claim 3 is characterized in that: The construction of the neural radiation field hybrid model described in step 3 includes the following steps: Step 3-1, construct two granularity models under the neural radiation field hybrid model, namely a coarse model and a fine model; wherein the coarse model includes: a sampling point coordinate feature encoder, a direction vector feature encoder, a volume density decoder and a color decoder; the output of the coarse model is the volume density σ of the sampling point coarse and color c coarse ; The fine model includes: a sampling point coordinate feature encoder, a direction vector feature encoder, a volume density decoder, a color decoder and a mixing module; wherein the mixing module includes: a cross-ray cross-sampling point mixing module and a same-ray cross-sampling point mixing module; the output of the fine model is the volume density feature F of the sampling point σ 、Color feature F c , volume density σ fine 、Color c fine , the volume density after mixing across rays and sampling points σ inter_mixed , the color c after mixing across rays and sampling points inter_mixed , the volume density after mixing cross-rays and same-rays σ intra_inter_mixed and the color c after mixing the cross-light and the same-light intra_inter_mixed ; Step 3-2, constructing a cross-ray and cross-sampling point mixing module in the radiation field mixing model; Step 3-3, construct a same-ray cross-sampling point mixing module in the radiation field mixing model.

5. A universal and controllable radiation field enhancement method based on space perception according to claim 4, characterized in that: The sampling point coordinate feature encoder described in step 3-1 is composed of eight layers of two-dimensional convolutional layers with a convolution kernel of 1*1, and encodes the input sampling point coordinate features to obtain volume density features; The volume density decoder is composed of a two-dimensional convolution layer with a convolution kernel of 1*1, which decodes the volume density feature to obtain the volume density of each sampling point; The directional vector feature encoder is composed of a two-dimensional convolution layer with a convolution kernel of 1*1. After concatenating the volume density feature with the input directional vector feature, the directional vector feature encoder is used to encode the feature again to obtain the color feature. The color decoder is composed of a two-dimensional convolution layer with a convolution kernel of 1*1, which decodes the color features to obtain the color of each sampling point.

6. A universal and controllable radiation field enhancement method based on space perception according to claim 5, characterized in that: The cross-ray and cross-sampling point mixing module described in step 3-2 is composed of a volume density mixing sub-network and a color mixing sub-network, wherein each sub-network is composed of two layers of two-dimensional convolutional layers with a convolution kernel of k*k; The cross-ray and cross-sampling point mixing module is used to predict the volume density weight and color weight of each sampling point at the same depth as the adjacent sampling points on different rays, and obtain the volume density σ of each sampling point after cross-ray and cross-sampling point mixing. inter_mixed and the color c after mixing across rays and sampling points inter_mixed , the specific prediction method is as follows: Step 3-2-1: The volume density feature F obtained by the sampling point in the fine model σ Input the volume density hybrid subnetwork to predict the volume density weight W_inter of the adjacent sampling points at the same depth as the sampling point on different rays to the sampling point. σ , for the volume density σ fine Perform weighted summation to obtain the volume density σ after mixing across rays and sampling points inter_mixed , the specific method is as follows: Where k is the number of adjacent sampling points, is the volume density weight of the i-th adjacent sampling point to the sampling point, is the volume density of the i-th adjacent sampling point; Step 3-2-2: The color feature F obtained by the sampling point in the fine model c Input into the color mixing subnetwork, and predict the color weight W_inter of the adjacent sampling points at the same depth as the sampling point on different rays to the sampling point. c , for the color c fine Perform weighted summation to obtain the color c after mixing across light and sampling points inter_mixed , the specific method is as follows: Where k is the number of adjacent sampling points, is the body color weight of the i-th adjacent sampling point to the sampling point, is the color of the i-th adjacent sampling point.

7. A universal and controllable radiation field enhancement method based on space perception according to claim 6, characterized in that: The same-ray cross-sampling point mixing module described in step 3-3 is composed of a volume density mixing sub-network and a color mixing sub-network, where each sub-network is composed of two layers of one-dimensional convolution neural networks with a convolution kernel of k; The same-ray cross-sampling point mixing module is used to predict the volume density weight and color weight of each sampling point on the same ray to obtain the volume density σ of each sampling point after mixing across the ray and the same ray. intra_inter_mixed and the color c after mixing the cross-light and the same-light intra_inter_mixed , the specific prediction method is as follows: Step 3-3-1: The volume density feature F obtained by the sampling point in the fine model σ Input the volume density hybrid subnetwork to predict the volume density weight W_intra of the adjacent sampling point on the same ray to the sampling point σ , for step 3-2, the volume density σ after blending across rays and sampling points inter_mixed Perform weighted summation to obtain the volume density σ after mixing both the cross-ray and the same-ray intra_inter_mixed , the specific method is as follows: Where k is the number of adjacent sampling points, is the volume density weight of the i-th adjacent sampling point to the sampling point, is the volume density of the i-th adjacent sampling point after mixing across rays and sampling points; Step 3-3-2: The color feature F obtained by the sampling point in the fine model c Input into the color mixing subnetwork, and predict the color weight W_intra of the adjacent sampling point on the same ray to the sampling point c , for the color c after mixing across the light and across the sampling points in step 3-2 inter_mixed Perform weighted summation to obtain the color c after mixing the cross-light and the same-light intra_inter_mixed , the specific method is as follows: Where k is the number of adjacent sampling points, is the body color weight of the i-th adjacent sampling point to the sampling point, It is the color of the i-th adjacent sampling point after being mixed across the light and across the sampling point.

8. The universal and controllable radiation field enhancement method based on space perception according to claim 7, characterized in that: The calculation of the volume density and color of each sampling point described in step 4 includes the following steps: Step 4-1, encoding the coordinates and direction vectors of the sampling points to obtain coordinate features and direction vector features. The specific encoding method is as follows: γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) Among them, p is the coordinate of the sampling point or the direction vector of the light where it is located, L is the encoding frequency, and γ(p) represents the encoded feature after encoding, where the coordinate feature is obtained after the coordinate encoding, and the direction vector feature is obtained after the direction vector encoding; the coordinate feature tensor is Where D x =3×(2×L x +1); the direction vector feature tensor is Where D v =3×(2×L v +1), N is the number of sampling points on each ray; Step 4-2: The coordinate feature tensor of the initial sampling point described in step 2-2 With the direction vector feature tensor Input the coarse model in step 3 to obtain the volume density of the initial sampling point in the coarse model and color N init is the number of initial sampling points; Step 4-3, using the hierarchical sampling method, based on the volume density σ of the sampling points in the coarse model coarse , and sample again to obtain fine sampling points, as follows: Step 4-3-1, calculate the volume density σ of the sampling points on each ray in the coarse model coarse Normalize it and use it as the probability p that the sampling point is on the surface of the object in the current scene i , the specific method is as follows: Among them, N init is the number of initial sampling points on each ray; Step 4-3-2: According to the probability p of each sampling point on each ray being on the surface of the object in the current scene, i Distribution, generate hierarchical sampling points through inverse transform sampling; Step 4-3-3, merging the initial sampling point in step 2-2 and the hierarchical sampling point in step 4-3-2 into the final fine sampling point; Step 4-4, encode the coordinates and direction vectors of the fine sampling points using the encoding method in step 4-1 to obtain the coordinate feature tensor and the direction vector feature tensor And input the fine model constructed in step 3 to obtain the volume density characteristics of the fine sampling points Color Features Body density color Volume density after blending across rays and sampling points The color after blending across rays and sampling points Volume density after mixing across rays and within rays The color after mixing both the cross-ray and the same-ray N fine is the number of fine sampling points, the specific steps are as follows: Step 4-4-1, obtain the volume density feature F in the fine model through the coordinate feature encoder in the fine model σ ; Step 4-4-2, obtain the color feature F in the fine model through the direction vector feature encoder in the fine model C ; Step 4-4-3, obtain the volume density σ in the fine model through the volume density decoder in the fine model fine ; Step 4-4-4, obtain the color c in the fine model through the color decoder in the fine model fine ; Step 4-4-5, transform the above volume density feature F σ 、Color feature F c , volume density σ fine and color c fine , input the cross-ray and cross-sampling point mixing module to obtain the volume density σ after cross-ray and cross-sampling point mixing inter_mixed and color C inter_mixed ; Step 4-4-6, the above volume density feature F σ 、Color feature F c , the volume density after mixing across rays and sampling points σ inter_mixed and the color c after mixing across rays and sampling points inter_mixed , input the same light cross sampling point mixing module to obtain the volume density σ after mixing the cross light and the same light intra_inter_mixed With color c intra_inter_mixed .

9. The universal and controllable radiation field enhancement method based on space perception according to claim 8, characterized in that: The specific method for calculating the RGB color value of the pixel corresponding to each light ray described in step 5 includes: in, Represents the final RGB color value of the pixel corresponding to the light, σ i and c i is the volume density and color of the i-th sampling point, δ j =t j+1 -t j is the distance between adjacent sampling points on the same ray, T i is the cumulative penetration rate from the camera position to the i-th sampling point.

10. A universal and controllable radiation field enhancement method based on space perception according to claim 9, characterized in that: The training of the neural radiation field hybrid model constructed in step 3 described in step 6, that is, updating the parameters of the neural radiation field hybrid model using the Adam optimizer by calculating the loss, wherein the method of calculating the loss includes: Step 6-1, for the volume density σ obtained from the coarse model in step 4-2 coarse and color c coarse , calculate the pixel RGB value obtained by rendering the radiation field in the coarse model using the method in step 5 Step 6-2, for the volume density σ obtained from the fine model in step 4-2 fine and color c fine , calculate the pixel RGB value obtained by rendering the radiation field in the fine model using the method in step 5 Step 6-3, for the volume density σ mixed by the cross-ray and cross-sampling point mixing module in the fine model in step 4-4 inter_mixed and color c inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the cross-ray and cross-sampling point mixing module in the fine model using the method in step 5 Step 6-4, for the volume density σ mixed by the same light cross sampling point mixing module in step 4-5 intra_nter_mixed and color cintr a_inter_mixed , calculate the pixel RGB value obtained by mixing and rendering through the same light cross-sampling point mixing module in the fine model using the method in step 5 Step 6-5, calculate the loss as follows: L RF =L coarse +L fine Among them, L RF is the total loss of the neural radiation field hybrid model, R is the set of rays, and L coarse and L fine are the losses of the coarse model and the fine model respectively, It is the true RGB value of the pixel corresponding to the image block pixel.

Images