Neural radiation field optimization method and device based on energy loss uncertainty voting

Abstract

The present invention provides a neural radiation field optimization method and device based on energy loss uncertainty voting, which includes obtaining the data set required for training and testing the neural radiation field; changing the NLL function used in ActiveNeRF training to an energy loss function and using a more reasonable regularization term; using energy loss to train the model; building a multi-head attention network model; obtaining a data set for training the multi-head attention network model; training the multi-head attention network model; using the trained attention network model to replace the color values ​​of the sampling points with large noise in the test pose to improve the final rendering effect. The present invention uses the Bayesian method to model data uncertainty and uses energy loss for training, which alleviates the problem that negative log-likelihood loss will produce excessive variance regardless of whether the predicted mean is correct or not, thereby damaging model training. On this basis, the present invention further integrates data uncertainty into the multi-head attention network module to improve the final rendering quality.

Description

Technical Field

[0001] The present invention relates to the technical field of implicit three-dimensional reconstruction of computer vision, and in particular to a neural radiation field optimization method, device and storage medium based on energy loss uncertainty voting. Background Art

[0002] In recent years, Neural Radiance Fields (Nerf) has attracted significant attention in the fields of 3D computer vision and computer graphics due to its ability to render photorealistic images. However, NeRF is unaware of the noise level of the data. When the input camera pose contains errors and noise due to inaccurate estimation algorithms, the network struggles to accurately output rendered colors. To address this issue and improve rendering quality, Pan et al. (Pan X, Lai Z, Song S, et al. Activenerf: Learning where to see with uncertainty estimation[C] / / European Conference on Computer Vision. 2022: 230-246.) modeled the light color using a heteroscedastic Gaussian distribution based on the Bayesian criterion. The mean represents the predicted color value, and the variance represents the noise level of the input data. Both are output by the network and optimized using the negative log-likelihood loss (NLL) for training. This approach enables the network to adaptively weight noisy data for training, thereby reducing the impact of noise on the network model and improving rendering quality. While effective, NLL scoring isn't always applicable. Harakeh et al. (Harakeh A, Waslander S L. Estimating and evaluating regression predictive uncertainty in deepobject detectors[J]. arXiv preprint arXiv:2101.05036, 2021.) show that using NLL for training can lead to excessive variance in the network output, regardless of whether the predicted mean is correct, thus impairing network training. This issue also exists in NeRF, where using NLL for training can lead to poor rendering quality.

[0003] The attention mechanism can be traced back to the paper (Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017). It was first applied to natural language processing and later expanded to other fields with remarkable results. The attention mechanism allows the model to dynamically adjust the degree of attention it pays to different parts of the input. By calculating attention weights and performing a weighted summation of the elements of the input sequence, an attention-based output representation is generated. This output representation helps the model better capture important task-relevant information in the input data. Attention mechanisms can be categorized based on different criteria: self-attention, multi-head attention, dot-product attention, and summed attention. Summary of the Invention

[0004] The present invention proposes a neural radiation field optimization method and device based on energy loss uncertainty voting, which specifically involves deep neural network uncertainty modeling, attention mechanism and neural radiation field fields, and can solve at least one of the technical problems in the background technology.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A neural radiation field optimization method based on energy loss uncertainty voting includes the following steps:

[0007] Step 1: Obtain image datasets of different perspectives in a three-dimensional scene and the corresponding camera poses, and divide the datasets into training datasets and test datasets.

[0008] Step 2: Based on ActiveNeRF, the NLL function used in training is changed to an energy loss function.

[0009] Step 3: Build the GPU environment required for training, load the parameter configuration file and the model used in ActiveNeRF, and use the energy loss function to train on the training set.

[0010] Step 4: After training, an uncertainty-aware NeRF network model can be obtained. For any point in space, the network model can output the corresponding color value and data uncertainty.

[0011] Step 5: Train a multi-head attention network model. Its output is used to replace the color values ​​of sampling points with large data noise in the space to improve rendering quality. This mainly includes the following steps:

[0012] Step 5.1, prepare the dataset for training the attention network.

[0013] Step 5.2: Build a multi-head attention network model.

[0014] Step 5.3: Construct the loss function for training.

[0015] In step 6, the attention network model trained in step 5 is used to replace the color values ​​of the sampling points with large noise in the test pose to improve the final rendering effect.

[0016] Furthermore, in step 2, ActiveNeRF converts the light A sampling point on The color value of is modeled as a Gaussian distribution, that is ,in Represents the predicted color value, variance Indicates the corresponding data uncertainty. Therefore, it is necessary to add an output head to the MLP network model used by the original NeRF to output the variance value. Therefore, at a given point in space The network model finally outputs a five-dimensional vector , respectively represent the predicted color value, volume density and uncertainty. According to the volume density formula:

[0017]

[0018]

[0019] in Indicates the number of sampling points, Indicates sampling point With sampling point The Euclidean distance between them. Color Also obeys the Gaussian distribution, that is:

[0020] (2)

[0021] Then the given light True color , ActiveNeRF uses NLL for optimization training, and the loss is as follows:

[0022]

[0023] in Represents light gather, Representing a collection The number of elements in is a regularization term used to constrain the volume density to prevent artifacts from being too large. is a hyperparameter, The number of sampling points on the light. Since the function of the coarse network is to obtain the weight To perform better fine sampling, ActiveNeRF only performs Gaussian distribution modeling on the color values ​​output by the fine network model, keeping the function of the coarse network unchanged. Color value synthesized with fine network , the loss finally used by ActiveNeRF is as follows:

[0024]

[0025] (4)

[0026] As mentioned above, using NLL for training in the context of NeRF may result in a large variance in the network output. Too large, which makes the predicted value of light color Deviation from the true color value , which damages the training of the model. To this end, the method of the present invention proposes to use the energy loss function (ES) to replace NLL to train the detailed network, as follows: For the distribution of light color in formula (2) Independently sample M points, recorded as , then the true color of the given light , the energy loss can be expressed as:

[0027]

[0028] In addition, the method of the present invention also improves the regularization term in ActiveNeRF. In ActiveNeRF, if the hyperparameter The value is too large, the regular term This encourages the network to output smaller volume densities. To reduce the loss, the rendered color will be distorted and lighter. In order to compensate for this shortcoming, the present invention firstly adjusts the light output by the network. on The volume density value of each sampling point is scaled to (0,1], that is:

[0029] (6)

[0030] in The operation is only used for scaling and does not participate in training, so it does not carry gradients. Then use As the volume density The regularization term encourages Less than 0.5 tends to 0, Greater than 0.5 tends to 1, so in the end The energy loss can be expressed as:

[0031]

[0032] Given a set of rays , the final loss can be expressed as the MSE loss corresponding to the coarse network plus the energy loss of the fine network, that is:

[0033]

[0034]

[0035] Furthermore, for step 5.1, it is necessary to prepare a training set for training the multi-head attention network. Assume that the training set ,in , , then for any set of data , from which we can obtain the world coordinate system The starting position and direction information of the ray. Randomly extract rays, then for a ray Fine sampling possible points, input these points into the fine network trained with energy loss in step 4, and the corresponding color values ​​and data uncertainty can be obtained. These points are considered to be data with less noise, and the color values ​​output by the network are considered to be relatively accurate. Therefore, the color values ​​of these points are used as For the sake of convenience, consider this The first of the points points, and the corresponding network output color value is taken as the true value , is a 3-dimensional vector. Its position information And perspective information Combination of as query conditions ,Right now , is a 6-dimensional vector. Then for point In other The collection of position information and viewing angle information under the camera pose, that is, . for point In other The color value set under the camera pose can be obtained by The position information and viewing angle information at other camera poses are input into the trained detailed network to obtain In addition, the network will also output the corresponding data uncertainty set .

[0036] Furthermore, for step 5.2, it is necessary to build a multi-head attention network model to replace the color values ​​of sampling points with large data uncertainty on the light. Consider the color values ​​obtained in step 5.1. How does a certain point in the multi-head attention network model work? The true value corresponding to this point , query ,key ,value , uncertainty The goal is to 、 、 、 Get network output and the true value Assume that the attention model has output head, then for the Output header, which outputs the result It can be expressed as:

[0037]

[0038]

[0039]

[0040] Here we use the additive attention method to obtain the attention score, where and It is The two fully connected layers in the output head are responsible for converting the 6-dimensional query And 6-dimensional keys Map to In dimensional space, Corresponding to a fully connected layer, used to dimensional space is mapped to 1-dimensional space, is the activation function.

[0041] After scaling, the range becomes , To reduce the uncertainty of the data Therefore, the final output result is Is this The results of the output heads are accumulated, that is:

[0042]

[0043] Furthermore, in step 5.3, it is necessary to construct the loss function required for training the multi-head attention network model and perform training. In the data preparation phase of step 5.1, there are images, each of which is randomly selected rays, each of which takes If the data noise is small, then according to step 5.2, the final loss function can be obtained:

[0044]

[0045] Furthermore, in step 6, the attention network model trained in step 5 is used to replace the color values ​​of the sampling points with large noise under the test pose to improve the final rendering effect. Specifically, for the uncertainty perception NeRF network model trained with energy loss in step 2, given the test pose , you can get the predicted image results And the corresponding data uncertainty image In order to improve the quality of the predicted image, we first need to select middle For pixels with larger median values, get the corresponding light for one of the pixels Then from the light Select the sampling points with larger uncertainty from the fine sampling points, and combine the position information and viewing angle information corresponding to these sampling points into , the position information and viewing angle information of the sampling point in other test poses , color information composition , uncertainty information composition . Will eventually 、 、 、 Input into the multi-head attention network trained in step 5.3 to obtain the output And use the result to replace the light The color of the sampling point with large data uncertainty is calculated, and then the new light color is obtained through the volume rendering formula to replace the pixel color.

[0046] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0047] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0048] As can be seen from the above technical solution, the present invention's neural radiation field optimization method and device based on energy loss uncertainty voting solves the problem that NLL impairs network model training and reduces rendering quality in the uncertainty-aware NeRF context. Furthermore, by leveraging NeRF's uncertainty perception capabilities, a multi-head attention mechanism is used to further improve rendering quality.

[0049] The present invention relates to the field of uncertainty and implicit three-dimensional reconstruction, and discloses a neural radiation field optimization method based on energy loss uncertainty voting, the method comprising: obtaining an image dataset of different perspectives and corresponding camera poses in a certain three-dimensional scene, and dividing the dataset into a training dataset and a test dataset; changing the NLL function used in training to an energy loss function and using a more reasonable regularization term based on ActiveNeRF; using the proposed energy loss training model; building a multi-head attention network model; obtaining a dataset for training the multi-head attention network model; training the multi-head attention network model; using the trained attention network model to replace the color values ​​of sampling points with large noise in the test pose, thereby improving the final rendering effect. The present invention models data uncertainty through a Bayesian method and uses energy loss for training, effectively alleviating the problem that negative log-likelihood loss will produce excessive variance regardless of whether the predicted mean is correct or not, thereby damaging model training. At the same time, the present invention further utilizes data uncertainty, incorporates uncertainty into the attention mechanism, and further improves rendering quality.

[0050] Specifically, the neural radiation field optimization method based on energy loss uncertainty voting proposed in the present invention has the following advantages:

[0051] (1) Energy loss is used instead of negative log-likelihood loss to train uncertainty-aware neural radiation field networks, thereby avoiding the problem that negative log-likelihood loss outputs excessive variance regardless of the correctness of the predicted mean, which damages the training of the model.

[0052] (2) A new regular expression for volume density is constructed to replace the regular expression in ActiveNeRF. This regular expression does not require the specification of hyperparameters, thus avoiding the problem of improper hyperparameter selection in ActiveNeRF, which causes excessive penalty on volume density and leads to distorted rendering results.

[0053] (3) The attention mechanism is integrated into the neural radiance field to replace the color of sampling points with large data uncertainty in the neural radiance field, thereby improving the final rendering quality. In addition, the attention mechanism also considers the data uncertainty of the candidate value and incorporates it into the final weighted result, so that the candidate value with large uncertainty should contribute less to the final output result. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall flow chart of the method of the present invention, including a training flow chart and a testing flow chart;

[0055] Figure 2 This is a diagram of the network model structure of the uncertainty-aware NeRF of the present invention;

[0056] Figure 3 This is a structural diagram of the multi-head attention network model proposed by the method of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0058] like Figure 1 As shown, the neural radiation field optimization method based on energy loss uncertainty voting described in this embodiment is Figure 1 The training and testing processes are demonstrated. The training process explains how to use energy loss to train the uncertainty-aware NeRF network and how to train a multi-head attention network model to improve rendering quality. The specific steps include:

[0059] Step 1: Obtain a dataset for training uncertainty-aware NeRF. First, take a picture of a scene from different viewing angles. Then, the COLMAP technique was used to obtain the image. The camera pose of the image, the final dataset consists of images and the corresponding camera poses, i.e. .in , and Represents the height and width of the image, and 3 represents the number of channels of the image. Represents the camera pose, including a rotation matrix and a translation vector , and Used to convert the camera coordinate system to the world coordinate system. In addition, it also includes the height and width of the image and the focal length of the camera , used to convert the image coordinate system to the camera coordinate system. Finally, it also includes the near plane and the far plane , represents the imaging boundary, which is used to limit the range of sampling points. , evenly extracted from as the training set, i.e. , the rest data as the test set, that is, .

[0060] Step 2: Load the uncertainty-aware NeRF network model, such as Figure 2 As shown. The input of the network is the position information of a sampling point on the light The result after encoding And perspective information The result after encoding , the output of the network is the color information of the point , volume density and the corresponding data uncertainty The main part of the network consists of 8 fully connected layers, each of which contains 256 neurons and uses Function is activated. In addition, in order to retain the input information, the input information is activated at the 5th fully connected layer. It is spliced ​​with hidden features. and data uncertainty Through the output of the 8th hidden layer, in order to ensure the non-negativity of both, the volume density Used Activation, data uncertainty usage Activate. Color value It is related to the viewing angle, so the viewing angle information is spliced ​​in the 9th hidden layer , and finally output the color value through two connection layers, and use The activation function ensures the color range. The encoding formula used above is:

[0061]

[0062] When the location information When encoding, , for viewing angle information When encoding, .

[0063] Step 3: Use energy loss to train the network model in step 2. An image in , you can get A ray of light then passes through The transformation matrix in the world coordinate system is converted into light. For the sake of convenience, consider one of the rays ,in is the coordinate of the starting point of the light, is the direction of light, is the coefficient, and the true color corresponding to the light is recorded as First, the light exist and 64 points are uniformly sampled within the range, that is, for , bring it into In the expression, 64 sampling points are obtained. Then these sampling points are sent to the coarse network in NeRF to obtain the color values ​​corresponding to these sampling points. and body density , input it into the volume density formula in formula (1) to obtain the rough color value of the image And the weight information corresponding to each sampling point . Next, we can use the weight information again and 128 points are sampled within the range, and then the information of these 128 fine sampling points and the information of the previous 64 coarse sampling points are spliced ​​into 192 sampling points , input the information of these 192 sampling points into Figure 2 In the network model in , the corresponding color value can be obtained , volume density value and the corresponding uncertainty Since the color of the sampling point Modeling is Gaussian distribution, that is , so according to the volume density formula, we know that the light Color .in , , For the Sampling points and The Euclidean distance between the sampling points. In order to use the energy loss, the light The distribution it obeys Independent sampling samples . Ultimately, , , as well as Input into the energy loss formula (8) to obtain the light loss During the specific training, one image is randomly selected each time and Randomly select 1024 rays from the rays to obtain their average loss , then by The optimizer performs backpropagation for optimization. The specific optimization strategy and learning rate setting are consistent with NeRF.

[0064] Step 4: Construct a dataset for training the attention network. To construct a dataset for training the attention network, you need to prepare the following datasets: ,key ,value , true value , in addition, data uncertainty is introduced Next, we will elaborate on how to obtain these data. Assume that the fine network model trained with energy loss in step 3 is denoted as , Indicates the result of encoding the location information of the sampling point. Represents the result of encoding the perspective information of the sampling point, and the network model outputs a three-dimensional color value , one-dimensional volume density value and one-dimensional uncertainty For the sake of convenience, consider how to obtain a certain ray Corresponding query ,key ,value , true value and uncertainty First, the light exist and 64 points are evenly sampled within the range , Contains position information and viewing angle information, that is, .Will Input into the trained coarse network model, and then you can get the corresponding weight set , according to the weight set, the light can be Sample 128 points again Then and 192 sampling points can be obtained by splicing, that is, , .Will Input to the trained fine network model In the example above, you can get the color value set corresponding to these 192 sampling points. , a set of volume density values and the corresponding uncertainty In order to obtain 、 、 、 as well as The first step is to determine which points have smaller data uncertainty, and set a threshold for this:

[0065]

[0066] in , , It is a custom parameter. . Then the set middle The sampling points are considered to have less data uncertainty, assuming that there are So this The set of position information and viewing angle information corresponding to the sampling points is recorded as ,Right now , the corresponding color value is recorded as ,Right now The second step is to determine 、 、 ,for , which means that this Sampling points in other A set of position information and perspective information under each pose, namely ,in express Among the sampling points The sampling point is The position information of each camera pose, Indicates the corresponding viewing angle information. and , Indicates this Sampling points in other A set of color information under different poses, Indicates this Sampling points in other A set of uncertainty information under different poses. Input to the trained fine network model , then the color value set output by the model constitutes , the output uncertainty set is composed of , so there is ,in express Among the sampling points The sampling point is Color information under different camera poses. ,in express Among the sampling points The sampling point is The uncertainty information of the camera pose. Each image in the image is randomly selected The rays constitute the dataset for training the multi-head attention network, namely 、 、 、 、 .

[0067] Step 5: Build a multi-head attention network model, such as Figure 3 As shown; the network model contains For each attention head, its input is the position information and perspective information of a sampling point , the sampling point is in other Position information and viewing angle information under each camera pose , , the sampling point is in other Color information under each camera pose , and uncertainty information , . Indicates the The fully connected layer in the attention head receives an input dimension of , the output dimension is . Indicates adding multiple results. Is the activation function, used to convert the intermediate results of the attention head Converting to an attention score , and make . Indicates the The output of an attention head. Represents the prediction result of the sampling point color, In the algorithm of the present invention, , , , .

[0068] Step 6: Train the attention network model. The multi-head attention network model obtained in step 5 needs to be trained based on the data set obtained in step 4. The specific steps are as follows: For each sampling point, its corresponding query condition can be found in the data set obtained in step 4. , candidate key , candidate values , true value , uncertainty .Will 、 、 、 Input it into the multi-head attention network model in step 5 to obtain the color prediction value of the sampling point , the loss is the predicted value and the true value In the algorithm of the present invention, batch training is adopted, and the batch size is Set to 4096, so each time from the data set in step 4 The sampling points correspond to 、 、 、 Input into the multi-head attention network model, output prediction values, the final loss is Predicted values and the corresponding True value of military losses, namely:

[0069]

[0070] Next, just need to In the algorithm of the present invention, the optimizer adopts The optimization parameters are the fully connected layer parameters in the multi-head attention. The learning rate is 0.01 and decreases with the number of training times, and finally drops to 0.001. The number of training times is set to 100.

[0071] also, Figure 1 The test process describes how to use the trained uncertainty-aware NeRF network model and the multi-head attention model to improve the quality of the final test results. Given the test dataset divided in step 1 ,according to And the trained uncertainty-aware NeRF model can obtain the corresponding predicted image And the corresponding uncertainty image First, we need to select the pixel colors with greater uncertainty in the predicted image and replace and improve them. Therefore, a threshold is set according to Formula 13:

[0072]

[0073] in , , is the coefficient, which is 0.8 in the algorithm of the present invention. The uncertainty value of the pixel color is greater than All need to be replaced and upgraded. Specific steps are as follows Figure 1For the sake of convenience, let's consider how the color of a pixel with large uncertainty is replaced. This is divided into two steps:

[0074] Step 1): Get the light corresponding to the pixel color The sampling points on the ray and the color, volume density, and uncertainty corresponding to the sampling points. There are 192 sampling points on the graph, including 128 fine sampling points and 64 coarse sampling points. These 192 sampling points are recorded as ,in Indicates the The position information and viewing angle information corresponding to each sampling point, namely: Then input these sampling points into the trained uncertainty-aware NeRF network model to obtain the color set of these sampling points. , volume density collection , uncertainty set .

[0075] Step 2): Use the multi-head attention network to replace the color of the sampling points with large uncertainty in step 1, and then use volume rendering technology to recalculate the light after replacement color and replace the original light The corresponding pixel color. Specifically, we first need to select the light For the uncertainty set obtained in step 1, , first set a threshold:

[0076]

[0077] in , , is a custom coefficient, which is set to 0.8 in the algorithm of the present invention. The uncertainty value is greater than The color value of the sampling point needs to be replaced. Assuming that there are , then this The position information and viewing angle information corresponding to each sampling point constitute the query ,Right now , ;this Sampling points in other The position information and view information under the test pose constitute the candidate key ,in express The sampling points to be replaced The sampling point is Position information and viewing angle information under each test pose. Sampling points in other The color information under the test poses constitutes the candidate value ,in express The sampling points to be replaced The sampling point is Color information under the test pose; this Sampling points in other The uncertainty information under each test pose constitutes ,in express The sampling points to be replaced The sampling point is The uncertainty information under the test pose. Then 、 、 、 Input into the trained multi-head attention network model to get the output result ,in Indicates the color to be replaced. Finally, for light , which has a total of The color and volume density of each sampling point do not need to be changed. The color of the sampling point needs to use the output of the multi-head attention network Replace, the volume density does not need to be replaced. Output the processed 192 sampling points and volume density to the volume rendering formula 1, and you can get the information about light. The new color of the light is used to replace the original light The corresponding pixel color can complete the final enhancement.

[0078] Table 1

[0079]

[0080] Table 2

[0081]

[0082] Tables 1 and 2 show the results of the comparative experiments. Table 1 compares the effects of NLL and ES. Table 1 compares the index results of the original NeRF, NeRF trained with energy loss (ES), and NeRF trained with negative log-likelihood loss (NLL). The bolded ones indicate better results. Blender and LLFF belong to two different types of training data sets. PSNR stands for peak signal-to-noise ratio; the higher the value, the smaller the noise; SSIM stands for structural similarity; the higher the value, the more similar the two images; LPIPS stands for learned perceptual image block similarity. This evaluation indicator better reflects human perception; the smaller the value, the more realistic the image. Table 2 compares the results before and after optimization. Table 2 compares the results before and after optimization using a multi-head attention network.

[0083] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0084] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0085] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the neural radiation field optimization methods based on energy loss uncertainty voting in the above embodiments.

[0086] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above methods.

[0087] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0088] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0089] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A neural radiation field optimization method based on energy loss uncertainty voting, characterized in that: The following steps are involved: Step 1: Obtain image datasets of different perspectives and corresponding camera poses in a three-dimensional scene, and divide the datasets into training datasets and test datasets; Step 2: Based on ActiveNeRF, the NLL function used in training is changed to the energy loss function; Step 3: Build the GPU environment required for training, load the parameter configuration file and the model used in ActiveNeRF, and use the energy loss function to train on the training set; Step 4: After training, an uncertainty-aware NeRF network model is obtained. For any point in space, the NeRF network model can output the corresponding color value and data uncertainty. Step 5: Based on step 4, a multi-head attention network model is trained. The output of the model is used to replace the color values ​​of sampling points with large data noise in the space to improve the rendering quality. Step 6: Use the attention network model trained in step 5 to replace the color values ​​of the sampling points with large noise in the test pose to improve the final rendering effect; Step 2 includes the following: ActiveNeRF turns light A sampling point on The color value of is modeled as a Gaussian distribution, that is ,in Represents the predicted color value, variance Indicates the corresponding data uncertainty; An additional output head is added to the MLP network model used by the original NeRF to output the variance value, so a point in a given space The network model finally outputs a five-dimensional vector , respectively represent the predicted color value, volume density and uncertainty; According to the volume density formula: in Indicates the number of sampling points, Indicates sampling point With sampling point The Euclidean distance between them; we know that the light Color Also obeys the Gaussian distribution, that is: (2) Then the given light True color , ActiveNeRF uses NLL for optimization training, and the loss is as follows: in Represents light gather, Representing a collection The number of elements in is a regularization term used to constrain the volume density to prevent artifacts from being too large. is a hyperparameter, The number of sampling points on the light; Since the function of the coarse network is to obtain weights To perform better fine sampling, ActiveNeRF only performs Gaussian distribution modeling on the color values ​​output by the fine network model, keeping the function of the coarse network unchanged, so the color values ​​synthesized by the coarse network Color value synthesized with fine network , the loss finally used by ActiveNeRF is as follows: (4) ； Use the energy loss function ES to replace NLL to train the detailed network as follows: For the distribution of light color in formula (2) Independently sample M points, recorded as , then the true color of the given light , the energy loss is expressed as: ； In addition, the first ray output to the network is included on The volume density value of each sampling point is scaled to (0,1], that is: (6) in The operation is only used for scaling and does not participate in training, so it does not carry gradients; then use As the volume density The regularization term encourages Less than 0.5 tends to 0, Greater than 0.5 tends to 1, so in the end The energy loss is expressed as: Given a set of rays , the final loss is expressed as the MSE loss corresponding to the coarse network plus the energy loss of the fine network, that is: ； Step 5 includes the following steps, Step 5.1: Prepare the training set for training the multi-head attention network; Step 5.2: Build a multi-head attention network model to replace the color values ​​of sampling points with large data uncertainty on the light; Step 5.3: Construct the loss function required to train the multi-head attention network model and train it.

2. The neural radiation field optimization method based on energy loss uncertainty voting according to claim 1 is characterized in that: In step 5, Step 5.1 includes, Assume that the training set ,in , , then for any set of data , from which the world coordinate system is obtained The starting position information and direction information of the light ray; randomly extract rays, then for a ray Fine sampling possible points, input these points into the fine network trained with energy loss in step 4, and obtain the corresponding color values ​​and data uncertainty; then select These points are considered to be data with less noise, and the color values ​​output by the network are considered to be relatively accurate. Therefore, the color values ​​of these points are used as ; Consider this The first of the points points, and the corresponding network output color value is taken as the true value , is a 3-dimensional vector, and its position information And perspective information Combination of as query conditions ,Right now , is a 6-dimensional vector; then for point In other The collection of position information and viewing angle information under the camera pose, that is, ; for point In other The color value set under the camera pose is obtained by The position information and viewing angle information at other camera poses are input into the trained detailed network to obtain In addition, the network will also output the corresponding data uncertainty set ; Step 5.2 consists of considering the How does a certain point in the multi-head attention network model work? The true value corresponding to this point , query ,key ,value , uncertainty ; the goal is to 、 、 、 Get network output and the true value Use loss for training; Assuming that the attention model has output head, then for the Output header, which outputs the result Expressed as: Here we use the additive attention method to obtain the attention score, where and It is The two fully connected layers in the output head are responsible for converting the 6-dimensional query And 6-dimensional keys Map to In dimensional space, Corresponding to a fully connected layer, used to dimensional space is mapped to 1-dimensional space, is the activation function; After scaling, the range becomes , To reduce the uncertainty of the data The corresponding weight; therefore, the final output result Is this The results of the output heads are accumulated, that is: Step 5.3 includes, during the data preparation phase of step 5.1, images, each of which is randomly selected rays, each of which takes For a point with less data noise, the final loss function is obtained according to step 5.2: 。 3. The neural radiation field optimization method based on energy loss uncertainty voting according to claim 2 is characterized in that: Step 6 includes the following: For the uncertainty-aware NeRF network model trained with energy loss in step 2, given the test pose , get the predicted image result And the corresponding data uncertainty image ; In order to improve the quality of the predicted image, first select the predicted image result The corresponding data uncertainty image For pixels with larger median values, get the corresponding light for one of the pixels , then from the light Select the sampling points with larger uncertainty from the fine sampling points, and combine the position information and viewing angle information corresponding to these sampling points into , the position information and viewing angle information of the sampling point in other test poses , color information composition , uncertainty information composition ; will eventually 、 、 、 Input into the multi-head attention network trained in step 5.3 to obtain the output And use the result to replace the light The color of the sampling point with large data uncertainty is calculated, and then the new light color is obtained through the volume rendering formula to replace the pixel color.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 3.

Images