Neural radiation field optimization method and equipment based on energy loss uncertainty voting

Abstract

The invention discloses a neural radiation field optimization method and equipment based on energy loss uncertainty voting. The method comprises the following steps: acquiring a data set required for training and testing a neural radiation field; an NLL function used by ActiveNeRF training is changed into an energy loss function, and a more reasonable regular term is used; training the model by using the energy loss; building a multi-head attention network model; acquiring a data set for training the multi-head attention network model; training a multi-head attention network model; and the trained attention network model is used to replace the color value of the sampling point with large noise under the test pose, and the final rendering effect is improved. According to the method, data uncertainty is modeled through the Bayesian method, training is carried out through energy loss, and the problem that negative logarithm likelihood loss can generate overlarge variance regardless of whether a predicted mean value is correct or not, so that model training is damaged is solved. On the basis, the data uncertainty is further fused into a multi-head attention network module, and the final rendering quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of implicit three-dimensional reconstruction of computer vision, and in particular to a neural radiation field optimization method, device and storage medium based on energy loss uncertainty voting. Background Art

[0002] In recent years, Neural Radiance Field (Nerf) has attracted great attention in the fields of 3D computer vision and computer graphics because it can render photo-realistic images. However, NeRF cannot perceive the noise level of the data. When the input camera pose generates error noise due to inaccurate estimation algorithms, it will make it difficult for the network to accurately output the rendering color results. In order to solve this problem and improve the rendering quality, Pan et al. (Pan X, Lai Z, Song S, et al. Activerf: Learning where to see with uncertainty estimation[C] / / European Conference on Computer Vision. 2022: 230-246.) modeled the heteroscedastic Gaussian distribution of light color based on the Bayesian criterion, where the mean represents the predicted color value and the variance represents the noise level of the input data. Both are output by the network and optimized using negative log-likelihood loss (NLL). This method enables the network to adaptively weight the noisy data for training, thereby reducing the impact of the noise data on the network model and improving the rendering quality. Although effective, NLL is not a scoring method that can be applied in all scenarios. Harakeh et al. (Harakeh A, Waslander S L. Estimating and evaluating regression predictive uncertainty in deepobject detectors[J]. arXiv preprint arXiv:2101.05036, 2021.) pointed out that using NLL for training may cause the variance of the network output to be too large, regardless of whether the predicted mean is correct or not, thereby damaging the training of the network model. This problem also exists in NeRF, and using NLL for training may lead to poor rendering quality. The attention mechanism can be traced back to the literature (Vaswani A. Attention is all you need[J].Advances in Neural Information Processing Systems, 2017.), which was first applied to the field of natural language processing and later extended to other fields with remarkable results. The attention mechanism allows the model to dynamically adjust the degree of attention to different input parts. By calculating the attention weights, the elements of the input sequence are weighted and summed to obtain an attention-based output representation. This output representation can help the model better capture important task-related information in the input data. In addition, the attention mechanism can be divided into self-attention, multi-head attention, dot product attention, and sum attention according to different standards. Summary of the invention

[0003] The present invention proposes a neural radiation field optimization method and device based on energy loss uncertainty voting, which specifically relates to the fields of deep neural network uncertainty modeling, attention mechanism and neural radiation field, and can solve at least one of the technical problems in the background technology.

[0004] To achieve the above object, the present invention adopts the following technical solutions: A neural radiation field optimization method based on energy loss uncertainty voting includes the following steps: Step 1: Obtain image datasets of different perspectives in a three-dimensional scene and the corresponding camera poses, and divide the datasets into training datasets and test datasets.

[0005] Step 2: Based on ActiveNeRF, the NLL function used in training is changed to an energy loss function. Step 3: Build the GPU environment required for training, load the parameter configuration file and the model used in ActiveNeRF, and use the energy loss function to train on the training set. Step 4: After training, an uncertainty-aware NeRF network model can be obtained. For any point in space, the network model can output the corresponding color value and data uncertainty.

[0006] Step 5: Train a multi-head attention network model, whose output is used to replace the color values ​​of sampling points with large data noise in the space to improve the rendering quality. It mainly includes the following steps: Step 5.1, prepare the dataset for training the attention network.

[0007] Step 5.2, build a multi-head attention network model.

[0008] Step 5.3, construct the loss function for training.

[0009] Step 6: Use the attention network model trained in step 5 to replace the color values ​​of the sampling points with large noise in the test pose to improve the final rendering effect. Furthermore, in step 2, ActiveNeRF converts the light A sampling point on The color value of is modeled as a Gaussian distribution, that is ,in Represents the predicted color value, variance Indicates the corresponding data uncertainty. Therefore, it is necessary to add an output head to the MLP network model used by the original NeRF to output the variance value. The network model finally outputs a five-dimensional vector , respectively represent the predicted color value, volume density and uncertainty. According to the volume density formula:

[0010]

[0011] in represents the number of sampling points, Indicates sampling point With sampling point The Euclidean distance between them. Color It also obeys the Gaussian distribution, that is: (2) Then given the light True Color , ActiveNeRF uses NLL for optimization training, and the loss is as follows:

[0012] in Represents light gather, Representing a collection The number of elements in is a regularization term used to constrain the volume density to prevent artifacts from being too large. is a hyperparameter, is the number of sampling points on the light. Since the function of the coarse network is to obtain weights To perform better fine sampling, ActiveNeRF only performs Gaussian distribution modeling on the color values ​​output by the fine network model, keeping the function of the coarse network unchanged. Color value synthesized with fine network , the loss finally used by ActiveNeRF is as follows:

[0013] (4) As mentioned above, in the context of NeRF, using NLL for training may result in a large variance in the network output. Too large, making the predicted value of light color Deviation from the true color value , which damages the training of the model. To this end, the method of the present invention proposes to use the energy loss function (ES) to replace NLL to train the detailed network, as follows: For the distribution of light color in formula (2): Independently sample M points, denoted as , then the true color of the given light , the energy loss can be expressed as:

[0014] In addition, the method of the present invention also improves the regularization term in ActiveNeRF. In ActiveNeRF, if the hyperparameter The value is too large, the regular term This will encourage the network to output smaller volume density To reduce the loss, the rendered color will be distorted and lighter. In order to compensate for this shortcoming, the present invention firstly adjusts the light output by the network. on The volume density value of each sampling point is scaled to between (0,1], that is: (6) in The operation is only used for scaling and does not participate in training, so it does not carry gradients. Then use As the volume density The regularization term encourages Less than 0.5 tends to 0, Greater than 0.5 tends to 1, so in the end The energy loss can be expressed as:

[0015] Given a set of rays , the final loss can be expressed as the MSE loss corresponding to the coarse network plus the energy loss of the fine network, that is:

[0016]

[0017] Furthermore, for step 5.1, we need to prepare a training set for training the multi-head attention network. Assume that the training set ,in , , then for any set of data , from which we can obtain the world coordinate system The starting position and direction information of the light ray. Randomly extract rays, then for a ray Fine sampling possible points, and input these points into the fine network trained with energy loss in step 4 to obtain the corresponding color value and data uncertainty. These points are considered to be data with less noise, and the color values ​​output by the network are considered to be relatively accurate. Therefore, the color values ​​of these points are used as For the sake of convenience, consider this Point points, and take the color value of the corresponding network output as the true value , is a 3-dimensional vector. And perspective information As a query condition ,Right now , is a 6-dimensional vector. Then For point In other The collection of position information and viewing angle information under the camera pose, that is . For point In other The color value set under the camera pose can be obtained by The position information and viewing angle information under other camera poses are input into the trained detailed network to obtain In addition, the network will also output the corresponding data uncertainty set .

[0018] Furthermore, for step 5.2, a multi-head attention network model needs to be built to substitute the color values ​​of sampling points with large data uncertainty on the light. Consider the color values ​​obtained in step 5.1. How does a certain point in the multi-head attention network model work? The true value corresponding to this point , query ,key ,value , uncertainty The goal is to , , , Get network output With the true value Assume that the attention model has output head, then for the Output header, the output result It can be expressed as:

[0019]

[0020]

[0021] Here we use the additive attention method to obtain the attention score, where and It is The two fully connected layers in the output head are responsible for converting the 6-dimensional query And the 6-dimensional key Map to In dimensional space, Corresponding to a fully connected layer, used to dimensional space is mapped to 1-dimensional space, is the activation function.

[0022] After scaling, the range becomes , This is to reduce the value of high uncertainty in the data Therefore, the final output result is Is this The results of the output heads are accumulated, that is:

[0023] Furthermore, in step 5.3, it is necessary to construct the loss function required for training the multi-head attention network model and perform training. In the data preparation phase of step 5.1, there are images, each of which is randomly selected rays, each of which takes If there are points with less data noise, then according to step 5.2, the final loss function can be obtained:

[0024] Furthermore, in step 6, the attention network model trained in step 5 is used to replace the color values ​​of the sampling points with large noise in the test pose to improve the final rendering effect. Specifically, for the uncertainty-aware NeRF network model trained using energy loss in step 2, given the test pose , you can get the predicted image results And the corresponding data uncertainty image In order to improve the quality of the predicted image, we first need to select middle For pixels with larger median values, get the corresponding light for one of the pixels Then from the light Select sampling points with large uncertainty from the fine sampling points, and combine the position information and viewing angle information corresponding to these sampling points into , the position information and viewing angle information of the sampling point in other test poses , color information composition , uncertainty information composition . Eventually , , , Input into the multi-head attention network trained in step 5.3 to get the output And use the result to replace the light The color of the sampling point with large data uncertainty is calculated, and then the new light color is obtained through the volume rendering formula to replace the pixel color.

[0025] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0026] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0027] From the above technical solutions, it can be seen that the neural radiation field optimization method and device based on energy loss uncertainty voting of the present invention can solve the problem that NLL will damage the training of the network model and reduce the rendering quality under the uncertainty perception NeRF background. At the same time, the uncertainty perception ability of NeRF is utilized, and the multi-head attention mechanism is used to further improve the rendering quality.

[0028] The present invention relates to the field of uncertainty and implicit three-dimensional reconstruction, and discloses a neural radiation field optimization method based on energy loss uncertainty voting, the method comprising: obtaining image data sets of different perspectives and corresponding camera poses in a certain three-dimensional scene, and dividing the data sets into training data sets and test data sets; changing the NLL function used in training to an energy loss function based on ActiveNeRF and using a more reasonable regularization term; using the proposed energy loss training model; building a multi-head attention network model; obtaining a data set for training the multi-head attention network model; training the multi-head attention network model; using the trained attention network model to replace the color values ​​of the sampling points with large noise in the test pose, thereby improving the final rendering effect. The present invention models data uncertainty through a Bayesian method and uses energy loss for training, effectively alleviating the problem that negative log-likelihood loss will produce excessive variance, regardless of whether the predicted mean is correct or not, thereby damaging model training. At the same time, the present invention further utilizes data uncertainty, integrates uncertainty into the attention mechanism, and further improves the rendering quality.

[0029] Specifically, the neural radiation field optimization method based on energy loss uncertainty voting proposed in the present invention has the following advantages: (1) Energy loss is used instead of negative log-likelihood loss to train uncertainty-aware neural radiance field networks, thereby avoiding the problem that negative log-likelihood loss outputs excessive variance regardless of the correctness of the predicted mean, which damages the training of the model.

[0030] (2) A new regular expression for volume density is constructed to replace the regular expression in ActiveNeRF. This regular expression does not require the specification of hyperparameters, thus avoiding the problem of improper hyperparameter selection in ActiveNeRF, which results in excessively high penalty for volume density and causes distortion of rendering results.

[0031] (3) The attention mechanism is integrated into the neural radiance field to replace the color of the sampling points with large data uncertainty in the neural radiance field, thereby improving the final rendering quality. In addition, the attention mechanism also considers the data uncertainty of the candidate value and integrates it into the final weighted result, so that the candidate value with large uncertainty should contribute less to the final output result. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall flow chart of the method of the present invention, including a training flow chart and a testing flow chart; Figure 2 It is a network model structure diagram of uncertainty-aware NeRF of the present invention; Figure 3 This is a structural diagram of the multi-head attention network model proposed by the method of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0034] like Figure 1 As shown, the neural radiation field optimization method based on energy loss uncertainty voting described in this embodiment, Figure 1 The training process and testing process are shown. The training process explains how to use energy loss to train the uncertainty-aware NeRF network and how to train a multi-head attention network model to improve rendering quality. The specific steps include: Step 1: Obtain a dataset for training uncertainty-aware NeRF. First, take photos of a scene from different viewing angles. Then, the COLMAP technique is used to obtain the The camera pose of the image, the final dataset consists of this images and the corresponding camera poses, namely .in , and Represents the height and width of the image, and 3 represents the number of channels of the image. Represents the camera pose, including a rotation matrix and a translation vector , and Used to convert the camera coordinate system to the world coordinate system. In addition, it also contains the height and width of the image and the focal length of the camera , used to convert the image coordinate system to the camera coordinate system. Finally, it also includes the near plane and the far plane , represents the imaging boundary, which is used to limit the range of sampling points. , evenly extracted from As the training set, , the remaining data as the test set, that is, .

[0035] Step 2: Load the uncertainty-aware NeRF network model, such as Figure 2 The input of the network is the position information of a sampling point on the light. The result after encoding And perspective information The result after encoding , the output of the network is the color information of the point , volume density And the corresponding data uncertainty The main part of the network consists of 8 fully connected layers, each of which contains 256 neurons and uses Function is activated. In addition, in order to retain the input information, the input information is converted to It is concatenated with hidden features. and data uncertainty Through the output of the 8th hidden layer, in order to ensure the non-negativity of both, the volume density Used Activation, data uncertainty usage Activate. Color value Related to the perspective, so the perspective information is spliced ​​in the 9th hidden layer , and finally output the color value through two connection layers, and use The activation function ensures the color range. The encoding formula used above is:

[0036] When the location information When encoding, , for viewing angle information When encoding, .

[0037] Step 3: Use energy loss to train the network model in step 2. An image in , you can get A ray of light then passes through The transformation matrix in the world coordinate system is converted into a ray in the world coordinate system. For the sake of convenience, consider one of the rays ,in are the coordinates of the light’s starting point, is the light direction, is the coefficient, and the true color corresponding to the light is recorded as First, the light exist and 64 points are uniformly sampled within the range, that is, for , bring it into In the expression, 64 sampling points are obtained. Then these sampling points are sent to the coarse network in NeRF to obtain the color values ​​corresponding to these sampling points And body density , inputting it into the volume density formula in formula (1) can obtain the rough color value of the image And the weight information corresponding to each sampling point Next, we can use the weight information again and 128 points are sampled within the range, and then the information of these 128 fine sampling points and the information of the previous 64 coarse sampling points are spliced ​​into 192 sampling points , input the information of these 192 sampling points into Figure 2 In the network model in , the corresponding color value can be obtained , volume density value and the corresponding uncertainty Since the color of the sampling point Modeling is Gaussian distribution, that is , so according to the volume density formula, we know that the light Color .in , , For the Sampling points and The Euclidean distance between the sampling points. In order to use the energy loss, the light The distribution Independent sampling Samples . Finally, , , as well as Input into the energy loss formula (8) to obtain the light Loss During the specific training, one image is randomly selected each time and Randomly select 1024 rays from the rays to obtain their average loss , and then by The optimizer performs back propagation for optimization. The specific optimization strategy and learning rate setting are consistent with NeRF.

[0038] Step 4: Construct a dataset for training the attention network. To construct a dataset for training the attention network, you need to prepare the following datasets: ,key ,value , true value , and also introduces data uncertainty Next, we will explain in detail how to obtain these data. Assume that the fine network model trained with energy loss in step 3 is denoted as , Indicates the result after encoding the location information of the sampling point. Represents the result of encoding the viewing angle information of the sampling point. The network model outputs a three-dimensional color value , one-dimensional volume density value And one-dimensional uncertainty For the sake of convenience, consider how to obtain a certain ray Corresponding query ,key ,value , true value and uncertainty First, the light exist and 64 points are sampled evenly within the range , Contains position information and viewing angle information, that is, .Will Input into the trained coarse network model, then the corresponding weight set can be obtained , according to the weight set, the light Sample 128 points again Then and 192 sampling points can be obtained by splicing, that is, , .Will Input to the trained fine network model In the above example, you can get the color value set corresponding to these 192 sampling points. , the volume density value set and the corresponding uncertainty In order to obtain , , , as well as , the first step is to determine which points have less data uncertainty, and set a threshold for this:

[0039] in , , It is a custom parameter. . Then the set middle The sampling points are considered to have small data uncertainty. So this The set of position information and viewing angle information corresponding to the sampling points is recorded as ,Right now , the corresponding color value is recorded as ,Right now The second step is to determine , , ,for , which indicates that Sampling points in other A set of position information and perspective information under each pose, namely ,in express Among the sampling points The sampling point is The position information of the camera pose, Indicates the corresponding viewing angle information. and , Indicates this Sampling points in other A set of color information under each position, Indicates this Sampling points in other The uncertainty information under each position is a set. Input to the trained fine network model , then the set of color values ​​output by the model constitutes , the output uncertainty set is composed of , so there is ,in express Among the sampling points The sampling point is Color information at each camera pose. ,in express Among the sampling points The sampling point is The uncertainty information of the camera pose. Each of the images is randomly selected The rays constitute the dataset for training the multi-head attention network, namely , , , , .

[0040] Step 5: Build a multi-head attention network model, such as Figure 3 As shown; the network model contains For each attention head, its input is the position information and perspective information of a certain sampling point , the sampling point is in other Position information and viewing angle information under each camera pose , , the sampling point is in other Color information at each camera pose , and uncertainty information , . Indicates The fully connected layer in the attention head receives an input dimension of , the output dimension is . Indicates adding multiple results. is the activation function, which is used to convert the intermediate results of the attention head Transformation to attention score , and make . Indicates The output of an attention head. Represents the prediction result of the sampling point color, In the algorithm of the present invention, , , , .

[0041] Step 6: Train the attention network model. The multi-head attention network model obtained in step 5 needs to be trained based on the data set obtained in step 4. The specific steps are as follows: For each sampling point, its corresponding query condition can be found in the data set obtained in step 4. , candidate key , candidate value , true value , uncertainty .Will , , , Input it into the multi-head attention network model in step 5 to obtain the color prediction value of the sampling point , the loss is the predicted value With the true value In the algorithm of the present invention, batch training is adopted, and the batch size Set to 4096, so each time from the data set in step 4 The sampling points correspond to , , , Input into the multi-head attention network model, output prediction values, the final loss is Predicted values With the corresponding True value of military losses, namely:

[0042] Next, we only need to In the algorithm of the present invention, the optimizer adopts The optimization parameters are the fully connected layer parameters in the multi-head attention. The learning rate is 0.01 and decreases with the number of training times, and finally drops to 0.001. The number of training times is set to 100.

[0043] also, Figure 1 The test process describes how to use the trained uncertainty-aware NeRF network model and the multi-head attention model to improve the quality of the final test results. Given the test dataset divided in step 1 ,according to And the trained uncertainty-aware NeRF model can obtain the corresponding predicted image And the corresponding uncertainty image First, we need to select the pixel colors with greater uncertainty in the predicted image and replace and improve them, so a threshold is set according to formula 13:

[0044] in , , is the coefficient, which is 0.8 in the algorithm of the present invention. The uncertainty value of the pixel color is greater than All need to be replaced and upgraded. Specific steps are as follows Figure 1 For the sake of convenience, let's consider how a pixel with large uncertainty is replaced. This is divided into two steps: Step 1): Get the light corresponding to the pixel color The sampling points on the graph and the color, volume density, and uncertainty corresponding to the sampling points. There are 192 sampling points on the network, including 128 fine sampling points and 64 coarse sampling points. These 192 sampling points are recorded as ,in Indicates The position information and viewing angle information corresponding to the sampling points, namely: Then these sampling points are input into the trained uncertainty-aware NeRF network model to obtain the color set of these sampling points. , volume density collection , uncertainty set .

[0045] Step 2): Use a multi-head attention network to replace the color of the sampling points with large uncertainty in step 1, and then use volume rendering technology to recalculate the light after replacement color and replace the original light The corresponding pixel color. Specifically, we first need to select the light For the uncertainty set obtained in step 1, , first set a threshold:

[0046] in , , is a custom coefficient, and its value is 0.8 in the algorithm of the present invention. The uncertainty value is greater than The color value of the sampling point needs to be replaced. Assuming that there are , then this The position information and viewing angle information corresponding to each sampling point constitute the query ,Right now , ;this Sampling points in other The position information and view information under the test poses constitute the candidate key ,in express The sampling point to be replaced The sampling point is The position information and viewing angle information of each test pose. Sampling points in other The color information under the test poses constitutes the candidate value ,in express The sampling point to be replaced The sampling point is Color information under the test pose; this Sampling points in other The uncertainty information under each test pose constitutes ,in express The sampling point to be replaced The sampling point is The uncertainty information under the test pose. Then , , , Input into the trained multi-head attention network model to get the output result ,in Indicates the color used for replacement. Finally, for light , which has a total of The color and volume density of each sampling point do not need to be changed. The color of the sampling point needs to use the output of the multi-head attention network Replace the volume density. Output the processed 192 sampling points and volume density to the volume rendering formula 1 to obtain the light The new color replaces the original light with the changed color The corresponding pixel color can complete the final enhancement.

[0047] Table 1

[0048] Table 2

[0049] Tables 1 and 2 show the results of the comparative experiments. Table 1 is a comparison of the effects of NLL and ES. Table 1 compares the index results of the original NeRF, NeRF trained with energy loss (ES), and NeRF trained with negative log-likelihood loss (NLL). The bolded ones indicate better results. Blender and LLFF belong to two different types of training data sets. PSNR indicates peak signal-to-noise ratio. The higher the value, the smaller the noise. SSIM indicates structural similarity. The higher the value, the more similar the two images are. LPIPS indicates the similarity of learned perceptual image blocks. This evaluation index better reflects human perception. The smaller the value, the more realistic the image is. Table 2 is a comparison of the effects before and after optimization. Table 2 compares the results before and after optimization using a multi-head attention network. On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0050] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0051] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the neural radiation field optimization methods based on energy loss uncertainty voting in the above-mentioned embodiments.

[0052] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above methods.

[0053] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0054] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0055] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0056] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural radiation field optimization method based on energy loss uncertainty voting, characterized in that: The following steps are involved: Step 1, obtain image datasets of different perspectives in a three-dimensional scene and corresponding camera poses, and divide the datasets into training datasets and test datasets; Step 2: Based on ActiveNeRF, the NLL function used in training is changed to an energy loss function; Step 3: Build the GPU environment required for training, load the parameter configuration file and the model used in ActiveNeRF, and use the energy loss function to train on the training set; Step 4: After training, an uncertainty-aware NeRF network model is obtained. For any point in space, the NeRF network model can output the corresponding color value and data uncertainty. Step 5: Based on step 4, a multi-head attention network model is trained, and its output is used to replace the color values ​​of sampling points with large data noise in the space to improve the rendering quality; Step 6: Use the attention network model trained in step 5 to replace the color values ​​of the sampling points with large noise in the test pose to improve the final rendering effect.

2. The neural radiation field optimization method based on energy loss uncertainty voting according to claim 1 is characterized in that: Step 2 includes the following: ActiveNeRF turns light A sampling point on The color value of is modeled as a Gaussian distribution, that is ,in Represents the predicted color value, variance Indicates the corresponding data uncertainty; An additional output head is added to the MLP network model used by the original NeRF to output the variance value, so that a point in a given space The network model finally outputs a five-dimensional vector , respectively represent the predicted color value, volume density and uncertainty; According to the volume density formula: in represents the number of sampling points, Indicates sampling point With sampling point The Euclidean distance between Color It also obeys the Gaussian distribution, that is: (2) Given a ray True Color , ActiveNeRF uses NLL for optimization training, and the loss is as follows: in Represents light gather, Representing a collection The number of elements in is a regularization term used to constrain the volume density to prevent artifacts from being too large. is a hyperparameter, The number of sampling points on the light; Since the function of the coarse network is to obtain weights To perform better fine sampling, ActiveNeRF only performs Gaussian distribution modeling on the color values ​​output by the fine network model, keeping the function of the coarse network unchanged, so the color values ​​synthesized by the coarse network Color value synthesized with fine network , the loss finally used by ActiveNeRF is as follows: (4) ； Use the energy loss function ES to replace NLL to train the detailed network as follows: For the distribution of light color in formula (2) Independently sample M points, denoted as , then the true color of the given light , the energy loss is expressed as: ； In addition, the first rays output to the network are included on The volume density value of each sampling point is scaled to between (0,1], that is: (6) in The operation is only used for scaling and does not participate in training, so it does not carry gradients; then use As the volume density The regularization term encourages Less than 0.5 tends to 0, Greater than 0.5 tends to 1, so in the end The energy loss is expressed as: Given a set of rays , the final loss is expressed as the MSE loss corresponding to the coarse network plus the energy loss of the fine network, that is: 。 3. The neural radiation field optimization method based on energy loss uncertainty voting according to claim 1 is characterized in that: Step 5 includes the following steps: Step 5.1: Prepare the training set for training the multi-head attention network; Assume that the training set ,in , , then for any set of data , from which the world coordinate system is obtained The starting position and direction information of the light ray; randomly extract rays, then for a ray Fine sampling possible points, input these points into the fine network trained with energy loss in step 4, obtain the corresponding color value and data uncertainty; then select These points are considered to be data with less noise, and the color values ​​output by the network are considered to be relatively accurate. Therefore, the color values ​​of these points are used as ; Consider this Point points, and take the color value of the corresponding network output as the true value , is a 3-dimensional vector, and its position information And perspective information As a query condition ,Right now , is a 6-dimensional vector; then For point In other The collection of position information and viewing angle information under the camera pose, that is ; For point In other The color value set under the camera pose is obtained by The position information and viewing angle information under other camera poses are input into the trained detailed network to obtain In addition, the network will also output the corresponding data uncertainty set ; Step 5.2: Build a multi-head attention network model to substitute the color values ​​of sampling points with large data uncertainty on the light; Consider the obtained step 5.1 How does a certain point in the multi-head attention network model work? The true value corresponding to this point , query ,key ,value , uncertainty ; the goal is to , , , Get network output With the true value Use loss for training; Assume that the attention model has output head, then for the Output header, the output result It is expressed as: Here we use the additive attention method to obtain the attention score, where and It is The two fully connected layers in the output head are responsible for converting the 6-dimensional query And the 6-dimensional key Map to In dimensional space, Corresponding to a fully connected layer, used to dimensional space is mapped to 1-dimensional space, is the activation function; After scaling, the range becomes , This is to reduce the value of high uncertainty in the data The corresponding weight; therefore, the final output result Is this The results of the output heads are accumulated, that is: Step 5.3: Construct the loss function required for training the multi-head attention network model and train it; in the data preparation stage of step 5.1, there are images, each of which is randomly selected rays, each of which takes If there are points with less data noise, then according to step 5.2, the final loss function is obtained: 。 4. The neural radiation field optimization method based on energy loss uncertainty voting according to claim 3 is characterized in that: In step 6, the following are included: For the uncertainty-aware NeRF network model trained with energy loss in step 2, given the test pose , get the predicted image result And the corresponding data uncertainty image ; In order to improve the quality of the predicted image, first select the predicted image result The corresponding data uncertainty image For pixels with larger median values, get the corresponding light for one of the pixels , then from the light Select sampling points with large uncertainty from the fine sampling points, and combine the position information and viewing angle information corresponding to these sampling points into , the position information and viewing angle information of the sampling point in other test poses , color information composition , uncertainty information composition ; will eventually , , , Input into the multi-head attention network trained in step 5.3 to get the output And use the result to replace the light The color of the sampling point with large data uncertainty is calculated, and then the new light color is obtained through the volume rendering formula to replace the pixel color.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 4.

Images