A 3D image rendering method and system based on neural radiation fields

Abstract

This application discloses a three-dimensional image rendering method and system based on neural radiation fields. The technical solution provided in this application determines the first three-dimensional coordinates of the target and the corresponding first ray direction. Then, the first three-dimensional coordinates and the first ray direction are input into a pre-constructed neural radiation field model. Based on the neural radiation field model, the volume density of the first three-dimensional coordinates at the corresponding first ray direction, as well as the diffuse reflection and specular reflection features of the corresponding color values, are predicted. Subsequently, volume rendering is performed based on the diffuse reflection features and volume density to obtain a first rendering result, and volume rendering is performed based on the specular reflection features and volume density to obtain a second rendering result. Finally, a rendered image of the target is output based on the first and second rendering results. Using the above technical means can ensure image rendering quality, shorten the image volume rendering time, thereby improving the real-time performance of three-dimensional image rendering and enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a three-dimensional image rendering method and system based on neural radiation fields. Background Technology

[0002] Neural Radiance Fields (NeRF) are an implicit representation of a 3D scene, representing it as the volume density and color value of any point in space. With this scene representation in the form of NeRF, the scene can be rendered to generate images from any viewpoint. Currently, 3D image rendering based on NeRF typically uses a fully connected neural network to construct a mapping from the image's 3D coordinates (x, y, z) to the volume density σ and color value c. Volume rendering is then performed based on the volume density σ and color value c, achieving 3D image rendering through extensive neural network inference.

[0003] However, in the above-mentioned 3D image rendering process, a lot of time is required for volume rendering for each 3D point. The whole process is relatively lengthy and complex, and 3D image rendering lacks real-time performance. Summary of the Invention

[0004] This application provides a three-dimensional image rendering method and system based on neural radiation fields, which can improve the real-time performance of three-dimensional image rendering while ensuring image rendering quality, and solve the technical problem of excessively long three-dimensional image rendering process.

[0005] In a first aspect, embodiments of this application provide a three-dimensional image rendering method based on neural radiation fields, comprising:

[0006] Determine the first three-dimensional coordinates of the target and the corresponding first ray direction;

[0007] Input the first three-dimensional coordinates and the first ray direction into the pre-constructed neural radiation field model, and predict the volume density of the first three-dimensional coordinates in the corresponding first ray direction, as well as the diffuse reflection and specular reflection features of the corresponding color values ​​based on the neural radiation field model;

[0008] A first rendering result is obtained by performing volume rendering based on diffuse reflection features and volume density. A second rendering result is obtained by performing volume rendering based on specular reflection features and volume density. The rendered image of the captured target is then output based on the first and second rendering results.

[0009] In a second aspect, embodiments of this application provide a three-dimensional image rendering system based on neural radiation fields, comprising:

[0010] The module is configured to determine the first three-dimensional coordinates of the target and the corresponding first ray direction.

[0011] The prediction module is configured to input the first three-dimensional coordinates and the first ray direction into a pre-constructed neural radiation field model, and predict the volume density of the first three-dimensional coordinates in the corresponding first ray direction, as well as the diffuse reflection and specular reflection features of the corresponding color values ​​based on the neural radiation field model.

[0012] The rendering module is configured to perform volume rendering based on diffuse reflection features and volume density to obtain a first rendering result, and perform volume rendering based on specular reflection features and volume density to obtain a second rendering result, and output a rendered image of the captured target based on the first rendering result and the second rendering result.

[0013] In a third aspect, embodiments of this application provide a three-dimensional image rendering device based on neural radiation fields, comprising:

[0014] Memory and one or more processors;

[0015] The memory is configured to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional image rendering method based on neural radiation fields as described in the first aspect.

[0017] In a fourth aspect, embodiments of this application provide a storage medium containing computer-executable instructions configured, when executed by a computer processor, to perform the three-dimensional image rendering method based on neural radiation fields as described in the first aspect.

[0018] In a fifth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the three-dimensional image rendering method based on neural radiation fields as described in the first aspect.

[0019] This application embodiment determines the first three-dimensional coordinates of the target and the corresponding first ray direction. Then, the first three-dimensional coordinates and the first ray direction are input into a pre-constructed neural radiation field model. Based on the neural radiation field model, the volume density of the first three-dimensional coordinates at the corresponding first ray direction, as well as the diffuse reflection and specular reflection features of the corresponding color value, are predicted. Subsequently, volume rendering is performed based on the diffuse reflection features and volume density to obtain a first rendering result, and volume rendering is performed based on the specular reflection features and volume density to obtain a second rendering result. The rendered image of the target is then output based on the first and second rendering results. By employing the above technical means, by performing volume rendering of the target based on specular reflection features and diffuse reflection features respectively, and then combining the two volume rendering results to obtain the rendered image of the target, the image rendering quality is ensured, and the efficiency of 3D image rendering is improved through two parallel volume rendering processes, shortening the image volume rendering time, thereby improving the real-time performance of 3D image rendering and enhancing the user experience. Attached Figure Description

[0020] Figure 1 This is a flowchart of a three-dimensional image rendering method based on neural radiation fields provided in an embodiment of this application;

[0021] Figure 2 This is a flowchart of the training process for the neural radiation field model in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram illustrating the generation of the rendered image in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of loss constraints based on contour information and depth information in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of a three-dimensional image rendering system based on neural radiation fields provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of a three-dimensional image rendering device based on a neural radiation field provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0027] The three-dimensional image rendering method based on neural radiation field provided in this application aims to render the target object by using the neural radiation field model based on the specular reflection and diffuse reflection features of the target object's density and color values, respectively. Then, the two volume rendering results are combined to obtain the rendered image of the target object. This not only ensures the image rendering quality, but also improves the efficiency of three-dimensional image rendering through two parallel volume rendering processes.

[0028] Existing 3D image rendering schemes typically use a fully connected neural network to construct a mapping from the image's 3D coordinates (x, y, z) to volume density σ and color value c, and then perform volume rendering based on the volume density σ and color value c. For a 1080x1920 image, this requires approximately 400 million inferences. Furthermore, each 3D coordinate point requires a separate neural network inference, which is extremely time-consuming, thus failing to achieve real-time rendering. Therefore, this application provides a 3D image rendering method and system based on neural radiation fields to solve the technical problem of excessively long 3D image rendering time.

[0029] Example:

[0030] Figure 1 A flowchart of a three-dimensional image rendering method based on neural radiation fields, provided in an embodiment of this application, is given. This method can be executed by a three-dimensional image rendering device based on neural radiation fields. This device can be implemented through software and / or hardware. The device can consist of two or more physical entities, or it can consist of a single physical entity. Generally, this device can be an image processing server, computer, mobile phone, tablet, or other processing device.

[0031] The following description uses the neural radiation field-based 3D image rendering device as an example to illustrate the implementation of the neural radiation field-based 3D image rendering method. (Refer to...) Figure 1 The three-dimensional image rendering method based on neural radiation fields specifically includes:

[0032] S110. Determine the first three-dimensional coordinates of the target and the corresponding first ray direction;

[0033] S120. Input the first three-dimensional coordinates and the first ray direction into the pre-constructed neural radiation field model, and predict the volume density of the first three-dimensional coordinates in the corresponding first ray direction and the diffuse reflection and specular reflection features of the corresponding color value based on the neural radiation field model.

[0034] S130. Perform volume rendering based on diffuse reflection features and volume density to obtain a first rendering result, perform volume rendering based on specular reflection features and volume density to obtain a second rendering result, and output the rendered image of the captured target based on the first rendering result and the second rendering result.

[0035] In this embodiment of the application, to shorten the image rendering stage and improve the real-time performance of image rendering during 3D image rendering, a 3D radiation field model is used to generate rendered images through a neural radiation field model oriented towards real-time rendering. Specifically, the target object is rendered based on specular reflection features and diffuse reflection features respectively, using a space-for-time trade-off approach to reduce the inference time of the neural network model and improve the model's ability to render images in real time.

[0036] In the process of rendering a 3D image of a target, the 3D coordinates of the target and the corresponding ray direction are obtained and defined as the first 3D coordinates and the first ray direction. Then, a pre-constructed neural radiation field model is input with the first 3D coordinates and the first ray direction. Based on the neural radiation field model, the volume density and color value of the first 3D coordinates at the corresponding ray direction are predicted. The color value includes diffuse reflection features and specular reflection features. Volume rendering is then performed based on the neural radiation field model. A first rendering result is obtained based on diffuse reflection features and volume density, and a second rendering result is obtained based on specular reflection features and volume density. The rendered image of the target is then output based on the first and second rendering results. Thus, by performing two volume rendering processes on the target based on specular reflection features and diffuse reflection features respectively, the final rendered image is obtained by combining the two volume rendering results. Since the volume rendered image incorporates the volume rendering results of specular reflection features and diffuse reflection features, high image rendering quality can be guaranteed. Simultaneously, the two rendering processes are performed in parallel, which improves volume rendering efficiency and the real-time performance of 3D image rendering.

[0037] Prior to this, a neural radiation field model is pre-trained to execute the above-mentioned 3D image rendering process. Based on the model input of the 3D coordinates of the target and the direction of light, the rendered image corresponding to the target is output.

[0038] Specifically, refer to Figure 2 The training process for the neural radiation field model includes:

[0039] S1001. Obtain the second three-dimensional coordinates, second ray direction, and target image of the training target;

[0040] S1002. The neural radiation field model is trained using the second three-dimensional coordinates and the second ray direction as model inputs. Model calculations are performed based on the neural radiation field model, and the rendered image of the training target is output.

[0041] S1003. Based on the target image and the rendered image of the training target, calculate the corresponding first loss function value using the pre-constructed first loss function, and adjust the model parameters according to the first loss function value until the neural radiation field model converges.

[0042] When training the neural radiation field model, training samples are constructed using different training targets (such as different objects). Specifically, the model is trained by obtaining the three-dimensional coordinates and ray direction of the training target as model input, and these three-dimensional coordinates and ray direction are defined as the second three-dimensional coordinates and the second ray direction. Furthermore, the target image of the training target is acquired for subsequent loss function calculation of the model output.

[0043] When acquiring the second three-dimensional coordinates, images are captured corresponding to the training target. For example, several 360° surround shooting points are set from different locations corresponding to the shooting target, and images of the training target are captured at each shooting point using shooting devices such as mobile phones and cameras. The number of images to be captured is set according to the actual training needs, generally ensuring that about 50-150 usable images are sufficient. This application embodiment does not impose a fixed limit on the specific number of images captured, and will not be elaborated here.

[0044] Then, based on the captured 2D images, the 3D coordinates of each point on the training target are predicted. Based on the image coordinates of the captured images, combined with camera intrinsic and extrinsic parameters, the second 3D coordinates of each point on the training target are obtained through rotation and translation transformations. There are many implementation methods for determining 3D coordinates based on 2D images and camera parameters, which will not be elaborated here.

[0045] On the other hand, the second ray direction for each second three-dimensional coordinate is determined by pose estimation. Since the neural radiation field model requires ray direction as input, the camera pose of the photos in the training set needs to be estimated. Furthermore, based on the estimated camera pose, the second ray direction corresponding to the second three-dimensional coordinate is obtained using rotation and translation transformation matrices.

[0046] Furthermore, the target image is obtained by performing foreground segmentation on the corresponding training target's captured image and removing the background. Based on the captured training target image, foreground segmentation is used to retain the foreground portion of the image and remove the background portion to obtain the target image. This ensures the accuracy of the loss function calculation and avoids image noise affecting the adjustment of model parameters. This ensures that the rendered image generated by the model is clean and complete, optimizing the image rendering effect. The foreground segmentation can be performed using contour recognition segmentation to determine the foreground portion, thereby segmenting the target image. This application does not impose fixed limitations on the implementation method of image foreground segmentation, and will not elaborate further here.

[0047] Furthermore, based on the determined second three-dimensional coordinates and second ray direction, the second three-dimensional coordinate x and the corresponding second ray direction d are used as model inputs. Through model training, the volume density of the second three-dimensional coordinates in the corresponding second ray direction and the diffuse reflection and specular reflection features of the corresponding color values ​​are predicted. Then, volume rendering is performed based on the diffuse reflection features and volume density, and volume rendering is performed based on the specular reflection features and volume density. The rendered image of the training target can be output based on the volume rendering results of the above two parts.

[0048] Optionally, when performing model calculations for each second three-dimensional coordinate, in order to further shorten the model prediction time, this embodiment of the application maps the second three-dimensional coordinates and the second ray direction to the grid feature space based on the grid hash coding algorithm, filters the second three-dimensional coordinates and the second ray direction based on the grid feature space, obtains the corresponding fused feature information, and uses the fused feature information to perform model calculations for the neural radiation field model.

[0049] It is understandable that performing inference for each 3D coordinate using a fully connected neural network (8 layers, 512 dimensions) would consume a significant amount of time. Therefore, this application employs a grid-hash encoding algorithm, mapping the second 3D coordinates to a grid feature space using a specific hash function. Specifically, the side length of each unit grid can be represented as a corresponding coordinate component, generating encoded coordinate information based on the side length of each unit grid, thus fusing feature information. Furthermore, coordinate points within the grid are filtered, with 3D coordinates lacking obvious or important features being removed. For example, the 3D coordinates of a hollowed-out area in an image have less obvious features, and these 3D points can be removed to reduce the computational load. By pre-setting corresponding feature filtering rules, filtering is performed based on the feature information of the 3D coordinates to reduce computational load. For 3D coordinates at different resolutions, encoding can fuse multi-level information, thus accelerating model training and inference while reducing the size of the neural network. The grid encoding formula is as follows:

[0050] Encoded_feature=Hash_mapping(x,y,z)

[0051] Where x, y, z represent the second three-dimensional coordinates, Hash_mapping(x, y, z) represents the grid hash encoding of the second three-dimensional coordinates, and Encoded_feature represents the grid hash encoding result, i.e., the fused feature information.

[0052] Then, based on this fused feature information, the volume density of the second 3D coordinates in the corresponding second ray direction, as well as the diffuse and specular reflection features of the corresponding color value, are predicted. This process can be formally represented as:

[0053] F w :(x,d)→(c,σ)

[0054] Where σ represents volume density and c represents color value. This color value includes a 7-dimensional color feature output, including three-dimensional diffuse reflection features (i.e., the three dimensions of RGB) and four-dimensional specular reflection features (i.e., RGB plus the dimension of light influence).

[0055] Then, based on the volume density and color values ​​predicted by the model, volume rendering is performed using specular reflection and diffuse reflection features respectively. The rendered image of the training target can then be obtained from these two volume rendering results. It can be understood that volume density can be interpreted as the probability that a ray will terminate at a point in a 3D scene; this probability is differentiable. Since the points on a ray are continuous, the color of this ray can be obtained by integration. Volume rendering integrates the volume density and the color value of each point to obtain the color of this ray, thereby generating the corresponding rendered image.

[0056] Specifically, based on volume rendering technology, the volume density and color values ​​of the training target under different lighting directions can be combined into a complete rendered image. This rendered image is represented as follows:

[0057]

[0058] Reference Figure 3 To accelerate the rendering process, this embodiment adopts a space-for-time trade-off approach, decomposing the entire volume rendering process into two independent volume rendering steps. These steps are performed based on diffuse-only features and view-dependent features, respectively, to obtain the corresponding volume rendering results. The volume rendering formula is expressed as:

[0059] F w :(x)→(cd ,v s ,σ)

[0060]

[0061]

[0062] Among them, c d The diffuse reflectance characteristic representing color values, v s The specular reflection characteristic represents the color value, and σ represents the volume density. This represents the volume rendering result based on diffuse reflection features and volume density. This represents the volume rendering result based on specular reflection features and volume density.

[0063] Among them, the volume rendering result based on diffuse reflection features and volume density only includes the color information of the training target itself, while the model predicts the viewpoint-based specular reflection features v s A volume rendering operation is performed on the volume density σ. The result of this rendering, combined with the influence of lighting, determines the color information of the training target. Therefore, by combining the results of these two volume rendering operations, a higher quality rendered image can be obtained. Furthermore, since the two volume rendering processes are performed in parallel, the volume rendering time can be reduced, improving the efficiency of model image rendering.

[0064] Furthermore, this is based on the specular reflection feature v s The volume rendering result, including volume density σ, needs to be combined with ray d and passed through a multilayer perceptron to obtain the network output. Then, based on the network output and the volume rendering result based on diffuse reflection features, the final rendered image is determined.

[0065] When generating the rendered image of the training target, a small fully connected neural network (3 layers, 16 dimensions) is used for feature fusion. The feature fusion formula is expressed as:

[0066]

[0067] Therefore, the neural radiation field model only needs to perform a very small neural network inference for each ray during the image rendering process, and can achieve the effect of real-time image rendering in a short time.

[0068] Furthermore, considering that the quality of image data acquisition may be affected by environmental changes (especially lighting conditions) in the image rendering process, different rendering schemes can be adopted for different lighting environments. That is, when the lighting changes are stable and the lighting conditions are sufficient, the image is rendered using the aforementioned feature fusion method to obtain the rendered image. However, when the lighting changes drastically or the lighting conditions are insufficient, only the volume rendering result of the diffuse reflection features can be used as the final rendered image output, thus avoiding the interference of light changes. In this case, the rendered image is represented as:

[0069]

[0070] By ignoring volume rendering results based on specular reflection features in insufficient lighting conditions, and reducing the computation of these volume rendering results, the image rendering speed of the model can be further improved, enhancing the real-time performance of 3D image rendering. Therefore, corresponding setting standards are pre-set based on lighting conditions for lighting condition judgment. Then, during image rendering, by comparing the lighting conditions of the shooting environment with these setting standards, the appropriate volume rendering result can be adaptively selected to generate the 3D rendered image based on the comparison results.

[0071] Finally, the model is trained using the target image C as supervised information, and the loss function is calculated by combining the rendered image. This loss function is defined as the first loss function, which is expressed as follows:

[0072]

[0073] The parameters are used to iteratively train the neural radiation field model until the value of the first loss function reaches the set value, indicating that the neural radiation field model has the ability to generate high-quality rendered images. The model converges, thus completing the training of the neural radiation field model.

[0074] Optionally, when calculating the loss function, a pre-constructed second loss function is used to calculate the corresponding value of the second loss function based on the depth and contour information of the training target. The model parameters are then adjusted according to the value of the second loss function until the neural radiation field model converges. (Refer to...) Figure 4 To avoid insufficient information obtained during data acquisition (such as blurry images, inaccurate camera pose estimation, and unclean foreground segmentation), which could lead to hazy or flawed rendered images, this application also uses the depth information (Depth loss) and contour information (Mask loss) of the training target as prior information constraints to supervise the training process. A second loss function is calculated by comparing the depth and contour information of the training target with the depth and contour information of the rendered image. The calculated value of this second loss function is used to constrain the rendered results during training to have a certain degree of physical consistency, resulting in cleaner and more complete rendered images generated by the model, thus optimizing image rendering quality.

[0075] Subsequently, based on the neural radiation field model that has been trained, when rendering a 3D image of any captured target, the first 3D coordinates and the first ray direction are first determined, referring to the image generation method of the trained target. The first 3D coordinates and the first ray direction are then predicted by the model after being mesh-encoded, obtaining the volume density of the first 3D coordinates in the corresponding first ray direction, as well as the diffuse and specular reflection features of the corresponding color values. Then, referring to the volume rendering process of the trained target, volume rendering is performed based on the diffuse reflection features and volume density to obtain the first rendering result, and volume rendering is performed based on the specular reflection features and volume density to obtain the second rendering result. The rendered image of the captured target is then output based on the first and second rendering results.

[0076] Specifically, when outputting a rendered image of the target based on the first and second rendering results, the second rendering result and the first ray direction are input into a multilayer perceptron to obtain the network output result. The first rendering result or the feature fusion result of the first rendering result and the network output result is used as the rendered image of the target. The features of the second rendering result and the first ray direction are fused using the multilayer perceptron. Subsequently, when using the first rendering result or the feature fusion result of the first rendering result and the network output result as the rendered image of the target, based on the ambient lighting conditions, if the lighting conditions of the target meet the set standard, the feature fusion result is used as the rendered image of the target; if the lighting conditions of the target do not meet the set standard, the first rendering result is used as the rendered image of the target. By adaptively selecting the appropriate volume rendering result to generate the rendered image under different lighting conditions, image quality is ensured while minimizing model computation time and improving the real-time performance of model image rendering.

[0077] As described above, the first three-dimensional coordinates of the target and the corresponding first ray direction are determined. These coordinates are then input into a pre-constructed neural radiation field model. Based on this model, the volume density of the first three-dimensional coordinates along the corresponding first ray direction, as well as the diffuse and specular reflection features of the corresponding color value, are predicted. Subsequently, volume rendering is performed based on the diffuse reflection features and volume density to obtain a first rendering result, and volume rendering is performed based on the specular reflection features and volume density to obtain a second rendering result. The rendered image of the target is then output based on both the first and second rendering results. By employing this technique, and performing volume rendering of the target based on specular and diffuse reflection features respectively, and then combining the two volume rendering results to obtain the rendered image of the target, image rendering quality is ensured. Furthermore, the parallel volume rendering process improves the efficiency of 3D image rendering, reduces volume rendering time, and thus enhances the real-time performance of 3D image rendering, improving the user experience.

[0078] Based on the above embodiments, Figure 5 A schematic diagram of the structure of a three-dimensional image rendering system based on neural radiation fields provided in this application. (Reference) Figure 5 The three-dimensional image rendering system based on neural radiation field provided in this embodiment specifically includes: a determination module 21, a prediction module 22, and a rendering module 23.

[0079] The determining module 21 is configured to determine the first three-dimensional coordinates of the shooting target and the corresponding first ray direction;

[0080] The prediction module 22 is configured to input the first three-dimensional coordinates and the first ray direction into a pre-constructed neural radiation field model, and predict the volume density of the first three-dimensional coordinates in the corresponding first ray direction, as well as the diffuse reflection characteristics and specular reflection characteristics of the corresponding color value based on the neural radiation field model.

[0081] The rendering module 23 is configured to perform volume rendering based on diffuse reflection features and volume density to obtain a first rendering result, and perform volume rendering based on specular reflection features and volume density to obtain a second rendering result, and output a rendered image of the captured target based on the first rendering result and the second rendering result.

[0082] Specifically, the training process for the neural radiation field model includes:

[0083] Obtain the second three-dimensional coordinates, second ray direction, and target image of the training target;

[0084] The neural radiation field model is trained using the second three-dimensional coordinates and the second ray direction as model inputs. Model calculations are performed based on the neural radiation field model, and the rendered image of the training target is output.

[0085] Based on the target image and the rendered image of the training target, the corresponding first loss function value is calculated using a pre-constructed first loss function. The model parameters are adjusted according to the first loss function value until the neural radiation field model converges.

[0086] The model calculations based on the neural radiation field model include:

[0087] The second three-dimensional coordinates and the second ray direction are mapped to the grid feature space based on the grid hash coding algorithm. The second three-dimensional coordinates and the second ray direction are then filtered based on the grid feature space to obtain the corresponding fused feature information. The fused feature information is then used to perform model calculations for the neural radiation field model.

[0088] The target image is obtained by segmenting the foreground of the corresponding training target image and removing the background.

[0089] The training process for the neural radiation field model also includes:

[0090] Based on the depth and contour information of the training target, the corresponding second loss function value is calculated using a pre-constructed second loss function. The model parameters are then adjusted according to the second loss function value until the neural radiation field model converges.

[0091] Specifically, based on the first rendering result and the second rendering result, a rendered image of the target is output, including:

[0092] The second rendering result and the first ray direction are input into the multilayer perceptron to obtain the network output result;

[0093] The rendered image of the target is the first rendered result or the feature fusion result of the first rendered result and the network output result.

[0094] Specifically, the rendered image of the target image, which uses the first rendering result or the feature fusion result of the first rendering result and the network output result, includes:

[0095] When the lighting conditions of the target meet the set standards, the feature fusion result is used as the rendered image of the target.

[0096] If the lighting conditions of the target do not meet the set standard, the first rendering result is used as the rendered image of the target.

[0097] As described above, the first three-dimensional coordinates of the target and the corresponding first ray direction are determined. These coordinates are then input into a pre-constructed neural radiation field model. Based on this model, the volume density of the first three-dimensional coordinates along the corresponding first ray direction, as well as the diffuse and specular reflection features of the corresponding color value, are predicted. Subsequently, volume rendering is performed based on the diffuse reflection features and volume density to obtain a first rendering result, and volume rendering is performed based on the specular reflection features and volume density to obtain a second rendering result. The rendered image of the target is then output based on both the first and second rendering results. By employing this technique, and performing volume rendering of the target based on specular and diffuse reflection features respectively, and then combining the two volume rendering results to obtain the rendered image of the target, image rendering quality is ensured. Furthermore, the parallel volume rendering process improves the efficiency of 3D image rendering, reduces volume rendering time, and thus enhances the real-time performance of 3D image rendering, improving the user experience.

[0098] The three-dimensional image rendering system based on neural radiation fields provided in this application embodiment can be configured to execute the three-dimensional image rendering method based on neural radiation fields provided in the above embodiment, and has corresponding functions and beneficial effects.

[0099] Based on the above practical examples, this application also provides a three-dimensional image rendering device based on neural radiation fields, referring to... Figure 6The neural radiation field-based 3D image rendering device includes a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory 32, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the neural radiation field-based 3D image rendering method described in any embodiment of this application (e.g., the determination module, prediction module, and rendering module in a neural radiation field-based 3D image rendering system). The communication module 33 is configured to perform data transmission. The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory, thereby implementing the aforementioned neural radiation field-based 3D image rendering method. The input device 34 can be configured to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 35 may include a display screen or other display device. The aforementioned neural radiation field-based 3D image rendering device can be configured to execute the neural radiation field-based 3D image rendering method provided in the above embodiments, possessing corresponding functions and beneficial effects.

[0100] Based on the above embodiments, this application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, these computer-executable instructions are configured to perform a three-dimensional image rendering method based on neural radiation fields. The storage medium can be any type of memory device or storage device. Of course, the computer-executable instructions provided in this application are not limited to the three-dimensional image rendering method based on neural radiation fields described above; they can also execute related operations in the three-dimensional image rendering method based on neural radiation fields provided in any embodiment of this application.

[0101] Based on the above embodiments, this application also provides a computer program product. The technical solution of this application, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the three-dimensional image rendering method based on neural radiation field described in the various embodiments of this application.

Claims

1. A three-dimensional image rendering method based on neural radiation fields, characterized in that, include: Determine the first three-dimensional coordinates of the target and the corresponding first ray direction; The first three-dimensional coordinates and the first ray direction are input into a pre-constructed neural radiation field model. Based on the neural radiation field model, the volume density of the first three-dimensional coordinates in the corresponding first ray direction and the diffuse reflection and specular reflection features of the corresponding color values ​​are predicted. A first rendering result is obtained by performing volume rendering based on the diffuse reflection features and the volume density. A second rendering result is obtained by performing volume rendering based on the specular reflection features and the volume density. The rendered image of the captured target is output according to the first rendering result and the second rendering result.

2. The three-dimensional image rendering method based on neural radiation field according to claim 1, characterized in that, The training process for the neural radiation field model includes: Obtain the second three-dimensional coordinates, second ray direction, and target image of the training target; The neural radiation field model is trained using the second three-dimensional coordinates and the second ray direction as model inputs, and model calculations are performed based on the neural radiation field model to output a rendered image of the training target. Based on the target image and the rendered image of the training target, the corresponding first loss function value is calculated using a pre-constructed first loss function, and the model parameters are adjusted according to the first loss function value until the neural radiation field model converges.

3. The three-dimensional image rendering method based on neural radiation fields according to claim 2, characterized in that, The model calculation based on the neural radiation field model includes: The second three-dimensional coordinates and the second ray direction are mapped to the grid feature space based on the grid hash encoding algorithm. The second three-dimensional coordinates and the second ray direction are filtered based on the grid feature space to obtain the corresponding fused feature information. The fused feature information is then used to perform model calculations on the neural radiation field model.

4. The three-dimensional image rendering method based on neural radiation field according to claim 2, characterized in that, The target image is obtained by segmenting the foreground of the image corresponding to the training target and removing the background.

5. The three-dimensional image rendering method based on neural radiation fields according to claim 2, characterized in that, The training process for the neural radiation field model includes: Based on the depth and contour information of the training target, the corresponding value of the second loss function is calculated using a pre-constructed second loss function, and the model parameters are adjusted according to the value of the second loss function until the neural radiation field model converges.

6. The three-dimensional image rendering method based on neural radiation field according to claim 1, characterized in that, The step of outputting a rendered image of the target based on the first rendering result and the second rendering result includes: The second rendering result and the first ray direction are input into the multilayer perceptron to obtain the network output result; The first rendering result or the feature fusion result of the first rendering result and the network output result is used as the rendered image of the shooting target.

7. The three-dimensional image rendering method based on neural radiation field according to claim 6, characterized in that, The step of using the first rendering result or the feature fusion result of the first rendering result and the network output result as the rendered image of the shooting target includes: When the lighting conditions of the target object meet the set standard, the feature fusion result is used as the rendered image of the target object. If the lighting conditions of the target do not meet the set standard, the first rendering result shall be used as the rendered image of the target.

8. A three-dimensional image rendering system based on neural radiation fields, characterized in that, include: The module is configured to determine the first three-dimensional coordinates of the target and the corresponding first ray direction. The prediction module is configured to input the first three-dimensional coordinates and the first ray direction into a pre-constructed neural radiation field model, and predict the volume density of the first three-dimensional coordinates in the corresponding first ray direction, as well as the diffuse reflection characteristics and specular reflection characteristics of the corresponding color value based on the neural radiation field model. The rendering module is configured to perform volume rendering based on the diffuse reflection features and the volume density to obtain a first rendering result, perform volume rendering based on the specular reflection features and the volume density to obtain a second rendering result, and output a rendered image of the captured target based on the first rendering result and the second rendering result.

9. A three-dimensional image rendering device based on neural radiation fields, characterized in that, include: Memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional image rendering method based on neural radiation fields as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are configured to perform the three-dimensional image rendering method based on neural radiation fields as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the three-dimensional image rendering method based on neural radiation fields as described in any one of claims 1-7.

Images