Image rendering model training, image rendering method and device

Abstract

The present invention provides an image rendering model training method, an image rendering method and an apparatus. The model training method includes: inputting multi-angle target scene graphs into a volume rendering model to obtain volume rendering images output by the volume rendering model; training an initialized neural radiance field with implicit scene representation ability based on the volume rendering images and the multi-angle target scene graphs to obtain an image rendering model; wherein, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graphs to obtain an explicit density distribution matrix for characterizing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates volume rendering images based on the voxel features containing density and color values in each sampling point. The present invention can quickly and directly determine sampling points based on the explicit density distribution matrix, improving the training and inference efficiency of the image rendering model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image rendering model training, an image rendering method and a device. Background Art

[0002] NeRF (Neural Radiance Fields) is a depth rendering method, and its main features are implicit scene representation and volume rendering of images. NeRF first learns the scene representation based on the images from different perspectives of the scene, and then generates images at any specified angle from the representation.

[0003] After learning the scene representation based on the images from different perspectives of the scene, if an image at any specified angle needs to be generated, it is necessary to sample in the scene representation and perform image rendering based on the voxels obtained by sampling to obtain the rendered image.

[0004] Since the distribution of scene features in the volume generated based on NeRF is uneven, in order to improve the sampling efficiency, NeRF adopts a secondary sampling method, specifically: first uniformly sample 64 voxels on a projection line, calculate the density distribution along the ray, and then sample 128 samples on the ray according to the density distribution of the ray.

[0005] From the above NeRF-based sampling method, it can be seen that when generating a projection rendering image, it is necessary to sample hundreds of times on the ray corresponding to each image pixel and perform corresponding real-time calculations, which has a large amount of calculation and is very time-consuming. At the same time, when training the network, it is necessary to iterate multiple times on multiple scene images, and the amount of calculation is even greater. Summary of the Invention

[0006] The present invention provides an image rendering model training, an image rendering method and a device to solve the defect of large calculation amount and low efficiency in image rendering in the prior art.

[0007] The present invention provides an image rendering model training method, including:

[0008] Determine multi-angle target scene graphs of a target scene;

[0009] Input the multi-angle target scene graphs into a volume rendering model to obtain volume rendering images output by the volume rendering model;

[0010] Based on the volume rendering images and the multi-angle target scene graphs, train an initialized neural radiance field with implicit scene expression ability to obtain an image rendering model;

[0011] Among them, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix representing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features containing density and color values in each sampling point.

[0012] According to an image rendering model training method provided by the present invention,

[0013] Inputting the multi-angle target scene graph into the volume rendering model to obtain the volume rendering image output by the volume rendering model includes:

[0014] Inputting the multi-angle target scene graph into the reconstruction layer of the volume rendering model, and the reconstruction layer performs projection reconstruction on the multi-angle target scene graph to obtain the explicit density distribution matrix output by the reconstruction layer;

[0015] Inputting the explicit density distribution matrix into the sampling layer of the volume rendering model, and the sampling layer determines sampling points in the projection direction based on the explicit density distribution matrix to obtain the voxel features of each sampling point output by the sampling layer;

[0016] Inputting the voxel features of each sampling point into the rendering layer of the volume rendering model, and the rendering layer performs volume rendering based on the initial parameters of the volume rendering model and the voxel features of each sampling point to obtain the volume rendering image output by the rendering layer; the angle of the volume rendering image is the same as that of the multi-angle target scene graph.

[0017] According to an image rendering model training method provided by the present invention,

[0018] The sampling points are determined based on the following steps:

[0019] Read the density distribution of each voxel point on the target projection line from the explicit density distribution matrix, and the target projection line is determined based on the 3D scene corresponding to the target scene and the pixel points in the projection direction;

[0020] Determine the sampling points based on the density distribution of each voxel point.

[0021] According to an image rendering model training method provided by the present invention, the target projection line is determined based on the following steps:

[0022] Perform normalization processing on the 3D scene;

[0023] Determine a projection line of pixel points perpendicular to the projection direction from the normalized 3D scene as the target projection line.

[0024] According to an image rendering model training method provided by the present invention, the loss function of the image rendering model is determined based on the mean variance between the volume rendering image and the multi-angle target scene graph.

[0025] The present invention also provides an image rendering method, including:

[0026] Determine the target angle of the rendering image corresponding to the target scene;

[0027] Based on the image rendering model, obtain the rendering image corresponding to the target angle;

[0028] Wherein, the image rendering model is trained based on the image rendering model training method as described above.

[0029] The present invention also provides an image rendering model training device, including:

[0030] A first determination unit, configured to determine a multi-angle target scene graph of the target scene;

[0031] A volume rendering unit, configured to input the multi-angle target scene graph into a volume rendering model, and obtain a volume rendering image output by the volume rendering model;

[0032] A model training unit, configured to train an initialized neural radiance field with implicit scene expression ability based on the volume rendering image and the multi-angle target scene graph, and obtain an image rendering model;

[0033] Wherein, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix for characterizing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features including density and color value in each sampling point.

[0034] The present invention also provides an image rendering device, including:

[0035] A second determination unit, configured to determine the target angle of the rendering image corresponding to the target scene;

[0036] An image rendering unit, configured to obtain the rendering image corresponding to the target angle based on the image rendering model;

[0037] Wherein, the image rendering model is trained based on the image rendering model training method as described above.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of any one of the above-mentioned image rendering methods are implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned image rendering methods are implemented.

[0040] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned image rendering methods are implemented.

[0041] For the image rendering model training, image rendering method and device provided by the present invention, after performing projection reconstruction on multi-angle target scene graphs through a volume rendering model to obtain a display density distribution matrix, and determining sampling points in the projection direction based on the explicit density distribution matrix, a volume rendering image is generated based on the voxel features of each sampling point. Since the explicit density distribution matrix can represent the 3D scene density of the scene to be rendered, sampling points can be directly and quickly determined based on the explicit density distribution matrix, avoiding the problems of large computational complexity and low efficiency caused by secondary sampling in traditional methods, and improving the volume rendering efficiency. Since the image rendering model is trained based on the volume rendering image and multi-angle target scene graphs, the training and inference efficiency of the image rendering model is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 is a schematic flowchart of the image rendering model training method provided by the present invention;

[0044] Figure 2 is a schematic flowchart of the image rendering method provided by the present invention;

[0045] Figure 3 is a schematic structural diagram of the image rendering model training device provided by the present invention;

[0046] Figure 4 is a schematic structural diagram of the image rendering device provided by the present invention;

[0047] Figure 5It is a schematic structural diagram of the electronic device provided by the present invention. Specific embodiments

[0048] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0049] NeRF is a depth rendering method, and its main features are implicit scene representation and volume rendering of images. The basic idea of NeRF is to regard the target scene as a 3D volume and train a neural network to represent the scene. By sampling the 3D volume along the projection line in the viewing direction, calculating the color features by the representation network function and performing projection integration, this method can generate a rendered image.

[0050] NeRF uses a neural radiance field to implicitly represent the color features of the scene. The neural radiance field is a deep neural network. The input of the network is the three-dimensional position coordinates of the volumetric scene voxels and the two-dimensional angular coordinates of the viewing camera, and the output is the color density feature of the voxel corresponding to the five-dimensional coordinates.

[0051] NeRF rendering first learns the scene representation based on the images from different perspectives of the scene, and then generates an image at any specified angle from the representation. Therefore, the implementation of the NeRF method can be divided into two steps: learning of scene representation and projection imaging of the scene.

[0052] The trained NeRF model becomes a 3D representation function of the scene. Given the projection direction, a rendered image can be generated according to the volume rendering steps, that is, the data saved in the three-dimensional volume forms a two-dimensional rendered image along the projection direction. The projection integration is implemented in the way of RayMarching, which consists of four steps: 1. Generate rays Ray cast pixel by pixel on the projection image; 2. Sample the voxels of the volume along the rays; 3. Obtain / calculate the voxel characteristics; 4. Accumulate the voxel characteristics to calculate the color grayscale value of the projection image.

[0053] Among them, the most important step in NeRF rendering is sampling, and the color information of each sampling point needs to be obtained by forward calculation of the implicit neural network function of NeRF. Since the scene feature distribution in the volume is uneven, to improve the sampling efficiency, NeRF adopts the method of secondary sampling. The NeRF method first uniformly samples 64 voxels on a projection line and calculates the density distribution along the ray. According to the distribution, NeRF then samples 128 samples on the ray.

[0054] As can be seen from the specific sampling method of NeRF described above, when generating a projection image, it is necessary to perform hundreds of samplings and real-time calculations of the NeRF neural network on the rays corresponding to each image pixel, which is very time-consuming. And when training the network, it is necessary to iterate multiple scene images multiple times, and the amount of calculation is even more huge.

[0055] In view of this, the present invention provides an image rendering model training method. Figure 1 It is a schematic flowchart of the image rendering model training method provided by the present invention. As Figure 1 shown, the method includes the following steps:

[0056] Step 110, determine multi-angle target scene graphs of the target scene;

[0057] Step 120, input the multi-angle target scene graphs into the volume rendering model to obtain a volume rendering image output by the volume rendering model;

[0058] Step 130, based on the volume rendering image and the multi-angle target scene graphs, train an initialized neural radiance field with implicit scene representation ability to obtain an image rendering model;

[0059] Among them, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graphs to obtain an explicit density distribution matrix for characterizing the 3D scene density of the target scene, and after determining the sampling points in the projection direction based on the density distribution matrix, generates a volume rendering image based on the voxel features containing density and color values in each sampling point.

[0060] Here, the target scene is a 3D scene, and the multi-angle target scene graphs of the target scene refer to two-dimensional images corresponding to the target scene from different perspectives. After inputting the multi-angle target scene graphs into the volume rendering model, the volume rendering model can perform volume rendering on the target scene to obtain a volume rendering image.

[0061] Among them, the process of performing volume rendering based on the volume rendering model is as follows:

[0062] Perform projection reconstruction on the multi-angle target scene graphs to obtain an explicit density distribution matrix. This explicit density matrix can be an explicit 3D voxel density representation matrix for characterizing the 3D scene density of the target scene. And since this matrix is an explicit matrix, 3D scene density information can be directly read from this matrix without having to calculate the 3D scene density information based on this matrix. Then, after determining the sampling points in the projection direction based on the density distribution matrix, generate a volume rendering image based on the voxel features containing density and color values in each sampling point.

[0063] It should be noted that if the traditional sampling method is adopted, the first sampling needs to be carried out on the projection line corresponding to the projection direction (for example, uniformly sampling 64 voxels), then the density distribution on the projection line is calculated based on these voxels, and the second sampling is carried out based on the density distribution (for example, sampling 128 voxels on the projection line again). Since there are invalid voxels in the target scene (such as the voxels corresponding to the background), in the traditional method, when calculating the density distribution based on the voxels, the invalid voxels will also be calculated, thus wasting the calculation amount.

[0064] However, in the embodiment of the present invention, the multi-angle target scene map is projected and reconstructed to obtain an explicit density distribution matrix for characterizing the 3D scene density. Then, after determining the sampling points in the projection direction according to the explicit density distribution matrix, a volume rendering image is generated based on the voxel features of each sampling point. Among them, the voxel features of each sampling point include the voxel density and voxel color value information of each sampling point. Based on the voxel density and voxel color value, the voxel color of each sampling point can be determined, and then volume rendering is performed based on the voxel color to obtain a volume rendering image.

[0065] It can be understood that when determining the sampling points in the projection direction based on the explicit density distribution matrix, the sampling points distributed at the positions with larger voxel density can be made denser, and the sampling points at the positions with smaller voxel density can be made sparser, thus completing the sampling process.

[0066] It can be seen that the embodiment of the present invention omits the first sampling and the calculation process based on the first sampling result in the traditional method, greatly reducing the calculation amount and improving the efficiency of volume rendering.

[0067] After obtaining the volume rendering image, since the neural radiance field has not been trained at this time, the obtained volume rendering image is inaccurate. Therefore, the embodiment of the present invention compares these volume rendering images with the same angles as the multi-angle target scene map with the real input image (multi-angle target scene map) to obtain a loss function, and converges this loss function to train the neural radiance field. After the training is completed, an image rendering model is obtained, and this image rendering model can be used to generate more scene rendering images with different angles. Among them, the loss function of the volume rendering model can be calculated based on the variance between the volume rendering image and the multi-angle target scene map. The embodiment of the present invention does not make specific limitations on this.

[0068] In addition, the volume rendering model after the training is completed samples based on the display density distribution matrix, omitting the first sampling and the calculation process based on the first sampling result in the traditional method, greatly reducing the calculation amount, improving the efficiency of volume rendering, and further improving the training efficiency of the image rendering model.

[0069] The image rendering model training method provided by the embodiments of the present invention projects and reconstructs multi-angle target scene graphs through a volume rendering model to obtain an explicit density distribution matrix. After determining the sampling points in the projection direction based on the explicit density distribution matrix, a volume rendering image is generated based on the voxel features of each sampling point. Since the explicit density distribution matrix can represent the 3D scene density of the scene to be rendered, the sampling points can be quickly and directly determined based on the explicit density distribution matrix, avoiding the problem of large computational complexity and low efficiency caused by secondary sampling in traditional methods, and improving the volume rendering efficiency. Since the image rendering model is trained based on the volume rendering image and the multi-angle target scene graph, the training and inference efficiency of the image rendering model is further improved.

[0070] Based on the above embodiments, inputting the multi-angle scene graph into the volume rendering model to obtain the volume rendering image output by the volume rendering model includes:

[0071] Inputting the multi-angle target scene graph into the reconstruction layer of the volume rendering model, and the reconstruction layer projects and reconstructs the multi-angle target scene graph to obtain the explicit density distribution matrix output by the reconstruction layer;

[0072] Inputting the explicit density distribution matrix into the sampling layer of the volume rendering model, and the sampling layer determines the sampling points in the projection direction based on the explicit density distribution matrix to obtain the voxel features of each sampling point output by the sampling layer;

[0073] Inputting the voxel features of each sampling point into the rendering layer of the volume rendering model, and the rendering layer performs volume rendering based on the initial parameters of the volume rendering model and the voxel features of each sampling point to obtain the volume rendering image output by the rendering layer; the volume rendering image has the same angle as the multi-angle target scene graph.

[0074] Specifically, the reconstruction layer of the volume rendering model is used to project and reconstruct the multi-angle target scene graph, so as to obtain an explicit density distribution matrix for representing the 3D scene density. Since this matrix is an explicit matrix, the 3D scene density information can be directly read from the matrix without further calculating the 3D scene density information based on the matrix.

[0075] After obtaining the explicit density distribution matrix, the density distribution information of each voxel on the target projection line in the projection direction can be determined. Then, more sampling points can be collected for the voxels with a denser density distribution, and fewer sampling points can be collected for the voxels with a sparser density distribution. Thus, the sampling points obtained by the sampling layer based on the density distribution matrix sampling can accurately represent the voxel information in the projection direction, and the voxel features of the sampling points can be obtained.

[0076] The voxel features of each sampling point can include the voxel density and voxel color value information of each sampling point. Based on the voxel density and voxel color value, the voxel color of each sampling point can be determined. Furthermore, the rendering layer can perform volume rendering based on the voxel color to obtain a volume rendering image with the same angle as the target scene graph.

[0077] It can be seen that compared with the traditional method, in the embodiment of the present invention, the sampling layer determines the sampling points of the projection direction based on the explicit density distribution matrix, obtains the voxel features of each sampling point output by the sampling layer, and then the rendering layer can perform volume rendering based on the voxel features of each sampling point to obtain a volume rendering image, eliminating the process of the first sampling and the calculation based on the results of the first sampling in the traditional method, greatly reducing the amount of calculation and improving the efficiency of volume rendering.

[0078] Based on any of the above embodiments, the sampling points are determined based on the following steps:

[0079] Read the density distribution of each voxel point on the target projection line from the explicit density distribution matrix, where the target projection line is determined based on the 3D scene corresponding to the target scene and the pixel points in the projection direction;

[0080] Determine the sampling points based on the density distribution of each voxel point.

[0081] Specifically, the explicit density distribution matrix is used to represent the 3D scene density of the scene to be rendered. Since this matrix is an explicit matrix, the 3D scene density information can be directly read from the matrix without further calculation based on the matrix to obtain the 3D scene density information.

[0082] In addition, when it is necessary to obtain the volume rendering image in the projection direction, a projection line in the 3D scene that vertically points to a pixel point in the projection direction can be determined for a pixel point in the projection direction, and then a projection line with the same angle and position in the explicit density distribution matrix is used as the target projection line. Read the density distribution of each voxel point on the target projection line, and then determine the sampling points based on the density distribution of each voxel point and perform sampling.

[0083] It can be understood that when determining the sampling points in the projection direction based on the density distribution of each voxel point, the sampling points distributed at the locations with larger voxel density are made denser, and the sampling points at the locations with smaller voxel density are made sparser, thereby completing the sampling process.

[0084] Based on any of the above embodiments, the target projection line is determined based on the following steps:

[0085] Perform normalization processing on the 3D scene;

[0086] Determine a projection line that vertically points to the pixel point in the projection direction from the normalized 3D scene as the target projection line.

[0087] Specifically, after normalizing the 3D scene, the absolute values of the data in the 3D scene can be made into a certain relative value relationship, thereby simplifying the calculation and reducing the magnitude.

[0088] After normalizing the 3D scene, a projection line of pixel points perpendicular to the projection direction is determined from the normalized 3D scene as the target projection line. Then, the density distribution of each voxel point on the target projection line can be read, and sampling points are determined and sampled based on the density distribution of each voxel point, thereby avoiding the problem of large computational complexity and time consumption caused by the need for secondary sampling in traditional methods.

[0089] Based on any of the above embodiments, the loss function of the image rendering model is determined based on the mean variance between the volume rendering image and the multi-angle target scene graph.

[0090] Specifically, the mean variance between the volume rendering image and the multi-angle target scene graph is used to characterize the difference between the volume rendering image and the target scene graph at the corresponding angle. The greater the difference, the greater the mean variance, indicating that the similarity between the volume rendering image and the target scene graph at the corresponding angle is lower, that is, the loss function value is greater; the smaller the difference, the smaller the mean square value, indicating that the similarity between the volume rendering image and the target scene graph at the corresponding angle is higher, that is, the loss function value is smaller.

[0091] Based on any of the above embodiments, the present invention provides another method for training an image rendering model, and the method includes:

[0092] A random initial value is assigned to the initial model of the volume rendering model, and then the volume rendering method can be used to perform rendering on the input multi-angle sample scene graph. The mean variance between the rendered sample rendering image and the sample scene graph at the corresponding angle is used as the cost function, and the sample density distribution matrix is iteratively reconstructed to enable the sample density distribution matrix to finally explicitly express the voxel density distribution of the 3D scene corresponding to the sample scene graph at the corresponding angle with a precision of 64×64×64.

[0093] Then, the sample density distribution matrix generated based on the multi-angle sample scene graph in the previous step is used to guide the sampling process in the training and rendering tasks of the image rendering model (such as the NeRF model). Specifically, in the sampling process, the NeRF model first needs to determine a projection line perpendicular to a pixel point in the to-be-rendered image corresponding to the projection direction from the normalized 3D scene, and then determine the sample sampling points on the projection line.

[0094] Among them, when determining the sample sampling points based on the sample density distribution matrix, first draw projection lines at the same angle and position in the sample density distribution matrix, read the density value distribution on the projection line, and then determine the sampling point position distribution on the corresponding projection line of the actual NeRF model according to the read projection line density distribution values, so that the distribution is denser at the place with a larger voxel density and sparser at the place with a smaller voxel density. In this way, a projection line sampling of the NeRF model is completed, and so on.

[0095] The NeRF model completes multiple samplings according to the method of the previous step, and then performs rendering based on the sampling results. The generated volume rendering image is used to complete the training task of the image rendering model.

[0096] Based on any of the above embodiments, the present invention provides an image rendering method, as Figure 2 shown, the method includes:

[0097] Step 210, determining the target angle of the rendering image corresponding to the target scene;

[0098] Step 220, obtaining the rendering image corresponding to the target angle based on the image rendering model;

[0099] Among them, the image rendering model is trained based on the image rendering model training method described in any of the above embodiments.

[0100] Specifically, after obtaining the volume rendering image based on the volume rendering model, since the neural radiance field has not been trained at this time, the obtained volume rendering image is inaccurate. Therefore, in the embodiments of the present invention, the loss function is obtained by comparing these volume rendering images with the same angles as the multi-angle target scene images with the real input images (multi-angle target scene images), and the loss function is converged to train the neural radiance field. After the training is completed, the image rendering model is obtained.

[0101] After obtaining the trained image rendering model, rendering images at any angle can be generated based on the image rendering model; the target angle here can be any angle.

[0102] Next, the image rendering model training device provided by the present invention will be described. The image rendering model training device described below can be mutually corresponding and referred to with the image rendering model training method described above.

[0103] Based on any of the above embodiments, the present invention provides an image rendering model training device, as Figure 3 shown, the device includes:

[0104] The first determination unit 310 is used to determine the multi-angle target scene images of the target scene;

[0105] A volume rendering unit 320, configured to input the multi-angle target scene graph into a volume rendering model to obtain a volume rendering image output by the volume rendering model;

[0106] A model training unit 330, configured to train an initialized neural radiance field with implicit scene representation ability based on the volume rendering image and the multi-angle target scene graph to obtain an image rendering model;

[0107] Wherein, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix for characterizing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features containing density and color values in each sampling point.

[0108] Based on any of the above embodiments, the volume rendering unit 320 includes:

[0109] A reconstruction unit, configured to input the multi-angle target scene graph into a reconstruction layer of the volume rendering model, and the reconstruction layer performs projection reconstruction on the multi-angle target scene graph to obtain the explicit density distribution matrix output by the reconstruction layer;

[0110] A sampling unit, configured to input the explicit density distribution matrix into a sampling layer of the volume rendering model, and the sampling layer determines sampling points in the projection direction based on the explicit density distribution matrix to obtain voxel features of each sampling point output by the sampling layer;

[0111] A rendering unit, configured to input the voxel features of each sampling point into a rendering layer of the volume rendering model, and the rendering layer performs volume rendering based on initial parameters of the volume rendering model and the voxel features of each sampling point to obtain the volume rendering image output by the rendering layer; the volume rendering image has the same angle as the multi-angle target scene graph.

[0112] Based on any of the above embodiments, the apparatus further includes:

[0113] A reading unit, configured to read density gradations of voxel points on a target projection line from the explicit density distribution matrix, where the target projection line is determined based on a 3D scene corresponding to the target scene and pixel points in the projection direction;

[0114] A first determination unit, configured to determine the sampling points based on the density distribution of each voxel point.

[0115] Based on any of the above embodiments, the apparatus further includes:

[0116] A processing unit for normalizing the 3D scene;

[0117] A second determination unit for determining a projection line of pixel points vertically pointing in the projection direction from the normalized 3D scene as the target projection line.

[0118] Based on any of the above embodiments, the loss function of the image rendering model is determined based on the mean variance between the volume rendering image and the multi-angle target scene graph.

[0119] Based on any of the above embodiments, the present invention further provides an image rendering device, as Figure 4 shown, the device includes:

[0120] A second determination unit 410 for determining the target angle of the rendering image corresponding to the target scene;

[0121] An image rendering unit 420 for obtaining the rendering image corresponding to the target angle based on the image rendering model;

[0122] wherein, the image rendering model is trained based on the image rendering model training method described in any of the above embodiments.

[0123] Figure 5 is a schematic structural diagram of an electronic device provided by the present invention, as Figure 5 shown, the electronic device may include: a processor 510, a memory 520, a communication interface 530, and a communication bus 540. Among them, the processor 510, the memory 520, and the communication interface 530 complete mutual communication through the communication bus 540. The processor 510 can call the logical instructions in the memory 520 to execute the image rendering model training method, and the method includes: determining a multi-angle target scene graph of the target scene; inputting the multi-angle target scene graph into a volume rendering model to obtain a volume rendering image output by the volume rendering model; training an initialized neural radiance field with implicit scene expression ability based on the volume rendering image and the multi-angle target scene graph to obtain an image rendering model; wherein, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix representing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features containing density and color values in each sampling point.

[0124] and / or, to execute an image rendering method, which includes: determining a multi-angle scene graph of a scene to be rendered; inputting the multi-angle scene graph into an image rendering model to obtain a target-angle rendered image output by the image rendering model; wherein, the image rendering model is trained based on the image rendering model training method as described above.

[0125] In addition, when the logical instructions in the above-mentioned memory 520 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0126] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the

[0127] image rendering model training method, which includes: determining a multi-angle target scene graph of a target scene; inputting the multi-angle target scene graph into a volume rendering model to obtain a volume rendering image output by the volume rendering model; training an initialized neural radiance field with implicit scene representation ability based on the volume rendering image and the multi-angle target scene graph to obtain an image rendering model; wherein, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix representing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features containing density and color values in each sampling point.

[0128] And / or, to execute an image rendering method, the method includes: determining a multi-angle scene graph of a scene to be rendered; inputting the multi-angle scene graph into an image rendering model to obtain a target-angle rendered image output by the image rendering model; wherein, the image rendering model is trained based on the image rendering model training method as described above.

[0129] In another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to execute each of the above-provided

[0130] Image rendering model training method, the method includes: determining a multi-angle target scene graph of a target scene; inputting the multi-angle target scene graph into a volume rendering model to obtain a volume rendered image output by the volume rendering model; training an initialized neural radiance field with implicit scene representation ability based on the volume rendered image and the multi-angle target scene graph to obtain an image rendering model; wherein, the volume rendering model is trained based on multi-angle sample scene graphs; the volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix for characterizing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendered image based on the voxel features containing density and color values in each sampling point.

[0131] And / or, to execute an image rendering method, the method includes: determining a multi-angle scene graph of a scene to be rendered; inputting the multi-angle scene graph into an image rendering model to obtain a target-angle rendered image output by the image rendering model; wherein, the image rendering model is trained based on the image rendering model training method as described above.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an image rendering model, characterized in that, Including: Determining a multi - perspective target scene graph of a target scene; Inputting the multi - perspective target scene graph into a volume rendering model to obtain a volume rendering image output by the volume rendering model; Training an initialized neural radiance field with implicit scene representation ability based on the volume rendering image and the multi - perspective target scene graph to obtain an image rendering model; Wherein, the volume rendering model is trained based on multi - perspective sample scene graphs; The volume rendering model first performs projection reconstruction on the multi - perspective target scene graph to obtain an explicit density distribution matrix representing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features containing density and color values in each sampling point.

2. The method for training an image rendering model according to claim 1, wherein The step of inputting the multi - perspective target scene graph into the volume rendering model to obtain a volume rendering image output by the volume rendering model includes: Inputting the multi - perspective target scene graph into the reconstruction layer of the volume rendering model, and the reconstruction layer performs projection reconstruction on the multi - perspective target scene graph to obtain the explicit density distribution matrix output by the reconstruction layer; Inputting the explicit density distribution matrix into the sampling layer of the volume rendering model, and the sampling layer determines sampling points in the projection direction based on the explicit density distribution matrix to obtain voxel features of each sampling point output by the sampling layer; Inputting the voxel features of each sampling point into the rendering layer of the volume rendering model, and the rendering layer performs volume rendering based on the initial parameters of the volume rendering model and the voxel features of each sampling point to obtain the volume rendering image output by the rendering layer; the volume rendering image has the same angle as the multi - perspective target scene graph.

3. The method for training an image rendering model according to claim 1, wherein The sampling points are determined based on the following steps: Reading the density distribution of each voxel point on the target projection line from the explicit density distribution matrix, and the target projection line is determined based on the 3D scene corresponding to the target scene and the pixel points in the projection direction; Determining the sampling points based on the density distribution of each voxel point.

4. The method for training an image rendering model according to claim 3, wherein The target projection line is determined based on the following steps: Performing normalization processing on the 3D scene; Determining a projection line perpendicular to the pixel points in the projection direction from the normalized 3D scene as the target projection line.

5. The method for training an image rendering model according to any one of claims 1 to 4, characterized in that, The loss function of the image rendering model is determined based on the mean variance between the volume rendering image and the multi - perspective target scene graph.

6. An image rendering method, characterized in that, Including: Determining a target angle of a rendering image corresponding to a target scene; Obtaining a rendering image corresponding to the target angle based on the image rendering model; Wherein, the image rendering model is trained by the image rendering model training method according to any one of claims 1 to 5.

7. An image rendering model training device, characterized in that Including: A first determination unit for determining a multi - perspective target scene graph of a target scene; A volume rendering unit for inputting the multi - perspective target scene graph into a volume rendering model to obtain a volume rendering image output by the volume rendering model; A model training unit for training an initialized neural radiance field with implicit scene representation ability based on the volume rendering image and the multi-angle target scene graph to obtain an image rendering model; wherein, the volume rendering model is trained based on the multi-angle sample scene graph; The volume rendering model first performs projection reconstruction on the multi-angle target scene graph to obtain an explicit density distribution matrix for characterizing the 3D scene density of the target scene, and after determining sampling points in the projection direction based on the density distribution matrix, generates the volume rendering image based on the voxel features including density and color value in each sampling point.

8. An image rendering device, characterized in that, It includes: A second determination unit for determining the target angle of the rendering image corresponding to the target scene; An image rendering unit for obtaining the rendering image corresponding to the target angle based on the image rendering model; wherein, the image rendering model is trained based on the image rendering model training method according to any one of claims 1 to 5.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the image rendering model training method according to any one of claims 1 to 5, and / or, when the processor executes the program, it implements the steps of the image rendering method according to claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the image rendering model training method according to any one of claims 1 to 5, and / or, when the computer program is executed by the processor, it implements the steps of the image rendering method according to claim 6.

Images