Neural rendering model training method, rendering method and electronic equipment

Abstract

The invention discloses a neural rendering model training method, a rendering method and electronic equipment. The neural rendering model training method comprises the following steps: acquiring a sample image collected in a target scene and a camera pose for shooting the sample image; and for each sample image, based on the sample image and the pose of the camera for shooting the sample image, obtaining the coordinate and the light direction of a light sampling point of the sample image. And according to a distance norm from the light sampling point of the sample image to the camera pose center of the target scene, converting the coordinate of the light sampling point into a coordinate in a normalized space corresponding to the target scene, so as to determine the target scene based on the sample image, the light direction of the light sampling point of the sample image and the coordinate of the normalized space. And training the neural rendering model. The three-dimensional space corresponding to the target scene is mapped into the normalized space, so that even if the target scene exceeds the camera pose space, the neural rendering model can be trained by using the coordinates of the normalized space, and the rendering effect of the target scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rendering, in particular to a neural rendering model training method, a rendering method and an electronic device. BACKGROUND

[0002] NeRF-based 3D rendering is a technology that uses a neural radiance field (NeRFs) model to generate realistic three-dimensional scenes and objects. NeRFs combines computer graphics and deep learning techniques to achieve high-quality rendering effects by modeling the implicit function of the scene. This novel rendering method has received widespread attention in recent years and has shown great potential in virtual reality, augmented reality and other fields.

[0003] The present application inventors found during the research on the NeRFs model that if the object to be rendered is located in the camera pose space, the existing NeRFs model can achieve better rendering effect, if the object to be rendered exceeds the camera pose space, such as the object to be rendered includes a foreground region and a background region, the foreground region is located in the camera pose space, but the background region exceeds the camera pose space, and this kind of scene often has the problem of inconsistent lighting, which will cause the background region of the object rendered by the existing NeRFs model to appear blurred and unclear. SUMMARY

[0004] Therefore, the present application provides a neural rendering model training method, a rendering method and an electronic device to improve the rendering effect of the object to be rendered.

[0005] In a first aspect, the present application provides a neural rendering model training method, which comprises:

[0006] obtaining a sample image collected in a target scene and a camera pose for shooting the sample image, the sample image being more than two;

[0007] for each sample image, obtaining the coordinates and light direction of the light sampling points of the sample image based on the sample image and the camera pose for shooting the sample image;

[0008] transforming the coordinates of the light sampling points of the sample image into coordinates in the corresponding normalized space of the target scene according to the distance norm of the light sampling points of the sample image to the center of the camera pose of the target scene;

[0009] training a neural rendering model based on the sample image, the light direction and the normalized space coordinates of the light sampling points of the sample image.

[0010] In a possible implementation, the training of the neural rendering model based on the sample image and the light direction and the coordinate of the normalized space of the light sampling point of the sample image specifically includes:

[0011] For each sample image, the embedding layer of the neural rendering model obtains appearance encoding of the sample image, the appearance encoding representing illumination changes of the sample image, and the appearance encoding being used for supervision of the pixel color value predicted by the neural rendering model and the true value color value, so that the neural rendering model sufficiently learns the influence of illumination changes of different environments (such as morning, noon, evening, rainy day, sunny day, etc.) on color;

[0012] The multilayer perceptron of the neural rendering model is trained based on the appearance encoding of the sample image and the light direction and the coordinate of the normalized space of the light sampling point of the sample image.

[0013] In a possible implementation, the training of the multilayer perceptron of the neural rendering model based on the appearance encoding of the sample image and the light direction and the coordinate of the normalized space of the light sampling point of the sample image specifically includes:

[0014] The feature of the light sampling point is obtained through linear interpolation based on the light direction of the light sampling point of the sample image, the coordinate of the normalized space, and the feature of the vertex of the hash voxel grid corresponding to the normalized space.

[0015] The multilayer perceptron of the neural rendering model is trained based on the appearance encoding of the sample image and the feature of the light sampling point of the sample image.

[0016] In a possible implementation, the training of the neural rendering model based on the sample image and the light direction and the coordinate of the normalized space of the light sampling point of the sample image specifically includes:

[0017] The feature of the light sampling point is obtained through linear interpolation based on the light direction of the light sampling point of the sample image, the coordinate of the normalized space, and the feature of the vertex of the hash voxel grid corresponding to the normalized space.

[0018] The multilayer perceptron of the neural rendering model is trained based on the feature of the light sampling point of the sample image.

[0019] In a possible implementation, the normalized space includes a 0-1 space and a 1-2 space, and the transformation of the coordinate of the light sampling point into the coordinate in the normalized space corresponding to the target scene according to the distance norm of the light sampling point of the sample image to the camera pose center of the target scene specifically includes:

[0020] if the distance norm of the ray sampling point of the sample image to the camera pose center of the target scene is less than or equal to 1, the coordinates of the ray sampling point are determined as the coordinates of the corresponding 0-1 space of the target scene;

[0021] if the distance norm of the ray sampling point of the sample image to the camera pose center of the target scene is greater than 1, the coordinates of the ray sampling point are nonlinearly transformed and mapped to the corresponding 1-2 space of the target scene.

[0022] In a possible implementation, the obtaining, based on the sample image and the camera pose for shooting the sample image, of the coordinates and the ray direction of the ray sampling point of the sample image specifically includes:

[0023] generating, based on the sample image and the camera pose for shooting the sample image, a ray from the camera to a pixel point in the sample image, uniformly sampling on the ray to obtain the coordinates and the ray direction of the ray sampling point of the sample image.

[0024] In a second aspect, the present application provides a rendering method, the method comprising:

[0025] obtaining the size and the camera pose of a to-be-generated image of a target scene;

[0026] generating, based on the size and the camera pose of the to-be-generated image, a ray from the camera to each pixel point in the to-be-generated image, uniformly sampling on the ray to obtain the coordinates and the ray direction of the ray sampling point of the to-be-generated image;

[0027] the pre-trained neural rendering model integrates the volume density and the color predicted by the neural rendering model for the ray sampling point based on the coordinates and the ray direction of the ray sampling point of the to-be-generated image to obtain the pixel color value of the to-be-generated image; wherein the neural rendering model is trained based on the method of any one of the implementation manners of the first aspect.

[0028] In a third aspect, the present application provides a training device of a neural rendering model, the device comprising:

[0029] a first obtaining unit configured to obtain a sample image collected in a target scene and a camera pose for shooting the sample image, the sample image being more than two;

[0030] a second obtaining unit configured to, for each sample image, obtain, based on the sample image and the camera pose for shooting the sample image, the coordinates and the ray direction of the ray sampling point of the sample image;

[0031] The transformation unit is used to transform the coordinates of the light sampling points into coordinates in the normalized space corresponding to the target scene based on the distance norm from the light sampling points of the sample image to the camera pose center of the target scene.

[0032] The training unit is used to train the neural rendering model based on the sample image, the ray direction of the ray sampling points of the sample image, and the coordinates in the normalized space.

[0033] Fourthly, this application provides a rendering apparatus, the apparatus comprising:

[0034] The third acquisition unit is used to acquire the size and camera pose of the image to be generated in the target scene;

[0035] The generation unit is used to generate light rays from the camera to each pixel in the image to be generated based on the size of the image to be generated and the camera pose, and to uniformly sample the light rays to obtain the coordinates and direction of the light ray sampling points in the image to be generated.

[0036] The rendering unit is used to integrate the volume density and color predicted by the light sampling points of the image to be generated based on the coordinates and light direction of the light sampling points to obtain the pixel color value of the image to be generated; wherein, the neural rendering model is trained based on the method described in any of the implementations of the first aspect above.

[0037] Fifthly, this application provides an electronic device, the device comprising: a memory and a processor;

[0038] The memory is used to store the relevant program code;

[0039] The processor is used to call the program code to execute the method described in either the first or second aspect.

[0040] Sixthly, this application provides a computer-readable storage medium for storing a computer program for performing the method described in any one of the first or second aspects above.

[0041] In a seventh aspect, this application provides a computer program product, which includes a computer program / instruction that, when executed by a processor, implements the method described in either the first or second aspect above.

[0042] Therefore, this application has the following beneficial effects:

[0043] To address the issue of background blurring and unclear rendering when rendering objects outside the camera pose space using existing neural rendering models, this application differs from existing neural rendering model training techniques. After obtaining ray sampling points from sample images, this application transforms the coordinates of these ray sampling points into coordinates in the normalized space corresponding to the target scene based on the distance norm from these ray sampling points to the camera pose center of the target scene. Since the normalized space is a bounded space, this transformation compresses the ray sampling points of the objects outside the camera pose space into the bounded space. Training the neural rendering model using coordinates in the normalized space allows the model to more fully learn the features of the target scene that extend beyond the camera pose space, thus improving the rendering effect of the target scene. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments provided in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0045] Figure 1 A flowchart illustrating a training method for a neural rendering model provided in an embodiment of this application;

[0046] Figure 2a A schematic diagram of a target scene space provided for an embodiment of this application;

[0047] Figure 2b A schematic diagram of a normalized space provided for an embodiment of this application;

[0048] Figure 2c A schematic diagram of another normalized space provided for an embodiment of this application;

[0049] Figure 3 A flowchart illustrating a rendering method provided in an embodiment of this application;

[0050] Figure 4 A schematic diagram of a training device for a neural rendering model provided in an embodiment of this application;

[0051] Figure 5 A schematic diagram of a rendering apparatus provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely exemplary implementations of this application and not all implementation methods. Those skilled in the art can obtain other embodiments in conjunction with the embodiments of this application without creative effort, and these embodiments are also within the protection scope of this application.

[0054] NeRF-based 3D rendering is a technique that uses NeRFs to generate realistic 3D scenes and objects. NeRFs combine computer graphics and deep learning techniques, achieving high-quality rendering results by modeling the implicit functions of the scene. This novel rendering method has received widespread attention in recent years and has shown great potential in fields such as virtual reality and augmented reality.

[0055] In the process of researching NeRFs models, the inventors of this application discovered that if the object to be rendered is located in the camera pose space, existing NeRFs models can achieve better rendering results. However, if the object to be rendered exceeds the camera pose space, such as when a drone takes an image of a building while circling it, the drone cannot get very close to the building. Therefore, the image taken by the drone not only contains the building but also other objects at a certain distance from it. For NeRFs models, the drone captures a large, boundless scene. In this scene, since the building is relatively close to the drone, it belongs to the foreground region of the image. Other objects besides the building belong to the background region of the image. The foreground region, i.e., the building, is usually located in the camera pose space, which can be understood as the flight area formed when the drone circles the building. It is understandable that the background region exceeds the camera pose space. Furthermore, drones are usually taken outdoors, and images taken from different angles outdoors often have inconsistent lighting. If existing NeRFs models are used to render scene images taken by drones, the background will be blurred and unclear.

[0056] Based on this, this application provides a method for training a neural rendering model to improve the rendering effect of the object to be rendered. Specifically, after obtaining the ray sampling points of the sample image, the coordinates of the ray sampling points are transformed into coordinates in the normalized space corresponding to the target scene based on the distance norm from the ray sampling points of the sample image to the camera pose center of the target scene. Since the normalized space is a bounded space, transforming the coordinates of the ray sampling points into coordinates in the normalized space corresponding to the target scene compresses the ray sampling points of the object to be rendered in the target scene that are outside the camera pose space into the bounded space. Training the neural rendering model using coordinates in the normalized space enables the neural rendering model to more fully learn the features of the target scene that are outside the camera pose space, thus improving the rendering effect of the target scene.

[0057] To facilitate understanding of the technical solutions provided in the embodiments of this application, the technical solutions will be described below in conjunction with specific accompanying drawings.

[0058] See Figure 1 The flowchart below illustrates a training method for a neural rendering model provided in this application embodiment. The method may include the following steps:

[0059] S101: Acquire the sample images captured in the target scene and the camera pose (position and orientation) of the captured sample images.

[0060] The target scene can be any area in the real world, such as a scenic spot or a street. Sample images are typically captured (collected) by drones or other aircraft equipped with visual sensors within the target scene. When capturing sample images using a drone, multiple sample images can be taken, and the camera poses for these images can be different. For example, if the target scene is a scenic spot with a wooden pagoda as its distinctive building, a drone can be used to fly around the pagoda and take images from different positions using different or the same pose. The objective of this application is to train a neural rendering model using sample images captured from this target scene. This allows the neural rendering model to fully learn the features of the wooden pagoda and its surrounding environment within the scenic spot. Furthermore, the trained neural rendering model can render an image containing the wooden pagoda and its surrounding environment based on any camera pose, with the surrounding environment clearly visible in the image.

[0061] S102: For each sample image, based on the sample image and the camera pose at which the sample image was captured, obtain the coordinates of the light sampling points and the light direction of the light rays in the sample image.

[0062] Specifically, for each pixel in the sample image, a ray of light from the camera position to that pixel can be generated, and the ray can be uniformly sampled to obtain multiple ray sampling points. Specific implementation methods for ray sampling points can be found in existing technologies and will not be elaborated here.

[0063] S103: Based on the distance norm from the ray sampling point in the sample image to the camera pose center of the target scene, transform the coordinates of the ray sampling point into coordinates in the normalized space corresponding to the target scene.

[0064] As mentioned above, images captured by drones or other aircraft contain background regions that extend beyond the camera pose space. In order to move the foreground and background regions of the target scene captured by the drone from an unbounded scene into a bounded space, this application utilizes scene contraction to perform spatial contraction. Specifically, based on the distance norm from the light sampling points of the sample image to the camera pose center of the target scene, the coordinates of the light sampling points of the target scene are transformed into coordinates in a normalized space.

[0065] The camera pose center can be calculated from the camera pose of the sample image. The normalization space of this application can be divided into 0-1 space and 1-2 space. The coordinates of the light sampling points of the target scene can be transformed into coordinates in 0-1 space and 1-2 space according to the distance norm from the light sampling points of the sample image to the camera pose center of the target scene.

[0066] In practice, if the distance norm from the ray sampling point in the sample image to the camera pose center of the target scene is less than or equal to 1, the coordinates of the ray sampling point can be determined as the coordinates in the 0-1 space corresponding to the target scene, that is, the three-dimensional coordinates of the ray sampling point are not changed. If the distance norm from the ray sampling point in the sample image to the camera pose center of the target scene is greater than 1, it indicates that the space where the ray sampling point is located may exceed the camera pose space, and the coordinates of the ray sampling point can be non-linearly transformed to map to the 1-2 space corresponding to the target scene.

[0067] In one possible implementation, the coordinates of the ray sampling points in the target scene are transformed into coordinates in 0-1 space and 1-2 space, which can be expressed by the following formula, where x represents the coordinates of the ray sampling point, and ||x|| represents the distance norm from the ray sampling point to the camera pose center:

[0068]

[0069] See Figure 2a The diagram shown is a schematic representation of the target scene space provided in an embodiment of this application. The target scene contains light sampling points that extend beyond the camera pose space. Figure 2bBased on the L-2 norm, the distance norm between the ray sampling point and the camera pose center is calculated. The L-2 norm represents the Euclidean distance between the ray sampling point and the camera pose center. Based on the L-2 norm, the ray sampling point of the target scene will shrink into a sphere, that is, the normalized space is a bounded spherical space.

[0070] In one possible implementation, the L-∞ norm between the ray sampling point and the camera pose center can also be calculated. That is, based on the coordinate difference between the ray sampling point and the camera pose center, a vector is determined, and then the maximum value of the elements in the vector is the L-∞ norm. (See [link to relevant documentation]). Figure 2c As shown, the light sampling points of the target scene will shrink into a cube, meaning that the normalized space is a bounded cubic space.

[0071] S104: Train the neural rendering model based on the sample image, the ray direction of the ray sampling point of the sample image, and the coordinates of the normalized space.

[0072] After transforming the ray sampling points of the sample images in the target scene into coordinates in a normalized space, the neural rendering model can be trained using the ray direction of the ray sampling points and their coordinates in the normalized space. This neural rendering model can be a NeRF (Neural Receptive Fields) model, which includes a multilayer perceptron. The training process mainly includes the following steps:

[0073] A1: For any ray sampling point, input the normalized space coordinates of the ray sampling point and the ray direction into the neural rendering model, and output the color information and volume density of the ray sampling point. The volume density represents the degree to which light is absorbed or scattered when passing through the ray sampling point.

[0074] A2: Use volume rendering to process the color information and volume density of multiple ray sampling points in the same ray direction to obtain the pixel color value of the predicted pixel, which is the pixel on the ray where the ray sampling point is located.

[0075] Specifically, for each pixel, after determining multiple ray sampling points on the ray from the camera to that pixel, the volume density and color predicted by these ray sampling points are integrated, where the weight of each ray sampling point is determined by the volume density, thereby obtaining the predicted pixel color value for that pixel. For example, volume rendering can use the following formula:

[0076] C(r)=∫ r T(s)σ(s)c(s)ds

[0077] Where r represents light ray r, s represents light ray sampling point s, C(r) represents the pixel color value rendered along the light ray direction, T(s) represents the transmittance of the medium from the camera position to the light ray sampling point s, σ(s) represents the volume density of the light ray sampling point s, and c(s) represents the color information of the light ray sampling point s.

[0078] A3: Determine the loss function based on the pixel color value of the pixel and the true color value of the pixel in the sample image.

[0079] After obtaining the pixel color value corresponding to a pixel using the volume rendering formula described above, a loss function can be determined based on the pixel color value and the ground truth color value of that pixel in the sample image. This loss function represents the error between the pixel color value and the ground truth color value. The larger the value of the loss function, the greater the error between the pixel color value and the ground truth color value, indicating that the parameters of the neural rendering model are less accurate.

[0080] A4: When the loss function is greater than or equal to the preset value, adjust the parameters of the neural rendering model based on the loss function, and re-execute the input of the normalized space coordinates of the light sampling points and the light direction into the neural rendering model and the subsequent training process until the determined loss function is less than the preset value, and obtain the trained neural rendering model.

[0081] When the loss function is greater than or equal to a preset value, it indicates that the error between the pixel color value and the true color value is large, and the accuracy of the neural rendering model does not meet the requirements. In this case, the parameters of the neural rendering model can be adjusted based on the loss function. After adjusting the parameters, the coordinates of the ray sampling points and the ray direction are input into the neural rendering model again, and the subsequent training process is repeated until the resulting loss function is less than the preset value. This indicates that the error between the pixel color value and the true color value meets the requirements, and a well-trained neural rendering model can be obtained. When adjusting the parameters of the neural rendering model, backpropagation algorithms and gradient descent can be used to adjust the parameters, making the output pixel color value closer to the true color value, thus obtaining a well-trained neural rendering model.

[0082] After uniformly sampling light rays to obtain multiple ray sampling points, each ray sampling point needs to have its features extracted by the neural rendering model, resulting in a large computational load during training. To reduce computational complexity, one possible implementation is to convert the normalized space of the target scene into a hash voxel grid. Based on the coordinates of the ray sampling points in the normalized space and the features of the vertices of the hash voxel grid, the features of each ray sampling point are obtained by linear interpolation of the vertex features. Then, the neural rendering model is trained based on the features of the ray sampling points in the sample image.

[0083] In practical implementation, the normalized space can be divided into a hash voxel grid composed of multiple cubes. For each cube, each of its eight vertices corresponds to an index, which can be obtained using a hash algorithm. Furthermore, each vertex corresponds to a density feature, which can be obtained based on existing multi-resolution hash encoding algorithms, and will not be elaborated upon here.

[0084] After dividing the normalized space into a hash voxel grid, the coordinates of the vertices of each cube can be obtained. For any ray sampling point, the target cube to which the ray sampling point belongs in the hash voxel grid can be determined based on the distance between the ray sampling point and the vertices of each cube. Then, linear interpolation is performed based on the features of the eight vertices of the target cube to determine the features of the ray sampling point.

[0085] Based on this, embodiments of this application provide a structure for a neural rendering model, which may include a hash encoding model, a first multilayer perceptron, and a second multilayer perceptron. Multilayer Perceptron (MLP)

[0086] The encoding model can determine the density features of light sampling points by linear interpolation based on the density features of the vertices of the cube in the hash voxel grid. The first multilayer perceptron can decode the density features of the light sampling points to obtain their density information. The second multilayer perceptron can learn the light direction and density information of the light sampling points and output their color information.

[0087] By utilizing the sparse structure of hash voxel grids as the scene representation of the neural rendering model, the features of ray sampling points are obtained by interpolating the features of vertices. This avoids the need for each ray sampling point to go through multiple perceptrons to extract features, which can reduce feature extraction time and improve training speed.

[0088] In practical applications, sample images may be affected by changes in lighting, weather conditions, etc., and the sample images obtained will not be exactly the same. In order to further improve the accuracy of the neural rendering model, one possible implementation is to introduce appearance encoding of the sample images to represent the lighting changes of the sample images and supervise the pixel color values ​​predicted by the neural rendering model and the ground truth color values.

[0089] Optionally, an embedding layer can be introduced into the neural rendering model. When a sample image is input into the embedding layer of the neural rendering model, the appearance code of the sample image can be obtained. That is, in the same sample image, the appearance code corresponding to each pixel and ray sampling point is the same. Subsequently, the multilayer perceptron of the neural rendering model can be trained based on the appearance code of the sample image, the ray direction of the ray sampling point, and the coordinates of the normalized space. This allows the neural rendering model to learn more information from the sample image during training, resulting in a more accurate neural rendering model. The embedding layer can be based on the Generative Latent Optimization (GLO) algorithm to learn the illumination change features in the sample image and generate the appearance code. The GLO algorithm can refer to existing technologies and will not be described in detail here.

[0090] In one possible implementation, the scene representation of the hash voxel grid and the appearance encoding of the sample images can be combined to train the neural rendering model. Based on the ray direction of the ray sampling points in the sample images, their coordinates in normalized space, and the features of the vertices of the corresponding hash voxel grid in normalized space, the features of the ray sampling points are obtained through linear interpolation. Then, based on the appearance encoding of the sample images and the features of the ray sampling points, the multilayer perceptron of the neural rendering model is trained.

[0091] The method described in this application compresses the light sampling points of the objects to be rendered in the target scene that extend beyond the camera pose space into a bounded space (normalized space). Training the neural rendering model using coordinates in this normalized space allows the model to learn more fully the features of the target scene that extend beyond the camera pose space, thus improving the rendering effect. Furthermore, the influence of different lighting factors can be considered during the training of the neural rendering model, making the trained model more accurate and improving its rendering performance.

[0092] Based on the above method embodiments, this application provides a rendering method. See also... Figure 3 The diagram shows a flowchart of a rendering method provided in an embodiment of this application. This method can be executed by a device with rendering capabilities. The processing device that trains the neural rendering model and the device with rendering capabilities can be the same device or different devices. When they are different devices, after the processing device trains the neural rendering model, it can send the neural rendering model to the device with rendering capabilities and deploy it. This allows the device with rendering capabilities to call the pre-deployed neural rendering model for rendering when rendering the target scene.

[0093] The method may include the following steps:

[0094] S301: Obtain the dimensions and camera pose of the image to be generated for the target scene.

[0095] The size of the image to be generated can be expressed in pixels, such as 8256×5504 pixels, and the camera pose can be any value. Since the user's eye views the image from the position of the camera, the camera pose in step S301 represents the position of the user's eye and the viewing angle of the image.

[0096] S302: Based on the size of the image to be generated and the camera pose, generate light rays from the camera to each pixel in the image to be generated, perform uniform sampling on the light rays, and obtain the coordinates of the light ray sampling points and the light ray direction in the image to be generated.

[0097] For each pixel in the image to be generated, a ray of light from the camera to the pixel can be generated based on the camera position, and the direction of the ray can be determined. Then, uniform sampling is performed on this ray to obtain the ray sampling point corresponding to each pixel. Since the positions of the camera and the pixels are known, interpolation calculations can be performed through uniform sampling to obtain the coordinates of each ray sampling point.

[0098] S303: The pre-trained neural rendering model integrates the volume density and color predicted by the light sampling points of the image to be generated based on the coordinates and light direction of the light sampling points to obtain the pixel color value of the image to be generated.

[0099] By inputting the coordinates and ray directions of the ray sampling points in the image to be generated into a pre-trained neural rendering model, the predicted volume density and color corresponding to each ray sampling point can be obtained. For each pixel, multiple ray sampling points along the ray direction are used to integrate the predicted volume density and color corresponding to these multiple ray sampling points, i.e., volume rendering, to obtain the pixel color value corresponding to that pixel. Performing the above steps for each pixel yields the pixel color value of each pixel in the image to be generated, and the target scene is rendered based on the pixel color values ​​of each pixel.

[0100] In one possible implementation, the influence of lighting factors from different angles on the target scene can also be considered. For each image to be generated, step 301 can further obtain the lighting environment parameters of the image to be generated, such as morning, evening, noon, rainy day, sunny day, etc. The lighting environment parameters can be input by the user. Specifically, the coordinates of the light sampling points of the image to be generated, the light direction, and the lighting environment parameters of the image to be generated are input into the neural rendering model to obtain the predicted volume density and color corresponding to each light sampling point. The volume density and color of multiple light sampling points on the same light ray are integrated to obtain the pixel color value of the pixel corresponding to the light ray. As mentioned above, the neural rendering model provided in this application includes an embedding layer. The embedding layer can obtain the corresponding appearance encoding through the lighting environment parameters. The appearance encoding characterizes the influence of different ambient lighting on imaging, making the rendered target scene image more in line with the effect of the real world and giving the user an immersive experience.

[0101] In another possible viewing method, the appearance code can be directly input. The input appearance code can be a pre-set appearance code. For example, it can be the average of the appearance codes corresponding to multiple sample images during training. This application embodiment does not specifically limit the method for determining the input appearance code.

[0102] The above is a neural rendering method provided by an embodiment of this application. The neural rendering model used in this method is implemented by the method provided in this application, which can solve the problem of unbounded scenes and obtain an image of the target scene where both the foreground and background regions are clear.

[0103] Based on the above method embodiments, this application also provides a training device for a neural rendering model. See also Figure 4 The diagram shown is a schematic of a training device for a neural rendering model provided in an embodiment of this application.

[0104] The first acquisition unit 401 is used to acquire sample images collected in the target scene and the camera pose of the sample images, wherein there are two or more sample images.

[0105] The second acquisition unit 402 is used to acquire the coordinates and light direction of the light sampling points of each sample image based on the sample image and the camera pose of the sample image.

[0106] The transformation unit 403 is used to transform the coordinates of the light sampling points into coordinates in the normalized space corresponding to the target scene based on the distance norm from the light sampling points of the sample image to the camera pose center of the target scene.

[0107] Training unit 404 is used to train the neural rendering model based on the sample image, the ray direction of the ray sampling point of the sample image, and the coordinates of the normalized space.

[0108] In one possible implementation, the training unit 404 is specifically used to obtain the appearance code of the neural rendering model for each sample image, the embedding layer of the neural rendering model, the appearance code representing the illumination changes of the sample image, and the appearance code being used to supervise the pixel color values ​​predicted by the neural rendering model and the ground truth color values; and to train the multilayer perceptron of the neural rendering model based on the appearance code of the sample image and the ray direction and normalized space coordinates of the ray sampling points of the sample image.

[0109] In one possible implementation, the training unit 404 is specifically used to obtain the features of the light sampling points by linear interpolation based on the light direction of the light sampling points of the sample image, the coordinates of the normalized space, and the features of the vertices of the hash voxel grid corresponding to the normalized space; and to train the multilayer perceptron of the neural rendering model based on the appearance encoding of the sample image and the features of the light sampling points of the sample image.

[0110] In one possible implementation, the training unit 404 is specifically used to obtain the features of the light sampling points by linear interpolation based on the light direction of the light sampling points of the sample image, the coordinates in the normalized space, and the features of the vertices of the hash voxel grid corresponding to the normalized space; and to train the multilayer perceptron of the neural rendering model based on the features of the light sampling points of the sample image.

[0111] In one possible implementation, the normalization space includes a 0-1 space and a 1-2 space. The transformation unit 403 is specifically used to determine the coordinates of the ray sampling point as the coordinates of the 0-1 space corresponding to the target scene if the distance norm from the ray sampling point of the sample image to the camera pose center of the target scene is less than or equal to 1; and to perform a nonlinear transformation on the coordinates of the ray sampling point to map them to the 1-2 space corresponding to the target scene if the distance norm from the ray sampling point of the sample image to the camera pose center of the target scene is greater than 1.

[0112] In one possible implementation, the second acquisition unit 402 is specifically used to generate light rays from the camera to the pixels in the sample image based on the sample image and the camera pose of the camera that captured the sample image, and to perform uniform sampling on the light rays to obtain the coordinates and direction of the light ray sampling points in the sample image.

[0113] Furthermore, embodiments of this application also provide a rendering apparatus. See alsoFigure 5 The diagram shown is a schematic of a rendering apparatus provided in an embodiment of this application.

[0114] The device 500 includes:

[0115] The third acquisition unit 501 is used to acquire the size and camera pose of the image to be generated of the target scene;

[0116] The generation unit 502 is used to generate light rays from the camera to each pixel in the image to be generated based on the size of the image to be generated and the camera pose, and to perform uniform sampling on the light rays to obtain the coordinates and light ray direction of the light ray sampling points in the image to be generated.

[0117] The rendering unit 503 is used to integrate the volume density and color predicted by the light sampling points of the image to be generated based on the coordinates and light directions of the light sampling points of the image to be generated to obtain the pixel color value of the image to be generated; wherein, the neural rendering model is trained based on the training method of the neural rendering model described in the above method embodiment.

[0118] Based on the above method and device embodiments, this application also provides an electronic device. The following description will be provided in conjunction with the accompanying drawings.

[0119] See Figure 6 , Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application.

[0120] The device 600 includes: a memory 601 and a processor 602;

[0121] The memory 601 is used to store relevant program code;

[0122] The processor 602 is used to call the program code to execute the training method and rendering method of the neural rendering model described in the above method embodiments.

[0123] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program for executing the training and rendering methods of the neural rendering model described in the above method embodiments.

[0124] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0125] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units or modules described as separate components may or may not be physically separate. The components shown as units or modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the units or modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0128] It should also be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0129] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this application can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0130] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a neural rendering model, characterized in that, The method includes: Acquire sample images captured in the target scene and the camera poses of the sample images, wherein there are two or more sample images; For each sample image, based on the sample image and the camera pose that captured the sample image, the coordinates of the light sampling points and the light direction of the light rays in the sample image are obtained; Based on the distance norm from the ray sampling point of the sample image to the camera pose center of the target scene, the coordinates of the ray sampling point are transformed into coordinates in the normalized space corresponding to the target scene; The neural rendering model is trained based on the sample image, the ray direction of the ray sampling points of the sample image, and the coordinates in the normalized space.

2. The method according to claim 1, characterized in that, The training of the neural rendering model based on the sample image and the ray direction and normalized space coordinates of the ray sampling points in the sample image specifically includes: For each sample image, the embedding layer of the neural rendering model obtains the appearance code of the sample image. The appearance code represents the illumination changes of the sample image and is used to supervise the pixel color values ​​predicted by the neural rendering model and the ground truth color values. The multilayer perceptron of the neural rendering model is trained based on the appearance encoding of the sample image and the ray direction and normalized space coordinates of the ray sampling points of the sample image.

3. The method according to claim 2, characterized in that, The training of the multilayer perceptron of the neural rendering model based on the appearance encoding of the sample image and the ray direction and normalized space coordinates of the ray sampling points of the sample image specifically includes: Based on the ray direction of the ray sampling point in the sample image, the coordinates in the normalized space, and the features of the vertices of the hash voxel grid corresponding to the normalized space, the features of the ray sampling point are obtained through linear interpolation. The multilayer perceptron of the neural rendering model is trained based on the appearance encoding of the sample image and the features of the light sampling points of the sample image.

4. The method according to claim 1, characterized in that, The training of the neural rendering model based on the sample image and the ray direction and normalized space coordinates of the ray sampling points in the sample image specifically includes: Based on the ray direction of the ray sampling point in the sample image, the coordinates in the normalized space, and the features of the vertices of the hash voxel grid corresponding to the normalized space, the features of the ray sampling point are obtained through linear interpolation. The multilayer perceptron of the neural rendering model is trained based on the features of the light sampling points in the sample image.

5. The method according to any one of claims 1-4, characterized in that, The normalized space includes a 0-1 space and a 1-2 space. The step of transforming the coordinates of the ray sampling points into coordinates in the normalized space corresponding to the target scene based on the distance norm from the ray sampling points of the sample image to the camera pose center of the target scene specifically includes: If the distance norm from the ray sampling point of the sample image to the camera pose center of the target scene is less than or equal to 1, then the coordinates of the ray sampling point are determined as the coordinates of the target scene in the 0-1 space. If the distance norm from the ray sampling point of the sample image to the camera pose center of the target scene is greater than 1, then the coordinates of the ray sampling point are nonlinearly transformed and mapped to the 1-2 space corresponding to the target scene.

6. The method according to any one of claims 1-4, characterized in that, The step of obtaining the coordinates and ray direction of the ray sampling points in the sample image based on the sample image and the camera pose used to capture the sample image specifically includes: Based on the sample image and the camera pose that captured the sample image, light rays from the camera to the pixels in the sample image are generated, and uniform sampling is performed on the light rays to obtain the coordinates of the light sampling points and the direction of the light rays in the sample image.

7. A rendering method, characterized in that, The method includes: Obtain the dimensions and camera pose of the target scene's image to be generated; Based on the size of the image to be generated and the camera pose, light rays from the camera to each pixel of the image to be generated are generated, and uniform sampling is performed on the light rays to obtain the coordinates and direction of the light ray sampling points of the image to be generated. The pre-trained neural rendering model integrates the volume density and color predicted by the light sampling points based on the coordinates and light directions of the light sampling points of the image to be generated, and obtains the pixel color value of the image to be generated; wherein the neural rendering model is trained based on any one of the methods in claims 1-6.

8. A training device for a neural rendering model, characterized in that, The device includes: The first acquisition unit is used to acquire sample images collected in the target scene and the camera pose of the sample images, wherein there are two or more sample images. The second acquisition unit is used to acquire the coordinates and light direction of the light sampling points of each sample image based on the sample image and the camera pose of the camera that captured the sample image. The transformation unit is used to transform the coordinates of the light sampling points into coordinates in the normalized space corresponding to the target scene based on the distance norm from the light sampling points of the sample image to the camera pose center of the target scene. The training unit is used to train the neural rendering model based on the sample image, the ray direction of the ray sampling points of the sample image, and the coordinates in the normalized space.

9. A rendering apparatus, characterized in that, The device includes: The third acquisition unit is used to acquire the size and camera pose of the image to be generated in the target scene; The generation unit is used to generate light rays from the camera to each pixel in the image to be generated based on the size of the image to be generated and the camera pose, and to uniformly sample the light rays to obtain the coordinates and direction of the light ray sampling points in the image to be generated. The rendering unit is used to integrate the volume density and color predicted by the light sampling points of the image to be generated based on the coordinates and light direction of the light sampling points to obtain the pixel color value of the image to be generated; wherein the neural rendering model is trained based on the method of any one of claims 1-6.

10. An electronic device, characterized in that, The device includes: a memory and a processor; The memory is used to store the relevant program code; The processor is used to call the program code to execute the training method of the neural rendering model according to any one of claims 1 to 6 or the rendering method according to claim 7.

Images