Image rendering, feature extraction model training, rendering model training method and device

Abstract

The present disclosure provides an image rendering method and device, a feature extraction model training method and device, and a rendering model training method and device, relates to the technical field of artificial intelligence, specifically to the technical field of computer vision, augmented reality, virtual display, deep learning and the like, and can be applied to the scene of meta universe, AIGC (AI-Generated Content, using artificial intelligence to generate content) and the like. The specific implementation scheme of the image rendering method is as follows: camera pose data of a target object under multiple views is obtained; based on the camera pose data, a plurality of intermediate feature maps are determined; based on the camera pose data, a target camera pose to be rendered is determined; based on the plurality of intermediate feature maps and the target camera pose, a final feature map corresponding to the target camera pose is determined; and based on the final feature map, a final rendering image is obtained. This embodiment improves the rendering efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, augmented reality, virtual display, deep learning and the like, which can be applied to the scene of meta universe, AIGC and the like, and in particular to an image rendering method and device, a feature extraction model training method and device, a rendering model training method and device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] In image rendering, a neural radiance field model is generally used to implicitly learn a static 3D scene. In use, the neural radiance field model is generally trained using multi-view images, and a rendered image is obtained through inference of the neural radiance field model. However, in the rendering process, each ray needs to be calculated through inference, which is time-consuming and directly hinders the application of the neural radiance field model in most scenarios. SUMMARY

[0003] An image rendering method and device, a feature extraction model training method and device, a rendering model training method and device, an electronic device, a computer readable storage medium and a computer program product are provided.

[0004] According to a first aspect, an image rendering method is provided, which includes: obtaining camera pose data of a target object under multiple views; determining a plurality of intermediate feature maps based on the camera pose data; determining a target camera pose to be rendered based on the camera pose data; determining a final feature map corresponding to the target camera pose based on the plurality of intermediate feature maps and the target camera pose; and obtaining a final rendered image based on the final feature map.

[0005] According to a second aspect, a feature extraction model training method is provided, which includes: obtaining a pre-set neural radiance field network; the neural radiance field network includes a first multi-layer perceptron and a second multi-layer perceptron; the first multi-layer perceptron is used to represent the corresponding relationship between camera pose data and an intermediate feature map; the second multi-layer perceptron is used to represent the corresponding relationship between ray orientation data, an intermediate feature map and a rendered image; obtaining a pose-ray sample set, the pose-ray sample set including at least two consecutive frames of pose-ray samples of a target object under the same scene, the pose-ray sample including camera pose data and ray orientation data corresponding to the camera pose data; inputting the camera pose data in the selected pose-ray sample into the first multi-layer perceptron and inputting the ray orientation data in the selected pose-ray sample into the second multi-layer perceptron to obtain a rendered image output by the neural radiance field network; and in response to the neural radiance field network satisfying a training completion condition, obtaining a feature extraction model corresponding to the first multi-layer perceptron.

[0006] According to the third aspect, a rendering model training method is provided, which includes: acquiring a preset neural rendering network and a pose ray sample set, wherein the pose ray sample set includes: at least two consecutive frames of pose ray samples of a target object in the same scene; inputting the selected pose ray samples into a pre-trained neural radiation field model to obtain an intermediate feature map output by the neural radiation field model; projecting the intermediate feature map onto the plane position where the intermediate feature map corresponding to the target camera pose is located to obtain at least two projected feature maps; connecting all the projected feature maps in the at least two projected feature maps and inputting them into the neural rendering network to obtain a rendering map output by the neural rendering network; and obtaining a neural rendering model corresponding to the neural rendering network in response to the neural rendering network meeting the training completion conditions.

[0007] According to a fourth aspect, an image rendering apparatus is provided, comprising: a data acquisition unit configured to acquire camera pose data of a target object from multiple perspectives in multiple frames; a feature determination unit configured to determine intermediate feature maps of multiple frames based on the camera pose data; a pose determination unit configured to determine the pose of a target camera to be rendered based on the camera pose data; a final determination unit configured to determine a final feature map corresponding to the target camera pose based on the intermediate feature maps of multiple frames and the target camera pose; and a final obtaining unit configured to obtain a final rendered image based on the final feature map.

[0008] According to a fifth aspect, a feature extraction model training apparatus is provided, comprising: a network acquisition unit configured to acquire a pre-set neural radiation field network; the neural radiation field network comprising: a first multilayer perceptron and a second multilayer perceptron; wherein the first multilayer perceptron is used to characterize the correspondence between camera pose data and intermediate feature maps; the second multilayer perceptron is used to characterize the correspondence between ray orientation data, intermediate feature maps and rendered images; a sample acquisition unit configured to acquire a pose ray sample set, the pose ray sample set comprising: at least two consecutive frames of pose ray samples of a target object in the same scene, the pose ray samples comprising: camera pose data and ray orientation data corresponding to the camera pose data; an input unit configured to input the camera pose data from the selected pose ray samples into the first multilayer perceptron and input the ray orientation data from the selected pose ray samples into the second multilayer perceptron to obtain a rendered image output by the neural radiation field network; and a first obtaining unit configured to obtain a feature extraction model corresponding to the first multilayer perceptron in response to the neural radiation field network satisfying the training completion condition.

[0009] According to a sixth aspect, a rendering model training apparatus is provided, the apparatus comprising: a sample obtaining unit configured to obtain a preset neural rendering network and a pose ray sample set, the pose ray sample set comprising: at least two consecutive frame pose ray samples of a target object in a same scene; a feature extracting unit configured to input a selected pose ray sample into a pre-trained neural radiance field model to obtain an intermediate feature map output by the neural radiance field model; a projecting unit configured to project the intermediate feature map to a plane position corresponding to an intermediate feature map where a target camera pose is located to obtain at least two frame projection feature maps; a rendering unit configured to input all projection feature maps in the at least two frame projection feature maps into the neural rendering network after being connected to obtain a rendering image output by the neural rendering network; and a second obtaining unit configured to obtain a neural rendering model corresponding to the neural rendering network in response to the neural rendering network satisfying a training completion condition.

[0010] According to a seventh aspect, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any implementation manner of the first aspect.

[0011] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform the method described in any implementation manner of the first aspect.

[0012] According to a ninth aspect, a computer program product is provided, comprising a computer program, the computer program being used to implement the method described in any implementation manner of the first aspect when executed by a processor.

[0013] Embodiments of the present disclosure provide an image rendering method and device, which first obtains camera pose data of a target object under multiple views; secondly, determines a plurality of intermediate feature maps based on the camera pose data; thirdly, determines a target camera pose to be rendered based on the camera pose data; fourthly, determines a final feature map corresponding to the target camera pose based on the plurality of intermediate feature maps and the target camera pose; and finally, obtains a final rendering image based on the final feature map. Thus, the plurality of intermediate feature maps of the camera pose data are used as references for the final feature map corresponding to the target camera pose, the intermediate feature map of the target camera pose is reconstructed and refined, the calculation amount of the final rendering image corresponding to the target camera pose is reduced relative to a rendering image obtained by ray reasoning, the memory occupied by rendering is reduced, and the image rendering efficiency is effectively improved.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0016] Figure 1 This is a flowchart of an embodiment of the image rendering method according to the present disclosure;

[0017] Figure 2 This is a flowchart of an embodiment of the feature extraction model training method according to the present disclosure;

[0018] Figure 3 This is a flowchart of an embodiment of the rendering model training method according to the present disclosure;

[0019] Figure 4 This is a schematic diagram of the structure of an embodiment of the image rendering apparatus according to the present disclosure;

[0020] Figure 5 This is a schematic diagram of a structure of an embodiment of the feature extraction model training apparatus according to the present disclosure;

[0021] Figure 6 This is a schematic diagram of the structure of an embodiment of the rendering model training apparatus according to the present disclosure;

[0022] Figure 7 This is a block diagram of an electronic device used to implement the image rendering method, feature extraction model training method, and rendering model training method of the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In this embodiment, "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0025] In the prior art, when image rendering is performed by a NeRF model (Neural Radiance Fields), a rendering of a view angle that has not been learned can be rendered, and the reconstruction of a small object has a rich application scenario. However, the current use mode is generally to train the model by using multi-view images, and then obtain a rendered image by performing inference on the model. However, when the NeRF is predicted, a large amount of calculation is required for each ray, and the time consumption is very slow, which directly hinders the application of the NeRF model in most scenarios.

[0026] The present disclosure provides an image rendering method, which synthesizes a final feature map under a target camera pose by using an intermediate feature map under a multi-view, and does not need to predict ray information by using a model, thereby improving the availability of the NeRF model in more scenarios, Figure 1 A flow 100 of one embodiment of the image rendering method according to the present disclosure is shown, and the image rendering method includes the following steps:

[0027] In step 101, camera pose data under multiple views of a target object is obtained.

[0028] In this embodiment, the camera pose data is position and posture data of the target object in the same scene captured by the camera. The camera pose data can be a camera pose trajectory obtained by performing multiple view transformations on the camera when capturing the target object. The camera pose data can also be a camera pose trajectory obtained by capturing the target object at different views by using multiple cameras.

[0029] In this embodiment, the camera pose data includes a camera ID, a camera intrinsic parameter, a camera pose, and a range of the scene. Specifically, the multiple frames of camera pose data include at least two frames of camera pose, and each frame of camera pose corresponds to a camera image under a view.

[0030] In step 102, an intermediate feature map is determined based on the camera pose data.

[0031] In this embodiment, the camera pose data is used to reflect images of the target object captured by the camera under different views, and the multiple frames of camera pose data are used to reflect multiple frames of images of the target object captured by the camera under multiple views. Each frame of image in the multiple frames of images has corresponding image features, and the image features include color features, texture features, shape features, spatial features, and the like. The intermediate feature map is a set of image features, and the intermediate feature map can also be a representation of the image features on the model. Through the intermediate feature map, the model can determine the features of the image.

[0032] In this embodiment, the camera pose data corresponds to the camera images under different views, and the camera pose data of different frames reflects the camera images of different frames, and further reflects the intermediate feature maps of the camera images of different frames.

[0033] In the embodiment, the determining the plurality of intermediate feature maps based on the camera pose data comprises: determining a plurality of camera images of the target object based on the camera pose data; and extracting image features of the plurality of camera images to obtain the plurality of intermediate feature maps.

[0034] In step 103, the target camera pose to be rendered is determined based on the camera pose data.

[0035] In the embodiment, the camera pose data comprises a plurality of camera poses, and all camera poses in the camera pose data are determined, and the camera pose to be rendered is taken as the target camera pose.

[0036] In the embodiment, the target camera pose to be rendered refers to a camera pose for which a rendered image is to be generated. For the obtained camera pose data, camera poses in the camera pose data are extracted to obtain a camera pose trajectory, and if there is an intermediate feature map for which a camera pose is not determined or a rendered image is not generated in the camera pose trajectory, the camera pose for which the intermediate feature map is not determined or the rendered image is not generated is taken as the target camera pose.

[0037] In step 104, the final feature map corresponding to the target camera pose is determined based on the plurality of intermediate feature maps and the target camera pose.

[0038] In the embodiment, the view angles of each of the plurality of intermediate feature maps are different, and therefore the plurality of intermediate feature maps need to be transformed to the same view angle to obtain a plurality of intermediate feature maps under the same view angle, and then parameters of all intermediate feature maps under the same view angle are connected to obtain the final feature map corresponding to the target camera pose.

[0039] In the embodiment, the final feature map is an intermediate feature map of the target camera pose, and the final feature map is generated for the target camera pose based on parameters of intermediate feature maps of camera poses in the reference plurality of camera pose data, and the intermediate feature map of the target camera pose is optimized relative to the intermediate feature map obtained only by the target camera pose.

[0040] In step 105, the final rendered image is obtained based on the final feature map.

[0041] In the embodiment, the final feature map is an optimized intermediate feature of the target camera pose, and the rendered image generated based on the final feature map is generated in units of images, and the calculation amount is reduced and the way of obtaining the rendered image of the target camera pose is optimized relative to the rendered image of the target camera pose obtained only by the pose ray data.

[0042] In this embodiment, the final feature map can be input into a pre-trained neural rendering model to obtain a final rendering map output by the neural rendering model. The neural rendering model is used to represent the correspondence between the intermediate features and the rendering map. Inputting the final feature map as the intermediate feature map of the target camera pose into the neural rendering model can make the neural rendering model output the rendering map corresponding to the target camera pose.

[0043] The image rendering method provided by the embodiments of the present disclosure first acquires camera pose data of a target object under multiple views; secondly, determines multiple intermediate feature maps based on the camera pose data; thirdly, determines a target camera pose to be rendered based on the camera pose data; fourthly, determines a final feature map corresponding to the target camera pose based on the multiple intermediate feature maps and the target camera pose; and finally, obtains a final rendering map based on the final feature map. Thus, the multiple intermediate feature maps of the multiple camera pose data are referenced to reconstruct and refine the intermediate feature map of the target camera pose, the calculation amount of the final rendering map corresponding to the target camera pose is reduced relative to the rendering image obtained by ray reasoning, the memory occupied by rendering is reduced, and the image rendering efficiency is effectively improved.

[0044] In some optional implementations of the present embodiment, the determination of the final feature map corresponding to the target camera pose based on the multiple intermediate feature maps and the target camera pose comprises: determining an intermediate feature map corresponding to the target camera pose from the multiple intermediate feature maps; projecting each intermediate feature map in the multiple intermediate feature maps to a plane position where the intermediate feature map corresponding to the target camera pose is located to obtain multiple projection feature maps; and connecting all the projection feature maps in the multiple projection feature maps to obtain the final feature map corresponding to the target camera pose.

[0045] In the optional implementation, after obtaining the intermediate feature map corresponding to the target camera pose, the plane position where the intermediate feature map corresponding to the target camera pose is located can be determined, and the multiple intermediate feature maps can be projected to the plane position where the intermediate feature map is located, so as to unify all the views of the multiple intermediate feature maps to the view corresponding to the target camera pose.

[0046] In the optional implementation, the projection of each intermediate feature map in the multiple intermediate feature maps can be calculated by formula (1) to obtain a projection feature map corresponding to each intermediate feature map.

[0047]

[0048] In formula (1), F i is the i-th intermediate feature map in the multiple intermediate feature maps, and N is a natural number greater than 2. In formula (1), K is the camera intrinsic parameter, is a new plane position coefficient obtained according to the target camera pose, and Fi is a projection feature map of an i-th intermediate feature map in the plurality of intermediate feature maps.

[0049] In this embodiment, connecting all the projection feature maps in the plurality of projection feature maps comprises: connecting parameters of all the projection feature maps by using a connection string function (concat) to obtain the final feature map.

[0050] The final feature map obtaining method provided by this optional implementation projects the plurality of intermediate feature maps to the plane position where the intermediate feature map corresponding to the target camera pose is located, so that each frame of the plurality of intermediate feature maps provides a reference for the final feature map, thereby ensuring the richness of information of the final feature map.

[0051] Optionally, the determining, based on the plurality of intermediate feature maps and the target camera pose, of the final feature map corresponding to the target camera pose comprises: determining, from the camera pose data, camera poses corresponding to the plurality of intermediate feature maps, connecting each frame of the plurality of intermediate feature maps with the respective camera pose to obtain a plurality of processing feature maps corresponding to the plurality of intermediate feature maps; determining, from the plurality of processing feature maps, a processing feature map corresponding to the target camera pose; projecting each frame of the plurality of processing feature maps to the plane position where the processing feature map corresponding to the target camera pose is located to obtain a plurality of projection feature maps; and connecting all the projection feature maps in the plurality of projection feature maps to obtain the final feature map corresponding to the target camera pose.

[0052] In some optional implementations of this embodiment, the determining, based on the camera pose data, of the target camera pose comprises: extracting all the camera poses in the camera pose data to obtain a camera pose sequence; and selecting a camera pose to be rendered in the camera pose sequence, and taking the camera pose to be rendered as the target camera pose.

[0053] In this embodiment, the camera pose data comprises a camera identifier and a camera pose, and all the camera poses in the camera pose data can be extracted in chronological order. For the camera poses under different camera identifiers in the camera pose data, one of them can be taken, or the camera poses under different camera identifiers can be smoothed to obtain the same camera pose under different camera identifiers.

[0054] In this embodiment, the camera pose to be rendered is a camera pose for which a rendered image needs to be determined, and the camera pose to be rendered can be determined based on rendering requirements. Optionally, the camera pose to be rendered can be any one of the camera poses in the camera pose data.

[0055] In this embodiment, the camera pose sequence can be a camera pose sequence sorted in time sequence. After determining the intermediate feature map of the camera pose sequence at the previous time, the camera pose at the next time can be taken as the target camera pose, thereby providing a reliable implementation means for obtaining the effective intermediate feature map of the target camera pose (the final feature map of the target camera pose).

[0056] The method for obtaining the target camera pose provided by the optional implementation manner extracts all camera poses in the camera pose data to obtain a camera pose sequence, and selects a camera pose to be rendered in the camera pose sequence as the target camera pose, which can provide a reliable selection method for obtaining the target camera pose and ensure the reliability of the selection of the target camera pose.

[0057] Optionally, the determining of the target camera pose based on the camera pose data comprises: determining the identifier of the target camera, and extracting the camera pose corresponding to the identifier of the target camera from the camera pose data as the target camera pose.

[0058] Optionally, the determining of the target camera pose based on the camera pose data comprises: sorting all camera poses in the camera pose data in time sequence to obtain a camera pose sequence; and taking the camera pose in the camera pose sequence closest to the current time as the target camera pose.

[0059] Optionally, the determining of the target camera pose based on the camera pose data comprises: sorting all camera poses in the camera pose data in time sequence to obtain a camera pose sequence; and determining all camera poses with the final feature map in the camera pose sequence, and taking the camera pose without the final feature map in the camera pose sequence as the target camera pose. Through this optional implementation manner, the camera pose without the final feature map can be filtered out, and all camera poses without the final feature map can be taken as the target camera pose, which can ensure that all camera poses in the multi-frame camera pose data can have their own final feature maps.

[0060] In some optional implementation manners of this embodiment, the determining of the multi-frame intermediate feature map based on the camera pose data comprises:

[0061] The camera poses in the camera pose data are sequentially input into the pre-trained feature extraction model in the obtained time sequence to obtain the multi-frame intermediate feature maps output by the feature extraction model.

[0062] In the embodiment, when training the feature extraction model, a true value intermediate feature map is set for each camera pose in the camera pose data, and each camera pose is input into the pre-set feature extraction model to obtain the intermediate features corresponding to each camera pose output by the feature extraction model. The training of the feature extraction model is determined to be completed in response to the feature extraction model satisfying a training completion condition. The training completion condition includes that the number of training iterations of the feature extraction model reaches a preset iteration threshold, or the loss value of the feature extraction model reaches a preset loss threshold, wherein the loss value is a value obtained by performing loss function calculation on the intermediate features output by the feature extraction model and the true value intermediate feature map.

[0063] The method for obtaining multiple intermediate features provided in the embodiment inputs the camera pose in the camera pose data into the feature extraction model to obtain multiple intermediate feature maps, which is simple to implement, easy to operate, and improves the reliability of the intermediate feature map.

[0064] Optionally, the determination of the multiple intermediate feature maps based on the camera pose data includes inputting the camera pose data into multiple feature extraction algorithms to obtain the results of the multiple feature extraction algorithms, and adding the results of all the feature extraction algorithms to obtain the multiple intermediate feature maps. The feature extraction algorithm includes a color feature extraction algorithm, a texture feature extraction algorithm, a shape feature extraction algorithm, and a spatial relationship feature extraction algorithm.

[0065] In the embodiment, in the training of the feature extraction model, the true value intermediate feature map corresponding to the camera pose is relatively difficult to obtain. Therefore, the feature extraction model can be obtained through indirect means.

[0066] In some optional implementations of the embodiment, the feature extraction model can be obtained by training a neural radiance field network. The neural radiance field network includes a first multi-layer perceptron representing the correspondence between the camera pose data and the intermediate feature map, and the feature extraction model corresponds to the first multi-layer perceptron in the trained neural radiance field network.

[0067] In the embodiment, the neural radiance field network is first trained to obtain the trained neural radiance field network, and the first multi-layer perceptron in the trained neural radiance field network is used as the feature extraction model.

[0068] Specifically, the feature extraction model training step comprises: obtaining a preset neural radiance field network; the neural radiance field network comprises: a first multi-layer perceptron and a second multi-layer perceptron; the first multi-layer perceptron is configured to represent a correspondence between camera pose data and an intermediate feature map; the second multi-layer perceptron is configured to represent a correspondence between ray orientation data, the intermediate feature map, and a rendered image; obtaining a pose-ray sample set, the pose-ray sample set comprising: at least two consecutive frames of pose-ray samples of a target object in a same scene, each pose-ray sample comprising: camera pose data and corresponding ray orientation data; inputting the camera pose data in the selected pose-ray sample into the first multi-layer perceptron, and inputting the ray orientation data in the selected pose-ray sample into the second multi-layer perceptron to obtain a rendered image output by the neural radiance field network; in response to the neural radiance field network satisfying a training completion condition, obtaining a feature extraction model corresponding to the first multi-layer perceptron; the feature extraction model training step can refer to the operation and feature description shown in Figure 2

[0069] The method for training a feature extraction model provided in this embodiment first trains a neural radiance field network to obtain a trained neural radiance field network, and uses the first multi-layer perceptron in the trained neural radiance field network as the feature extraction model, which can improve the convenience of training the feature extraction model and ensure the reliability of the intermediate features.

[0070] In some optional implementations of this embodiment, the obtaining of the final rendered image based on the final feature map comprises: inputting the final feature map into a trained neural rendering model to obtain a final rendered image output by the neural rendering model; the neural rendering model is configured to represent a correspondence between the intermediate feature map and the rendered image.

[0071] In this embodiment, the final rendered image corresponds to the final feature map, and the final feature map is configured to reflect the features of the final rendered image.

[0072] In this embodiment, the neural rendering model can be a trained neural radiance field model, and the final feature map can be directly input into the trained neural radiance field model to obtain a final rendered image output by the neural radiance field model.

[0073] The method for obtaining a final rendered image provided in this embodiment can input the final feature map into a trained neural rendering model to simply and quickly obtain the final rendered image, thereby improving the reliability of obtaining the final rendered image.

[0074] In some optional implementations of this embodiment, the neural rendering model is trained based on an intermediate feature map output by a pre-trained neural radiance field model.

[0075] ​Specifically, the step of training the neural rendering model by the pre-trained neural radiance field model includes: obtaining a preset neural rendering network and a pose ray sample set, the pose ray sample set including: at least two frames of continuous pose ray samples of a target object in a same scene; inputting the selected pose ray sample into the pre-trained neural radiance field model to obtain an intermediate feature map output by the neural radiance field model; projecting the intermediate feature map to a plane position where the intermediate feature map corresponding to a target camera pose is located to obtain at least two frames of projection feature maps; inputting all the projection feature maps in the at least two frames of projection feature maps into the neural rendering network after connection to obtain a rendering image output by the neural rendering network; and obtaining a neural rendering model corresponding to the neural rendering network in response to the neural rendering network satisfying a training completion condition. The neural rendering model training step can be referred to the operation and feature description shown in Figure 3

[0076] In this embodiment, a neural rendering model is additionally added to the neural radiance field model, and the neural rendering model takes multiple frames of continuous intermediate feature maps as input and outputs a high-resolution rendering image, so as to accelerate rendering.

[0077] In this optional implementation, when training the neural rendering model, the intermediate feature corresponding to the pose ray data of different cameras is directly extracted from the trained neural radiance field model, and the extracted intermediate feature is taken as a sample of the neural rendering model, so that the training efficiency of the neural rendering model is improved.

[0078] Optionally, the neural rendering model is obtained by training a neural projection network, and the neural projection network includes: a first multi-layer perception, a projection module, and a rendering module, wherein the first multi-layer perception is used to represent the corresponding relationship between the camera pose data and the intermediate feature map; the projection module is used to represent the corresponding relationship between the multiple frames of intermediate feature maps and the final feature map; and the rendering module is used to represent the corresponding relationship between the final feature map and the rendering image.

[0079] The neural rendering model training method provided in this embodiment can simply and quickly obtain the final rendering image by training the neural projection network as a whole, and the reliability of the final rendering image is improved.

[0080] Figure 2 In addition to the flow 200 of one embodiment of the feature extraction model training method according to the features of the present disclosure, the feature extraction model training method includes the following steps:

[0081] In step 201, a pre-set neural radiance field network is obtained.

[0082] ​In this embodiment, the neural radiance field network comprises: a first multi-layer perceptron and a second multi-layer perceptron; wherein the first multi-layer perceptron is configured to represent the correspondence between the camera pose data and the intermediate feature map; and the second multi-layer perceptron is configured to represent the correspondence between the ray orientation data, the intermediate feature map and the rendered image. The first multi-layer perceptron and the second multi-layer perceptron are both artificial neural networks (ANN, Artificial Neural Network), which can have multiple hidden layers in addition to the input and output layers. The simplest MLP only contains one hidden layer, i.e., a three-layer network structure.

[0083] In actual application, after inputting the camera pose data into the first multi-layer perceptron, the ray density distribution and the intermediate feature map output by the first multi-layer perceptron can be obtained; and after inputting the intermediate feature map and the ray orientation data into the second multi-layer perceptron, the rendered image output by the second multi-layer perceptron can be obtained.

[0084] In step 202, a pose ray sample set is obtained, which comprises at least two continuous frames of pose ray samples of a target object in the same scene.

[0085] In this embodiment, the pose ray sample comprises camera pose data and ray orientation data corresponding to the camera pose data. The ray orientation corresponding to the camera pose is the orientation of the light ray emitted by the camera at the current camera pose. The camera orientation can be represented in polar coordinates, for example, the camera pose is (x, y, z), and the ray orientation is Therefore, the pose ray sample can be represented as

[0086] In this embodiment, the pose ray sample set comprises multiple frames of pose ray samples, each frame of pose ray sample corresponds to a target object, and the at least two continuous frames of pose ray samples are a set of camera pose data and ray orientation data at different camera poses of the camera.

[0087] In step 203, the camera pose data in the selected pose ray sample is input into the first multi-layer perceptron, and the ray orientation data in the selected pose ray sample is input into the second multi-layer perceptron, to obtain a rendered image output by the neural radiance field network.

[0088] In this embodiment, before step 203 is performed, at least one pose ray sample is selected from the pose ray sample set to obtain a selected pose ray sample.

[0089] In this embodiment, after inputting the camera pose data sample in the selected pose ray sample into the first multi-layer perception machine, the intermediate feature output by the first multi-layer perception machine is obtained; since the first multi-layer perception machine is connected with the second multi-layer perception machine in the neural radiance field network, the first multi-layer perception machine outputs an intermediate feature map, which is input into the second multi-layer perception machine together with the ray direction in the selected pose ray sample, and the second multi-layer perception machine obtains the rendering image based on the input intermediate feature map and the intermediate feature map.

[0090] In step 204, in response to the neural radiance field network satisfying the training completion condition, a feature extraction model corresponding to the first multi-layer perception machine is obtained.

[0091] In this embodiment, the training completion condition includes at least one of the following: the number of training iterations of the neural radiance field network reaches a first predetermined iteration threshold, and the loss value of the neural radiance field network is less than a first predetermined loss value threshold.

[0092] In this embodiment, the loss value of the neural radiance field network is calculated according to formula (2):

[0093] L c =MSE(c,c gt ) (2)

[0094] In formula (2), L c is the loss value of the neural radiance field network, c gt is the rendering image predicted by the neural radiance field network, and c is the true value of the rendering image input into the neural radiance field network.

[0095] Optionally, when the neural radiance field network does not satisfy the training completion condition, the related parameters in the neural radiance field network are adjusted, and steps 203 to 204 are executed again, so that the loss value of the neural radiance field network converges until the neural radiance field network satisfies the training completion condition.

[0096] The feature extraction model training method provided in this embodiment first trains the neural radiance field network through the pose ray sample, obtains the trained neural radiance field network satisfying the training completion condition, and takes the first multi-layer perception machine in the trained neural radiance field network as the feature extraction model, thereby optimizing the training step of the feature extraction model and ensuring the reliability of the feature extraction model training.

[0097] Figure 3 In addition to the flow 300 according to one embodiment of the rendering model training method, the above-mentioned rendering model training method includes the following steps:

[0098] In step 301, a preset neural rendering network and a pose ray sample set are obtained.

[0099] In this embodiment, the pose ray sample set includes at least two continuous frames of pose ray samples of the target object in the same scene.

[0100] In this embodiment, the neural rendering network is used to represent the correspondence between the final feature map and the rendering map, and the neural rendering network can adopt a multi-layer perceptron, preferably, the neural rendering network is a lightweight MLP network.

[0101] In this embodiment, the pose ray sample set includes multiple frames of pose ray samples, each frame of pose ray sample corresponds to a target object, and the at least two continuous frames of pose ray samples are camera pose data at different viewing angles and ray orientation data at the camera pose.

[0102] In step 302, the selected pose ray sample is input into the pre-trained neural radiation field model to obtain an intermediate feature map output by the neural radiation field model.

[0103] In this embodiment, before step 302 is performed, at least two pose ray samples of the target object in the same scene are selected from the pose ray sample set to obtain the selected pose ray sample.

[0104] In this embodiment, the selected pose ray sample has at least two frames, and the camera pose data in one of the at least two frames of pose ray samples is the target camera pose, and the plane position of the intermediate feature map of this frame of pose ray sample is the position to which all the intermediate feature maps need to be projected.

[0105] In step 303, the intermediate feature map is projected to the plane position of the intermediate feature map corresponding to the target camera pose to obtain at least two frames of projected feature maps.

[0106] In this embodiment, the projection of the intermediate feature map of the selected pose ray sample can be calculated by formula (1) to obtain at least two frames of projected feature maps.

[0107] In step 304, all the projected feature maps in the at least two frames of projected feature maps are connected and input into the neural rendering network to obtain a rendering map output by the neural rendering network.

[0108] In this embodiment, step 304 specifically includes: connecting the parameters of all the projected feature maps by using a connection string function (concat) to obtain connected projected feature maps; and inputting the connected projected feature maps into the neural rendering network to obtain a rendering map output by the neural rendering network.

[0109] In step 305, in response to the neural rendering network satisfying a training completion condition, a neural rendering model corresponding to the neural rendering network is obtained.

[0110] In this embodiment, the training completion condition comprises at least one of the following: the number of training iterations of the neural rendering network reaches a second predetermined iteration threshold, and the loss value of the neural rendering network is less than a second predetermined loss value threshold, wherein the second predetermined iteration threshold is irrelevant to the first predetermined iteration threshold, and the second predetermined loss value threshold is irrelevant to the first predetermined loss value threshold.

[0111] In this embodiment, the loss value of the neural rendering network is calculated according to formula (3):

[0112] L c ′=MSE(IM,IM gt ) (3)

[0113] In formula (2), L c ′ is the loss value of the neural rendering network, IM gt is the rendered image predicted by the neural rendering network, and IM is the true value of the rendered image input into the neural rendering network.

[0114] Optionally, when the neural rendering network does not meet the training completion condition, the related parameters in the neural rendering network are adjusted, and steps 302 to 305 are executed again, so that the loss value of the neural rendering network converges until the neural rendering network meets the training completion condition.

[0115] The rendering model training method provided in this embodiment selects at least two frames of pose ray samples, and according to the intermediate feature maps of the first few frames in the selected pose ray samples and the camera pose data, provides an intermediate feature map reference for the target camera pose of the current frame, obtains the final feature map corresponding to the target camera pose of the current frame, and performs image rendering in a lightweight rendering network by taking the feature map as a unit. In this way, the rendered image of the current frame can be obtained with very few parameters, and the calculation amount of the image rendering model is reduced.

[0116] The rendering model training method provided in this embodiment obtains the intermediate feature maps of the selected pose ray samples on the basis of the trained neural radiance field model, projects and connects the intermediate feature maps of the selected pose ray samples to obtain the final feature map of the target camera pose, and inputs the final feature map into the neural rendering network, thereby saving the calculation amount during image rendering, improving the reliability of the image rendering model, and guaranteeing the image rendering effect.

[0117] Further referring to Figure 4 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of an image rendering device, which corresponds to the method embodiment shown in Figure 1 , and the device can be applied in various electronic devices.

[0118] As Figure 4As shown, the image rendering apparatus 400 provided by the embodiment includes a data acquisition unit 401, a feature determination unit 402, a pose determination unit 403, a final determination unit 404, and a final obtaining unit 405. The data acquisition unit 401 can be configured to acquire camera pose data of a target object under multiple views. The feature determination unit 402 can be configured to determine a plurality of intermediate feature maps based on the camera pose data. The pose determination unit 403 can be configured to determine a target camera pose to be rendered based on the camera pose data. The final determination unit 404 can be configured to determine a final feature map corresponding to the target camera pose based on the plurality of intermediate feature maps and the target camera pose. The final obtaining unit 405 can be configured to obtain a final rendering map based on the final feature map.

[0119] In the embodiment, the specific processing of the data acquisition unit 401, the feature determination unit 402, the pose determination unit 403, the final determination unit 404, and the final obtaining unit 405 in the image rendering apparatus 400 and the technical effects brought by the specific processing can be respectively referred to Figure 1 The related descriptions of steps 101, 102, 103, 104, and 105 in the corresponding embodiment are not repeated here.

[0120] In some optional implementation manners of the embodiment, the final determination unit 404 is further configured to: determine an intermediate feature map corresponding to the target camera pose from the plurality of intermediate feature maps; project each intermediate feature map in the plurality of intermediate feature maps to a plane position where the intermediate feature map corresponding to the target camera pose is located, to obtain a plurality of projection feature maps; and connect all the projection feature maps in the plurality of projection feature maps to obtain the final feature map corresponding to the target camera pose.

[0121] In some optional implementation manners of the embodiment, the pose determination unit 403 is further configured to: extract all camera poses in the camera pose data to obtain a camera pose sequence; and select a camera pose to be rendered in the camera pose sequence as the target camera pose.

[0122] In some optional implementation manners of the present disclosure, the feature determination unit 402 is further configured to: input camera poses in the camera pose data into a pre-trained feature extraction model in a time sequence to obtain a plurality of intermediate feature maps output by the feature extraction model.

[0123] In some optional implementations of the present disclosure, the feature extraction model is obtained by training a neural radiance field network; the neural radiance field network comprises a first multi-layer perceptron representing the correspondence between the camera pose data and the intermediate feature maps; the feature extraction model corresponds to the first multi-layer perceptron in the trained neural radiance field network.

[0124] In some optional implementations of the present disclosure, the final obtaining unit 405 is further configured to: input the final feature map into a trained neural rendering model to obtain a final rendering map output by the neural rendering model; the neural rendering model is used to represent the correspondence between the intermediate feature map and the rendering map.

[0125] In some optional implementations of the present disclosure, the neural rendering model is trained by the intermediate feature map output by the pre-trained neural radiance field model.

[0126] The image rendering apparatus provided by the embodiments of the present disclosure first acquires, by the data acquisition unit 401, camera pose data of a target object under multiple views; secondly, the feature determination unit 402 determines, based on the camera pose data, multiple intermediate feature maps; thirdly, the pose determination unit 403 determines, based on the camera pose data, a target camera pose to be rendered; fourthly, the final determination unit 404 determines, based on the multiple intermediate feature maps and the target camera pose, a final feature map corresponding to the target camera pose; and finally, the final obtaining unit 405 obtains a final rendering map based on the final feature map. In this way, the multiple intermediate feature maps of the multiple camera pose data are referenced to reconstruct and refine the intermediate feature map of the target camera pose, thereby reducing the calculation amount of the final rendering map corresponding to the target camera pose, reducing the memory occupied by the rendering, and effectively improving the image rendering efficiency.

[0127] Continuing to refer to Figure 5 , as an implementation of the method shown in Figure 2 , the present disclosure provides an embodiment of a feature extraction model training apparatus. The apparatus embodiment corresponds to the method embodiment shown in Figure 2 , and the apparatus can be specifically applied to various electronic devices.

[0128] As shown in Figure 5As shown, the feature extraction model training apparatus 500 provided by the embodiment includes a network obtaining unit 501, a sample obtaining unit 502, an input unit 503, and a first obtaining unit 504. The network obtaining unit 501 can be configured to obtain a preset neural radiance field network. The neural radiance field network includes a first multi-layer perceptron and a second multi-layer perceptron. The first multi-layer perceptron is used to represent the corresponding relationship between camera pose data and an intermediate feature map. The second multi-layer perceptron is used to represent the corresponding relationship between ray orientation data, the intermediate feature map, and a rendered image. The sample obtaining unit 502 can be configured to obtain a pose-ray sample set. The pose-ray sample set includes at least two continuous frames of pose-ray samples of a target object in a same scene. Each pose-ray sample includes camera pose data and corresponding ray orientation data. The input unit 503 can be configured to input the camera pose data in the selected pose-ray sample into the first multi-layer perceptron, input the ray orientation data in the selected pose-ray sample into the second multi-layer perceptron, and obtain a rendered image output by the neural radiance field network. The first obtaining unit 504 can be configured to obtain a feature extraction model corresponding to the first multi-layer perceptron in response to the neural radiance field network satisfying a training completion condition.

[0129] In the embodiment, the specific processing of the network obtaining unit 501, the sample obtaining unit 502, the input unit 503, and the first obtaining unit 504 in the feature extraction model training apparatus 500 and the technical effects brought by the specific processing can be respectively referred to Figure 2 The related descriptions of steps 201, 202, 203, and 204 in the corresponding embodiment will not be repeated here.

[0130] The feature extraction model training apparatus provided by the embodiment of the present disclosure first includes the network obtaining unit 501 configured to obtain a preset neural radiance field network. The sample obtaining unit 502 is configured to obtain a pose-ray sample set. The pose-ray sample set includes at least two continuous frames of pose-ray samples of a target object in a same scene. The input unit 503 is configured to input the camera pose data in the selected pose-ray sample into the first multi-layer perceptron, input the ray orientation data in the selected pose-ray sample into the second multi-layer perceptron, and obtain a rendered image output by the neural radiance field network. The first obtaining unit 504 is configured to obtain a feature extraction model corresponding to the first multi-layer perceptron in response to the neural radiance field network satisfying a training completion condition. The neural radiance field network is trained by using the pose-ray samples to obtain a trained neural radiance field network satisfying the training completion condition. The first multi-layer perceptron in the trained neural radiance field network is used as the feature extraction model. The training step of the feature extraction model is optimized, and the reliability of the feature extraction model training is ensured.

[0131] Continuing to refer to Figure 6As an implementation of the method shown in the above Figure 3 As an implementation of the method shown in the above Figure 3 The device embodiment corresponds to the method embodiment shown in the above

[0132] As shown in the above Figure 6 The device 600 provided by the embodiment comprises a sample obtaining unit 601, a feature extraction unit 602, a projection unit 603, a rendering unit 604, and a second obtaining unit 605. The sample obtaining unit 601 is configured to obtain a preset neural rendering network and a pose-ray sample set, wherein the pose-ray sample set comprises at least two continuous frames of pose-ray samples of a target object in a same scene. The feature extraction unit 602 is configured to input the selected pose-ray sample into a pre-trained neural radiance field model to obtain an intermediate feature map output by the neural radiance field model. The projection unit 603 is configured to project the intermediate feature map to a plane position corresponding to an intermediate feature map in which a target camera pose is located to obtain at least two frames of projection feature maps. The rendering unit 604 is configured to input all the projection feature maps in the at least two frames of projection feature maps into the neural rendering network after being connected to obtain a rendering image output by the neural rendering network. The second obtaining unit 605 is configured to obtain a neural rendering model corresponding to the neural rendering network in response to the neural rendering network satisfying a training completion condition.

[0133] In the embodiment, the specific processing of the sample obtaining unit 601, the feature extraction unit 602, the projection unit 603, the rendering unit 604, and the second obtaining unit 605 in the device 600 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of the steps 301, 302, 303, 304, and 305 in the corresponding embodiment, which will not be repeated here. Figure 3 The corresponding embodiment corresponds to the method embodiment shown in the above

[0134] The embodiment of the present disclosure provides a rendering model training device. First, a sample acquisition unit 601 acquires a preset neural rendering network and a pose ray sample set, and the pose ray sample set includes at least two frames of pose ray samples of a target object in a same scene. A feature extraction unit 602 inputs the selected pose ray sample into a pre-trained neural radiation field model to obtain an intermediate feature map output by the neural radiation field model. A projection unit 603 projects the intermediate feature map to a plane position where the intermediate feature map is located corresponding to a target camera pose to obtain at least two frames of projection feature maps. A rendering unit 604 inputs all the projection feature maps in the at least two frames of projection feature maps into the neural rendering network after connection to obtain a rendering map output by the neural rendering network. A second obtaining unit 605 obtains a neural rendering model corresponding to the neural rendering network in response to the neural rendering network satisfying a training completion condition. Thus, the intermediate feature map of the selected pose ray sample is projected and connected to obtain a final feature map of the target camera pose, the final feature map is input into the neural rendering network, the calculation amount during image rendering is saved, the reliability of the image rendering model is improved, and the image rendering effect is guaranteed.

[0135] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0136] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0137] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0138] As Figure 7As shown, the device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0139] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, and the like; an output unit 707, such as various types of displays, speakers, and the like; a storage unit 708, such as a magnetic disk, an optical disk, and the like; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0140] The computing unit 701 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 701 performs various methods and processes described above, such as an image rendering method or a feature extraction model training method or a rendering model training method. For example, in some embodiments, the image rendering method or the feature extraction model training method or the rendering model training method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the image rendering method or the feature extraction model training method or the rendering model training method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the image rendering method or the feature extraction model training method or the rendering model training method by any other appropriate means, such as by means of firmware.

[0141] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0142] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable image rendering or feature extraction model training or rendering model training device to produce a machine such that the program code, when executed by the processor or controller, implements the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0145] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0147] It should be understood that various forms of flow shown above can be used, re-ordered, added to, or deleted from without departing from the spirit of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, without departing from the desired results of the technology disclosed in the present disclosure, and are not limited herein.

[0148] The specific embodiments described above have been shown by way of example, and anyone skilled in the art should understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalent substitutions, improvements, and the like, which are apparent to one skilled in the art, are intended to be encompassed within the scope of the present disclosure.

Claims

1. An image rendering method, the method comprising: Acquire camera pose data of a target object from multiple frames and multiple perspectives; Based on the camera pose data, a multi-frame intermediate feature map is determined. The camera pose data is used to reflect the multi-frame images of the target object captured from the multiple viewpoints. Each frame of the multi-frame images has image features. The intermediate feature map is a set of image features or the representation of image features on a model. Based on the camera pose data, the target camera pose is determined; Based on the multi-frame intermediate feature maps and the target camera pose, a final feature map corresponding to the target camera pose is determined, wherein the final feature map is an intermediate feature map corresponding to the target camera pose; Obtaining the final rendered image based on the final feature map includes: inputting the final feature map into a trained neural rendering model to obtain the final rendered image output by the neural rendering model; the neural rendering model is used to characterize the correspondence between the intermediate feature map and the rendered image.

2. The method according to claim 1, wherein, The step of determining the final feature map corresponding to the target camera pose based on the multi-frame intermediate feature maps and the target camera pose includes: Determine the intermediate feature map corresponding to the target camera pose from the multi-frame intermediate feature maps; Project each frame intermediate feature map in the multi-frame intermediate feature map onto the plane position where the intermediate feature map corresponding to the target camera pose is located to obtain a multi-frame projected feature map. By connecting all the projection feature maps in the multi-frame projection feature maps, the final feature map corresponding to the target camera pose is obtained.

3. The method according to claim 1, wherein, Determining the target camera pose based on the camera pose data includes: Extract all camera poses from the camera pose data to obtain a camera pose sequence; Select the camera pose to be rendered from the camera pose sequence, and use the camera pose to be rendered as the target camera pose.

4. The method according to claim 1, wherein, The process of determining the intermediate feature maps across multiple frames based on the camera pose data includes: The camera pose data is input into a pre-trained feature extraction model in the obtained time order to obtain multi-frame intermediate feature maps output by the feature extraction model.

5. The method according to claim 4, wherein, The feature extraction model is obtained by training a neural radiation field network; The neural radiation field network includes a first multilayer perceptron representing the correspondence between camera pose data and intermediate feature maps, and the feature extraction model corresponds to the first multilayer perceptron in the trained neural radiation field network.

6. The method according to claim 5, wherein, The neural radiation field network also includes a second multilayer perceptron for characterizing the correspondence between the ray orientation data, the intermediate feature map and the rendered map; as well as The feature extraction model is obtained by training the neural radiation field network, including: Obtain the pre-set neural radiation field network; Acquire a pose ray sample set, which includes: at least two consecutive frames of pose ray samples of a target object in the same scene, and the pose ray samples include: camera pose data and ray orientation data corresponding to the camera pose data. The following training steps are performed: the camera pose data in the selected pose ray samples are input into the first multilayer perceptron, and the ray orientation data in the selected pose ray samples are input into the second multilayer perceptron to obtain the rendering image output by the neural radiation field network; in response to the neural radiation field network meeting the training completion condition, the feature extraction model is obtained.

7. The method according to claim 1, wherein, The neural rendering model is trained using intermediate feature maps output from a pre-trained neural radiation field model.

8. The method according to claim 7, wherein, The neural rendering model is trained using the intermediate feature maps output by the neural radiation field model, including: The pose ray samples selected from the pose ray sample set are input into the neural radiation field model to obtain the intermediate feature map output by the neural radiation field model; wherein, the pose ray sample set includes: at least two consecutive frames of pose ray samples of the target object in the same scene; The intermediate feature map is projected onto the plane where the intermediate feature map corresponding to the target camera pose is located, to obtain at least two frames of projected feature maps. The projection feature maps of the at least two frames are connected and then input into a preset neural rendering network to obtain the rendering map output by the neural rendering network. In response to the neural rendering network meeting the training completion conditions, a neural rendering model corresponding to the neural rendering network is obtained.

9. An image rendering apparatus, the apparatus comprising: The data acquisition unit is configured to acquire camera pose data of the target object from multiple perspectives across multiple frames. The feature determination unit is configured to determine a multi-frame intermediate feature map based on the camera pose data, wherein the camera pose data is used to reflect the multi-frame images of the target object captured from the multiple viewpoints, each of the multi-frame images has image features, and the intermediate feature map is a set of image features or the representation of image features on a model. The pose determination unit is configured to determine the pose of the target camera based on the camera pose data; The final determination unit is configured to determine the final feature map corresponding to the target camera pose based on the multi-frame intermediate feature map and the target camera pose, wherein the final feature map is an intermediate feature map corresponding to the target camera pose; The resulting unit is configured to produce the final rendered image based on the final feature map. The final obtaining unit is further configured to: input the final feature map into a trained neural rendering model to obtain the final rendering map output by the neural rendering model; the neural rendering model is used to characterize the correspondence between the intermediate feature map and the rendering map.

10. The apparatus according to claim 9, wherein, The final determination unit is further configured to: determine an intermediate feature map corresponding to the target camera pose from the multi-frame intermediate feature map; project each frame intermediate feature map in the multi-frame intermediate feature map onto the plane position where the intermediate feature map corresponding to the target camera pose is located, to obtain a multi-frame projected feature map; and connect all the projected feature maps in the multi-frame projected feature map to obtain the final feature map corresponding to the target camera pose.

11. The apparatus according to claim 9, wherein, The pose determination unit is further configured to: extract all camera poses from the camera pose data to obtain a camera pose sequence; select the camera pose to be rendered from the camera pose sequence and use the camera pose to be rendered as the target camera pose.

12. The apparatus according to claim 9, wherein, The feature determination unit is further configured to: input the camera poses in the camera pose data into the pre-trained feature extraction model in the obtained time order, and obtain the multi-frame intermediate feature maps output by the feature extraction model.

13. The apparatus according to claim 12, wherein, The feature extraction model is obtained by training a neural radiation field network; The neural radiation field network includes a first multilayer perceptron representing the correspondence between camera pose data and intermediate feature maps, and the feature extraction model corresponds to the first multilayer perceptron in the trained neural radiation field network.

14. The apparatus according to claim 13, wherein, The neural radiation field network also includes a second multilayer perceptron for characterizing the correspondence between the ray orientation data, the intermediate feature map and the rendered map; as well as The feature extraction model is obtained by training the neural radiation field network, including: Obtain the pre-set neural radiation field network; Acquire a pose ray sample set, which includes: at least two consecutive frames of pose ray samples of a target object in the same scene, and the pose ray samples include: camera pose data and ray orientation data corresponding to the camera pose data. The following training steps are performed: the camera pose data in the selected pose ray samples are input into the first multilayer perceptron, and the ray orientation data in the selected pose ray samples are input into the second multilayer perceptron to obtain the rendering image output by the neural radiation field network; in response to the neural radiation field network meeting the training completion condition, the feature extraction model is obtained.

15. The apparatus according to claim 9, wherein, The neural rendering model is trained using intermediate feature maps output from a pre-trained neural radiation field model.

16. The apparatus according to claim 15, wherein, The neural rendering model is trained using the intermediate feature maps output by the neural radiation field model, including: The pose ray samples selected from the pose ray sample set are input into the neural radiation field model to obtain the intermediate feature map output by the neural radiation field model; wherein, the pose ray sample set includes: at least two consecutive frames of pose ray samples of the target object in the same scene; The intermediate feature map is projected onto the plane where the intermediate feature map corresponding to the target camera pose is located, to obtain at least two frames of projected feature maps. The projection feature maps of the at least two frames are connected and then input into a preset neural rendering network to obtain the rendering map output by the neural rendering network. In response to the neural rendering network meeting the training completion conditions, a neural rendering model corresponding to the neural rendering network is obtained.

17. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-8.

Images