Three-dimensional scene rendering method based on space-time super-resolution and neural radiation field

Abstract

The invention discloses a three-dimensional scene rendering method based on space-time super-resolution and a neural radiation field, and belongs to the field of three-dimensional scene reconstruction, and the method comprises the following steps: inputting a multi-view image into a neural radiation field to obtain a target frame low-resolution image and a support frame low-resolution image; performing up-sampling processing on the target frame low-resolution image to obtain a target frame high-resolution image of a first branch; based on the target frame low-resolution image and the support frame low-resolution image, adopting a video interpolation network for processing and carrying out image projection to obtain a target frame high-resolution image of a second branch; and inputting the target frame high-resolution image of the first branch and the target frame high-resolution image of the second branch into a lightweight neural renderer to obtain a target three-dimensional scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional scene reconstruction, and in particular relates to a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field. Background Art

[0002] The task of novel view synthesis (Novel View Synthesis) refers to rendering and generating a target image (Target Image) corresponding to a given source image (Source Image) and source pose (Source Pose), as well as a target pose (Target Pose). Novel view synthesis has a wide range of applications in 3D reconstruction, AR / VR and other fields. As users increasingly seek realistic and immersive experiences, end users strongly demand efficient and high-quality methods to synthesize novel views. Among various representative methods, neural radiance field (NeRF) has received widespread attention for its ability to generate high-fidelity images. However, with the increasing demand for realistic and high-quality visual experience, the challenge of achieving real-time performance in neural radiance field (NeRF) rendering remains. Balancing the trade-off between quality and performance is becoming increasingly challenging, as improving one often comes at the expense of the other. In addition, the computational requirements of NeRF result in long rendering times, making real-time rendering impossible. Therefore, the present invention provides a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiance fields. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field, comprising:

[0005] Inputting the multi-view images into the neural radiance field to obtain a target frame low-resolution image and a support frame low-resolution image;

[0006] Performing upsampling processing on the target frame low-resolution image to obtain a target frame high-resolution image of the first branch;

[0007] Based on the target frame low-resolution image and the support frame low-resolution image, a video interpolation network is used for processing and image projection is performed to obtain a target frame high-resolution image of a second branch;

[0008] The target frame high-resolution image of the first branch and the target frame high-resolution image of the second branch are input into a lightweight neural renderer to obtain a target three-dimensional scene.

[0009] Optionally, the process of obtaining a high-resolution image of a target frame in the second branch includes:

[0010] Extracting feature maps of the target frame low-resolution image and the support frame low-resolution image;

[0011] Creating an intermediate frame feature map based on the feature map using a learned feature interpolation function;

[0012] The intermediate frame feature map is reprojected onto the point cloud, and the intermediate frame feature map on the point cloud is projected back to the high-resolution image from the perspective of the target frame to obtain the target frame high-resolution image of the second branch.

[0013] Optionally, the expression for generating the intermediate frame feature map is:

[0014] L t ,L t-1 ,L t-2 =V(L t-1 ,L t-2 )

[0015] Where, L t represents the target frame, L t-2 Indicates support for frames, L t-1 represents the intermediate frame, and V represents the learned feature interpolation function.

[0016] Optionally, the expression for the process of reprojecting the intermediate frame feature map onto the point cloud and projecting the intermediate frame feature map on the point cloud back to the high-resolution image from the perspective of the target frame is:

[0017] H t→t ,H t-1→t ,H t-2→t =W(L t ,L t-1 ,L t-2 )

[0018] =P W (P t ,P t-1 ,P t-2 )

[0019] =P W (P U (L t ,L t-1 ,L t-2 ))

[0020] Where P t ,P t-1 ,P t-2 Respectively, through L t ,L t-1 ,L t-2 The point cloud obtained by reprojection oft→t ,H t-1→t ,H t-2→t Respectively represent the point cloud P t ,P t-1 ,P t-2 Projection to L t High-resolution images on the view; P U represents the function of reprojecting the low-resolution image onto the point cloud; P W represents the function of projecting the point cloud from the view of the t-th frame to the high-resolution image; W represents the function of warping the low-resolution image to the high-resolution image corresponding to the t-th frame view.

[0021] Optionally, the process of constructing a target three-dimensional scene based on the target frame high-resolution image of the first branch and the target frame high-resolution image of the second branch further includes a process of fusing a plurality of high-resolution images, wherein an expression for fusing the plurality of high-resolution images is:

[0022] N t =O(H t→t ,H t-1→t ,H t-2→t ,U t )

[0023] Where N t Represents a new view synthetic image generated from multiple high-resolution images using U-Net.

[0024] Optionally, the process of inputting the multi-view images into the neural radiance field to generate a low-resolution image also includes using photometric loss and depth loss to constrain the image quality of the neural radiance field;

[0025] Among them, the expressions of photometric loss and depth loss are:

[0026]

[0027] Where, L c Represents the luminosity loss, L d represents the depth loss.

[0028] The present invention also provides a computer terminal device, comprising:

[0029] one or more processors;

[0030] A memory, coupled to the processor, for storing one or more programs;

[0031] When the one or more programs are executed by the one or more processors, the one or more processors implement a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field is implemented.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] The present invention proposes a NeRF acceleration method for spatiotemporal super-resolution, which enables NeRF to achieve high-quality real-time rendering effects at a low computing cost.

[0035] The present invention proposes a new method using spatiotemporal super-resolution, which achieves high-quality rendering of NeRF with high FPS through temporal interpolation and spatial supersampling. Specifically, the present invention can intelligently interpolate information between adjacent frames in the time dimension through temporal interpolation, thereby achieving a more continuous effect. In conjunction with a lightweight super-resolution network, the time required for rendering and super-resolution of the neural radiation field can be greatly reduced, while maintaining high-quality rendering effects. At the same time, the present invention makes full use of depth information and can better realize the process of fusing multi-frame feature maps into the target frame. The present invention uses spatial supersampling technology to increase the density of sampling points in the spatial dimension, so that the rendered image can be greatly improved in detail and clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 A flow chart of a three-dimensional scene construction method according to an embodiment of the present invention;

[0038] Figure 2 is an overall flow chart of an embodiment of the present invention;

[0039] Figure 3 This is a frame feature time interpolation method based on deformable sampling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Embodiment 1

[0043] The existing solutions do not make full use of spatiotemporal information. In order to speed up rendering, a large amount of memory consumption must be sacrificed, and the rendering speed of real scenes is still poor. Therefore, this embodiment provides a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field. The method realizes high-quality rendering of NeRF with high FPS through temporal interpolation and spatial super-sampling. The present invention includes the following steps:

[0044] like Figure 1 As shown, a multi-view image is input into a neural radiation field to obtain a target frame low-resolution image and a support frame low-resolution image; the target frame low-resolution image is upsampled to obtain a target frame high-resolution image of the first branch; based on the target frame low-resolution image and the support frame low-resolution image, a video interpolation network is used for processing and image projection is performed to obtain a target frame high-resolution image of the second branch; the target frame high-resolution image of the first branch and the target frame high-resolution image of the second branch are input into a lightweight neural renderer to obtain a target three-dimensional scene.

[0045] As a specific implementation of this embodiment, Figure 2 As shown in Figure 2, a spatial and viewpoint-based super-resolution method is proposed. Specifically, first, a neural radiance field is iteratively trained using multi-view images as input. Through volume rendering, low-resolution RGB and depth images L are obtained respectively. t-2 and L t , where the subscript t represents the image at frame t and L represents the low-resolution image. Next, two branches are used to perform spatiotemporal super-resolution to obtain the final target resolution output O t , where O represents the output image. The first branch focuses on the direct upsampling operation applied to the target frame, resulting in a high-resolution image H of the target frame. t , where H represents the high-resolution image. The second branch implements the spatiotemporal super-resolution method: the video interpolation network generates RGB and depth images for the intermediate frames (L t-1 ), and then use the depth information to reproject the RGB image onto the point cloud, and project it back to a high-resolution image from the perspective of the target frame. Finally, the high-resolution target frame image obtained from the two branches is input into the lightweight neural renderer of this embodiment to generate the final output.

[0046] Spatiotemporal super-resolution NeRF

[0047] The spatial and viewpoint super-resolution module of this embodiment includes two main components: viewpoint interpolation and spatial upsampling, such as Figure 2By integrating these components, the present embodiment improves the resolution and coherence of the target frame, thereby improving the visual quality. The initial stage of the module in this embodiment focuses on viewpoint interpolation, including t ) and support frames (L t-2 ) and then learn the feature interpolation function V to directly create the intermediate feature map. The main goal is to accurately generate the intermediate frame (L t-1 ) can be expressed as:

[0048] L t ,L t-1 ,L t-2 =V(L t-1 ,L t-2 )

[0049] To approximate the forward and backward motion, this embodiment uses the motion information between the target frame and the support frame. Inspired by ZoomingSlow-Mo, such as Figure 3 As shown, the present embodiment integrates deformable convolution into the sampling function, enabling the present embodiment to effectively delve into complex local temporal contexts. This feature enables the viewpoint interpolation of the present embodiment to skillfully adapt to large movements between two frames. In addition, in combination with deformable convolution, the present embodiment learns an offset as a sampling parameter to implicitly capture forward and backward motion information. This offset facilitates precise alignment, ensuring temporal coherence and smooth transitions. Through these techniques, the present embodiment can generate intermediate feature maps that seamlessly connect target and support frames.

[0050] In addition to viewpoint interpolation, this embodiment also uses spatial upsampling to improve the resolution of the target frame. After performing viewpoint interpolation, this embodiment obtains feature maps of the support frame, intermediate frame, and target frame. Using the depth information, this embodiment projects the RGB image onto the point cloud, and then projects it onto the high-resolution image from the perspective of the target frame:

[0051] H t→t ,H t-1→t ,H t-2→t =W(L t ,L t-1 ,L t-2 )

[0052] =P W (P t ,P t-1 ,P t-2 )

[0053] =P W (P U (L t ,L t-1 ,L t-2 ))

[0054] Among them, P t ,P t-1 ,P t-2 Respectively, through L t ,L t-1 ,L t-2 The point cloud obtained by reprojection, H t→t ,H t-1→t ,H t-2→t Respectively represent the point cloud P t ,P t-1 ,P t-2 Projection to L t The high-resolution image on the view. Function P U , P W and W are used to reproject the low-resolution image onto the point cloud, project the point cloud from the view of the t-th frame onto the high-resolution image, and warp the low-resolution image onto the high-resolution image corresponding to the view of the t-th frame. By integrating these high-resolution frames obtained from different view warping and the upsampling results of the target frame, the present embodiment can obtain the upsampled target frame:

[0055] U t =U p (L t )

[0056] N t =O(H t→t ,H t-1→t ,H t-2→t ,U t )

[0057] Among them U t Represents the low-resolution image L t Upsampled frames, N t Represents a new view synthetic image generated from multiple high-resolution images using U-Net. Function U P and O represent upsampling and fusion of multiple high-resolution images through U-Net, respectively.

[0058] In order to minimize the time loss, the present embodiment can improve efficiency and optimize memory utilization by utilizing LR rendering buffers. Although these buffers incur time and memory costs, they provide a more efficient alternative for rendering HR frames. In addition, LR images from the rendering buffer can be used to warp into different views, which helps the spatial upsampling of the present embodiment. The method of the present embodiment optimizes real-time rendering by integrating rendering buffers, improving efficiency, memory usage, and spatial upsampling to obtain better visual quality.

[0059] The render buffer stores the low-resolution feature map {F t-L,,,,,F t-1} and depth map {D t-L ,,,,,D t-1 At the current viewpoint t, the low-resolution feature map F is rendered faster through the rendering buffer t and the depth map D t .

[0060] Loss Function

[0061] This embodiment uses photometric loss and depth loss to constrain the quality of NeRF generated images. The depth map generated during NeRF training is often inaccurate, which poses a challenge when performing spatial oversampling. Accurate depth information is critical to ensure that other frames can be correctly warped to the target frame. In order to solve this problem and improve the accuracy of depth information, this embodiment uses the trained network to generate a pseudo GT of the depth map and uses it to supervise the depth information.

[0062] The photometric loss and depth loss are calculated as follows:

[0063]

[0064]

[0065] The total loss function of the entire network is:

[0066] L total =L c +λ d L d

[0067] where λ d is the trade-off parameter for depth loss.

[0068] Compared with the existing methods, the present invention significantly improves the rendering frame rate of NeRF while maintaining high-quality rendering.

[0069] Embodiment 2

[0070] This embodiment also provides a computer terminal device, including:

[0071] one or more processors;

[0072] A memory, coupled to the processor, for storing one or more programs;

[0073] When one or more programs are executed by one or more processors, the one or more processors implement a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field.

[0074] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field is implemented.

[0075] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field, characterized in that: The following steps are involved: Inputting the multi-view images into the neural radiance field to obtain a target frame low-resolution image and a support frame low-resolution image; Performing upsampling processing on the target frame low-resolution image to obtain a target frame high-resolution image of the first branch; Based on the target frame low-resolution image and the support frame low-resolution image, a video interpolation network is used for processing and image projection is performed to obtain a target frame high-resolution image of a second branch; The target frame high-resolution image of the first branch and the target frame high-resolution image of the second branch are input into a lightweight neural renderer to obtain a target three-dimensional scene.

2. The three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field according to claim 1, characterized in that: The process of the target frame high-resolution image of the second branch includes: Extracting feature maps of the target frame low-resolution image and the support frame low-resolution image; Creating an intermediate frame feature map based on the feature map using a learned feature interpolation function; The intermediate frame feature map is reprojected onto the point cloud, and the intermediate frame feature map on the point cloud is projected back to the high-resolution image from the perspective of the target frame to obtain the target frame high-resolution image of the second branch.

3. The three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field according to claim 2 is characterized in that: The expression for generating the intermediate frame feature map is: L t ,L t-1 ,L t-2 =V(L t-1 ,L t-2 ) Where, L t represents the target frame, L t-2 Indicates support for frames, L t-1 represents the intermediate frame, and V represents the learned feature interpolation function.

4. The three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field according to claim 3 is characterized in that: The expression of the process of reprojecting the intermediate frame feature map onto the point cloud and projecting the intermediate frame feature map on the point cloud back to the high-resolution image from the perspective of the target frame is: H t→t ,H t-1→t ,H t-2→t =W(L t ,L t-1 ,L t-2 ) =P W (P t ,P t-1 ,P t-2 ) =P W (P U (L t ,L t-1 ,L t-2 )) Where P t ,P t-1 ,P t-2 Respectively, through L t ,L t-1 ,L t-2 The point cloud obtained by reprojection of t→t ,H t-1→t ,H t-2→t Respectively represent the point cloud P t ,P t-1 ,P t-2 Projection to L t High-resolution images on the view; P U represents the function of reprojecting the low-resolution image onto the point cloud; P W represents the function of projecting the point cloud from the view of the t-th frame to the high-resolution image; W represents the function of warping the low-resolution image to the high-resolution image corresponding to the t-th frame view.

5. The three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field according to claim 4, characterized in that: The process of constructing a target three-dimensional scene based on the target frame high-resolution image of the first branch and the target frame high-resolution image of the second branch also includes a process of fusing a plurality of high-resolution images, wherein the expression for fusing a plurality of high-resolution images is: N t =O(H t→t ,H t-1→t ,H t-2→t ,U t ) Where N t Represents a new view synthetic image generated from multiple high-resolution images using U-Net.

6. The three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field according to claim 1, characterized in that: The process of inputting multi-view images into the neural radiance field to generate a low-resolution image also includes using photometric loss and depth loss to constrain the image quality of the neural radiance field; Among them, the expressions of photometric loss and depth loss are: Where, L c Represents the luminosity loss, L d represents the depth loss.

7. A computer terminal device, characterized in that: include: one or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional scene rendering method based on spatiotemporal super-resolution and neural radiation field as described in any one of claims 1 to 6 is implemented.

Images