A denoising method and system for continuous frame rendering by ray tracing

Abstract

The present invention provides a denoising method and system for continuous frames of ray tracing rendering, including: obtaining the dual motion vectors of the current frame and the denoising result of the previous frame, calculating the distortion reprojection of the previous frame; obtaining the noise map and auxiliary features of the current frame, combining the distortion reprojection of the previous frame, and splicing them into a tensor; inputting the tensor into a spatiotemporal multiscale denoising network to obtain the denoising result of the current frame; wherein the spatiotemporal multiscale denoising network includes: a kernel prediction network and a spatiotemporal multiscale hybrid network, and the spatiotemporal multiscale hybrid network combines the time filtering result and the spatial multiscale hybrid result to obtain the denoising result of the current frame. Under the premise of ensuring the acquisition of high-quality denoising results, the over-blurring, coloring aliasing and tailing phenomena existing in the prior art are eliminated.

Description

Technical Field

[0001] The present invention belongs to the technical field of post-processing denoising of highly realistic rendering, and in particular relates to a denoising method and system for continuous frames of ray tracing rendering. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Ray tracing technology based on Monte Carlo (MC) integration is widely used in physically based realistic rendering. Traditional MC path tracing requires time-consuming calculations and a large number of samples per pixel (spp) to obtain impressive rendering results, but due to the real-time frame rate requirements (>30FPS) and the limitations of current hardware devices, real-time ray tracing (RTRT) can only use 1spp. The extremely low sampling rate will lead to high variance, which is visually manifested as interference noise.

[0004] Some methods (e.g., Temporal Anti-Aliasing (TAA) and post-processing denoising methods based on neural networks) have been proposed to obtain usable noise-free ray tracing rendering continuous frame sequences at interactive rates or even real-time rates. From the characteristics of ray tracing rendering continuous frames, it is known that in addition to noise (nosing), flickering (flicking) between continuous frames needs to be removed. Motion vector (Motion vector) introduces historical frame information for reuse in the time domain, which plays a key role in removing inter-frame flicker. The traditional screen-based motion vector (SVGF) is a two-dimensional vector that points from each pixel coordinate of the current frame to the corresponding pixel coordinate of the object in the previous frame, but it will fail in shadows, glossy material reflections and motion occlusion. In particular, for the motion occlusion area, this area is defined as: as the moving object moves, the background area that was occluded in the previous frame and just appeared in this frame. The SVGF motion vector in this area always points to the moving foreground object in the previous frame, resulting in the inability to reuse the time information and severe ghosting. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a denoising method and system for continuous frames rendered by ray tracing, which can remove shading aliasing, over-blurring and tailing phenomena while obtaining a noise-free and temporally stable sequence of frames to obtain high-quality denoising results.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a denoising method for continuous frames of ray tracing rendering, comprising:

[0008] Get the dual motion vectors of the current frame and the denoising result of the previous frame, and calculate the warped reprojection of the previous frame;

[0009] Get the noise map and auxiliary features of the current frame, combine them with the distortion reprojection of the previous frame, and splice them into a tensor;

[0010] Inputting the tensor into a spatiotemporal multiscale denoising network to obtain a denoising result of the current frame; wherein the spatiotemporal multiscale denoising network includes a kernel prediction network and a spatiotemporal multiscale hybrid network;

[0011] The kernel prediction network is used to extract features of the tensor to obtain pixel-by-pixel spatial kernels, pixel-by-pixel temporal kernels, temporal mixing weights, and inter-layer mixing weights;

[0012] The spatiotemporal multiscale mixing network applies a pixel-by-pixel spatial kernel to a multiscale noise map, and uses inter-layer mixing weights to perform weighted summation to obtain a spatial multiscale mixing result; at the same time, the pixel-by-pixel temporal kernel is applied to the distorted reprojection of the previous frame to obtain a temporal filtering result, and the spatial multiscale mixing result and the temporal filtering result are weightedly summed using the temporal mixing weights to obtain a spatiotemporal multiscale mixing result, which is used as the denoising result of the current frame.

[0013] Furthermore, the specific method for calculating the distortion reprojection of the previous frame is:

[0014] Creating a two-dimensional tensor storing pixel coordinates and directly adding the two motion vectors to obtain a mapping relationship between pixel coordinates of the current frame and the previous frame;

[0015] The pixel coordinate mapping relationship is used to quickly align the denoising result of the previous frame with the current frame to obtain a distorted reprojection of the previous frame.

[0016] Furthermore, the loss function of the spatiotemporal multi-scale denoising network adopts the sum of spatial loss, temporal loss and motion occlusion loss.

[0017] Furthermore, the specific calculation method of the motion occlusion loss is:

[0018] Obtain motion vectors of the current frame and multiple adjacent frames, denoising results of the current frame, and a comparison chart;

[0019] Calculating a time-domain progressive weighted mask of the current frame based on motion vectors of the current frame and a plurality of adjacent frames;

[0020] Use Hadamard product to multiply the time domain progressive weighted mask with the denoising result of the current frame and the comparison map according to the matrix corresponding pixels;

[0021] Based on the masked denoising result and the comparison image, the motion occlusion loss is calculated.

[0022] Furthermore, the time-domain progressive weighted mask of the current frame is a weighted sum of multiple masks;

[0023] The mask is the difference between the dual motion vector of the current frame or the adjacent frame and the SVGF motion vector.

[0024] Furthermore, the auxiliary features include albedo, depth and normal of the current frame.

[0025] A second aspect of the present invention provides a denoising system for continuous frames of ray tracing rendering, comprising:

[0026] A distortion reprojection calculation module is configured to: obtain a dual motion vector of a current frame and a denoising result of a previous frame, and calculate a distortion reprojection of the previous frame;

[0027] A splicing module is configured to: obtain the noise map and auxiliary features of the current frame, combine them with the warped reprojection of the previous frame, and splice them into a tensor;

[0028] A spatiotemporal multiscale denoising module is configured to: input the tensor into a spatiotemporal multiscale denoising network to obtain a denoising result of a current frame;

[0029] Among them, the spatiotemporal multiscale denoising network includes a kernel prediction network and a spatiotemporal multiscale hybrid network;

[0030] The kernel prediction network is used to extract features of the tensor to obtain pixel-by-pixel spatial kernels, pixel-by-pixel temporal kernels, temporal mixing weights, and inter-layer mixing weights;

[0031] The spatiotemporal multiscale mixing network applies the pixel-by-pixel spatial kernel to the multiscale noise map, and uses the inter-layer mixing weights to perform weighted summation to obtain a spatial multiscale mixing result; at the same time, the pixel-by-pixel temporal kernel is applied to the warped reprojection of the previous frame to obtain a temporal filtering result; using the temporal mixing weights, the spatial multiscale mixing result and the temporal filtering result are weighted summed to obtain a spatiotemporal multiscale mixing result, and it is used as the denoising result of the current frame.

[0032] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned method for denoising continuous frames of ray tracing rendering.

[0033] The fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the denoising method for continuous frames of ray tracing rendering as described above are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention provides a denoising method for continuous frames of ray tracing rendering, which can be applicable to real-time denoising under sparse MC path tracing indirect lighting. On the premise of obtaining noise-free and temporally stable sequence frames, it can remove shading aliasing, over-blurring and tailing caused by using SVGF motion vectors existing in existing post-processing denoising technologies, and obtain high-quality denoising results.

[0036] The present invention provides a denoising method for continuous frames rendered by ray tracing, which uses a dual motion vector method in the temporal reprojection stage, takes the denoising result of the previous frame as input, uses the dual motion vector to quickly warp it and align it with the current frame, and obtains a more reasonable warped reprojected image of the previous frame than the existing network-based real-time denoising technology.

[0037] The present invention provides a denoising method for continuous frames of ray tracing rendering, which proposes a mask-based motion occlusion loss function, which can make full use of the semantic information of SVGF motion vector and time-reliable dual motion vector, quickly calculate and obtain a time-domain progressive weighted mask, accelerate the convergence of the network in the motion occlusion area, and eliminate the tailing phenomenon in the area. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 is a schematic diagram of dual motion vectors according to the first embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a motion occlusion area obtained by distorted reprojection using different motion vectors according to the first embodiment of the present invention;

[0041] Figure 3 1 is a visualization result of different motion vectors and a generated distortion reprojection result diagram according to the first embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of a mask used in the loss function of the first embodiment of the present invention;

[0043] Figure 5 This is a flow chart of a denoising method for continuous frames rendered by ray tracing according to the first embodiment of the present invention;

[0044] Figure 6 is a diagram of a core prediction network structure of the first embodiment of the present invention;

[0045] Figure 7 It is a comparison diagram of denoising results of the first embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] Embodiment 1

[0050] This embodiment provides a denoising method for continuous frames of ray tracing rendering, which can be divided into four main stages: time reprojection stage, kernel prediction stage, spatiotemporal multi-scale mixing stage, and loop feedback stage. The flowchart is as follows: Figure 5 As shown, the specific steps include:

[0051] Step 1, Temporal reprojection stage: Get the temporal reliable dual motion vector (TRMV) of the current frame and the denoising result of the previous frame, and calculate the warped reprojection of the previous frame The process takes the final denoising result of the previous frame as input, uses temporally reliable dual motion vectors to map the input pixel by pixel, and quickly generates a more reasonable warped reprojected image of the previous frame that is aligned with the current frame, which is used for the input of the neural network and the subsequent pixel-by-pixel weighted mixing.

[0052] S101: Obtain the two-dimensional time-reliable dual motion vector (m) of the current frame from the ray tracing renderer x ,m y ), the sizes of the two dimensions are the resolution of the rendered image (1280×720), and the values ​​are calculated by Figure 1 As shown, it means the difference between the pixel position of the object surface where the shading point is located in the current frame and the pixel position that should be pointed to in the previous frame, where m x and m y The x-axis and y-axis offsets of the pixels in the current frame relative to the previous frame. Figure 1 The dual motion vectors obtained by the calculation method shown will have different results on the occluded area than using the traditional motion vector (SVGF motion vector).

[0053] S102: Get the denoising result of the previous frame from the network's loop feedback loop

[0054] S103: Create a two-dimensional tensor to store pixel coordinates and directly add it to the double motion vector obtained in step S101 to obtain the pixel coordinate mapping relationship between the current frame and the previous frame. According to formula (1), use the grid_sample function and bilinear interpolation mode in the Pytorch framework, use the pixel coordinate mapping just obtained, and add the denoising result of the previous frame obtained in step S102 to the current frame. Quickly align with the current frame to get the distorted reprojection of the previous frame

[0055]

[0056] The key step of the above synthesis method is S101, Figure 1 The calculation methods of SVGF motion vector and time-reliable dual motion vector are shown. Figure 1 As shown, the moving object (blue circle) is moving to the left, and the motion occlusion point x needs to be obtained. i The color of the point. The SVGF motion vector gives the x i →y correspondence (pointing to the moving object in the previous frame). The dual motion vector uses the motion of the occluding object to track the motion of y→z from the previous frame to the current frame. Then, according to x i and z (red dashed arrow), find the position x in the previous frame i-1 , we get x i →x i-1 This process can be simply expressed by formulas (2) to (4), where P is the viewport transformation matrix of each frame multiplied by the model view projection transformation matrix, and T is the geometric transformation matrix between frames.

[0057] x i-1 =y+(x i -z) (2)

[0058]

[0059]

[0060] Figure 2 The difference in motion occluded areas obtained by warping reprojection with two motion vectors is shown; Figure 3It presents this content in a visual way. The traditional motion vector closely follows the edge of the moving object, while the dual motion vector has a gap in the opposite direction of the motion of the occluded object. Therefore, in the motion occluded area, the former (SVGF motion vector) creates a repeated pasting pattern of the moving object, and its color is often very different from the background, while the latter (dual motion vector) reuses the background information to solve the problem of excessive color difference, further reducing the possibility of tailing in the final denoising result.

[0061] The warped reprojection result obtained in step 1 is superior to the results of other existing techniques using SVGF motion vectors because the repeated background regions are usually closer in color to the occluded background regions than the foreground moving objects, providing more effective and correct historical temporal information.

[0062] Step 2: Get the current frame 1spp noise map, geometry buffer (G buffer) auxiliary features (albedo, normal, depth), and combine them with the distortion reprojection of the previous frame obtained in step 1 spliced ​​into a tensor; the tensor is input into the spatiotemporal multi-scale denoising network based on neural network to obtain the denoising result of the current frame. The spatiotemporal multi-scale denoising network includes kernel prediction convolutional networks (KPCN) and spatiotemporal multi-scale hybrid network.

[0063] Using the multi-scale hierarchical kernel prediction network built on the Pytorch framework, the MC path tracing rendering results and geometry buffer (G buffer) based on the Optix framework, plus the distorted reprojected image of the previous frame in the temporal reprojection stage, are used as input. The hierarchical kernel prediction network is used to train the pixel-by-pixel kernel weights and hybrid weights of multi-scale spatiotemporal and temporal images, which are quickly applied to the current frame noisy images and the previous frame distorted reprojected images of different scales, and output high-quality denoised sequences without noise, flickering and smearing. Specifically, it includes:

[0064] Step 201, kernel prediction stage: collect the current frame 1spp noise map, G buffer auxiliary features (albedo, normal, depth) and the distorted reprojection obtained in step 1 The third dimension is concatenated into a (720, 1280, N) tensor as input and sent to the kernel prediction network to extract the features of the input tensor. The last three layers output pixel-by-pixel kernels of the corresponding image size (including pixel-by-pixel spatial kernel, pixel-by-pixel temporal kernel, temporal mixing weights and inter-layer mixing weights). Figure 6The details of the kernel prediction network are shown. The kernel prediction network is an encoder-decoder structure, using a convolutional layer with a kernel size of 3×3 and a Relu activation function. The first half of the layers use 2×2 maximum pooling for downsampling, and the second half of the layers use bilinear interpolation for upsampling. Skip connections are used between layers of the same scale to form a symmetrical U-shaped network structure. The kernel prediction network includes three output layers: the first output layer, i.e., the last layer, outputs (720, 1280, 51×3) kernels, the second output layer, i.e., the second to last layer, outputs (360, 640, 26×3) kernels, and the third output layer, i.e., the third to last layer, outputs (180, 320, 26×3) kernels. Among them, 51 = 5×5 spatial kernel + 5×5 temporal kernel + temporal mixing weight, 26 = 5×5 spatial kernel + inter-layer mixing weight, and 3 is the number of RGB channels.

[0065] Step 202, spatiotemporal multiscale mixing stage: mainly performing spatial hierarchical mixing and temporal mixing. The spatiotemporal multiscale mixing network applies the three-layer pixel-by-pixel spatial kernel obtained in step 201 to the noise map of corresponding size, and uses the inter-layer mixing weights for weighted summation to obtain the spatial multiscale mixing result; at the same time, the pixel-by-pixel temporal kernel is applied to the warped reprojection of the previous frame to obtain the temporal filtering result; the temporal mixing weights are used to weighted sum the spatial multiscale mixing result and the temporal filtering result to obtain the spatiotemporal multiscale mixing result, and it is used as the denoising result of the current frame.

[0066] For spatial mixing, as shown in formula (5), for each inter-layer mixing, c is a relatively coarse scale, f is a relatively fine scale, D is 2×2 maximum pooling downsampling, U is bilinear upsampling, and i is the filtering result after the kernel is applied. Use the three 5×5 pixel-by-pixel spatial kernels obtained from step 201, apply the spatial kernel to the noise map input of the original image size, 1 / 2 the original image size, and 1 / 4 the original image size, respectively, to obtain i, and use the inter-layer mixing weight α to weighted sum to obtain the spatial multi-scale mixing result Specifically, the third output layer and the second output layer are first mixed. For this mixture, the noise map (360, 640, 3) of the second output layer with a size of 1 / 2 of the original image is a relatively fine scale, and the noise map (180, 320, 3) of the third output layer with a size of 1 / 4 of the original image is a relatively coarse scale. The spatial kernel (360, 640, 25×3) obtained by the penultimate output layer of the kernel prediction network is applied to the noise map (360, 640, 3) of the second output layer to obtain i f Similarly, the (180, 320, 25×3) obtained from the third output layer of the kernel prediction network is applied to the third output layer noise map (180, 320, 3) to obtain i c Finally, the inter-layer mixing weight α (180, 320, 1) obtained by the third-to-last output layer of the kernel prediction network is used cMixing (Since the inter-layer mixing weight resolution is relatively coarse, it is necessary to f First downsample and apply weights, then upsample; for i c After applying the weights directly, upsampling is required), and then the mixed results of the third output layer and the second output layer are mixed with the first output layer in a similar layer. The formula is as follows:

[0067]

[0068] Where p is the pixel index of the relatively fine scale, is the value of the relatively fine-scale filtering result at pixel p. c [Di f ]] p is the filtering result of relatively fine scale i f After downsampling, Di f , applying a relatively coarse-scale inter-layer blending weight α c After that, U is upsampled to restore to the fine-scale size U[α c [Di f ]]The value of U[α at pixel p. c [i c ]] p is the filtering result of relatively coarse scale i c When applying a relatively coarse system of inter-layer blending weights α c Afterwards, up-sampling is performed to obtain a relatively fine-scale resolution U[α c [i c ]]At the value of pixel p. The spatial multi-scale mixing result is obtained in the spatial level mixing stage.

[0069] For temporal blending, a 5×5 pixel-by-pixel temporal kernel obtained from step 201 is applied to the warped reprojection obtained in step 1. Get the time filtering result Then use the time mixing weight α obtained in step 201 t , the time filtering result Mixed results with spatial multi-scale Weighted summation to obtain a multi-scale mixed result in time and space (i.e. the final denoising output of the current frame). The formula is as follows:

[0070]

[0071] Step 203, loop feedback stage, according to formula (7), the spatiotemporal multi-scale mixing result obtained in step 202 is subjected to L1 spatial loss and L1 temporal loss (ΔI i =I i -Ii-1 , ΔR i =R i -R i-1 ) and motion occlusion loss L occ , performs back propagation, and stores the mixed result to be used as the next iteration For step 1. That is, the loss function of the spatiotemporal multi-scale denoising network is the sum of spatial loss, temporal loss and motion occlusion loss:

[0072] L=L1(I i ,R i )+L1(ΔI i ,ΔR i )+L occ (7)

[0073] Among them, the motion occlusion loss L occ Specifically, for each frame in a continuous frame sequence, a single-layer mask is obtained using the traditional screen space motion vector of each frame and the time-reliable dual motion vector. The masks of the previous frame and the previous frame are superimposed using a progressively decreasing weight in the time domain, and the loss is calculated for the denoising result of the current frame and the reference image. The calculation process specifically includes:

[0074] (1) For the i-th frame, obtain the denoising result I of the current frame i , 2048spp comparison chart R i , and the following six motion vectors (motion vectors of the current frame and multiple adjacent frames): SVGF motion vector and dual motion vector of the i-th frame, i-1-th frame and i-2-th frame. For some special cases, such as the first frame of the sequence only gets two motion vectors, and the second frame only gets four motion vectors.

[0075] (2) According to Figure 4 The method shown uses the motion vector obtained in step (1) to quickly calculate the mask, wherein the mask is the difference between the dual motion vector of the current frame or the adjacent frame and the SVGF motion vector. Specifically: the mask is obtained by subtracting the SVGF motion vector from the dual motion vector of the i-th frame. Similarly, the mask is obtained by subtracting the SVGF motion vector from the double motion vector of the i-1th frame Subtract the SVGF motion vector from the double motion vector of the i-2th frame to get the mask For some special cases, such as the first frame of the sequence and Directly set to Frame 2 Directly set to

[0076] (3) In order to gradually accumulate in the time domain, the multiple masks obtained from step (2) are and Set different weights ( is 1, is 0.8, is 0.6), and then directly add them to obtain the final time-domain progressive weighted mask for the i-th frame;

[0077] (4) According to formulas (9) and (10), use the Hadamard product ⊙ and apply it to the I obtained in step (1) i , R i Multiply the elements at the same position in the matrix with the time-domain progressive weighted mask obtained in step S203 to obtain the denoising result of the current frame after masking And comparison chart According to formula (8), and Calculate the L1 loss and obtain the motion occlusion loss L OCC , merged into the final loss function.

[0078]

[0079]

[0080]

[0081] Since the occluded area of ​​each frame is too narrow and elongated relative to the entire input tensor, the network cannot spontaneously pay extra attention to these areas, which results in insufficient convergence of the occluded area (at least not as sufficient as other places), resulting in residual trailing visual artifacts. The motion occlusion loss function provided in this embodiment uses a time-domain progressive weighted mask to define the loss calculation range and attention level, which can increase the network's attention to these areas, further accelerate the convergence speed of these areas, and thus eliminate the residual trailing effect.

[0082] In this embodiment, the Pytorch framework, Xavier initialization and Adam optimizer are used, and the initial learning rate is 0.0001. 200 generations are trained each time. The NVIDIA Optix framework is used to obtain a 1spp noise map, a 2048spp comparison map, an auxiliary feature cache (albedo, depth, normal) and two motion vectors.

[0083] The dataset of this embodiment is obtained from different perspectives of 4 scenes, with a total of 281 slices, 142 for training and 139 for testing. Each slice contains 10 consecutive frames, the first 9 frames are used for training, and the last frame is used for verification. The distortion reprojection of the first frame of each slice And the denoising result of the previous frame is initialized to a noise input.

[0084] The graphics processor is NVIDIA GeForce RTX 2080Ti, with a video memory of 12GB. All rendering results of this embodiment are tested in a 1280×720 screen space. The denoising method of this embodiment can be performed at a real-time rate, with an average denoising time of 7.9ms per frame, which can be subdivided into four time-consuming parts: network feature extraction and pixel-by-pixel kernel prediction (3.1ms), kernel application to multi-scale images (3.4ms), mixed spatial and temporal filtering results (0.8ms) and calculation time reprojection (0.6ms).

[0085] Compared with the prior art, the numerical values ​​and results of this embodiment are significantly improved. This embodiment reduces the mean square error MSE to 0.0002, significantly eliminating the over-blurring, color aliasing and tailing problems existing in the prior art. Figure 7 The denoising result of this embodiment is shown.

[0086] The present invention uses a neural network-based spatiotemporal multi-scale network for indirect lighting time and space domain post-processing denoising of MC path tracking under real-time sparse sample count (1spp); removes noise and inter-frame flicker through a spatiotemporal multi-scale kernel prediction neural network, and can obtain a high-quality, noise-free, time-stable continuous frame sequence at a real-time rate, which solves the problems of shading aliasing and over-blurring compared with the prior art; in addition to generating high-quality noise-free images and maintaining temporal stability, a reasonable previous frame distorted reprojection image is generated through a temporally reliable dual motion vector (TRMV), and a motion occlusion loss function is used to reduce the residual tailing phenomenon, which is suitable for dynamic scenes with moving objects in the scene.

[0087] Embodiment 2

[0088] This embodiment provides a denoising system for continuous frames of ray tracing rendering, which specifically includes the following modules:

[0089] A distortion reprojection calculation module is configured to: obtain a dual motion vector of a current frame and a denoising result of a previous frame, and calculate a distortion reprojection of the previous frame;

[0090] A splicing module is configured to: obtain the noise map and auxiliary features of the current frame, combine them with the warped reprojection of the previous frame, and splice them into a tensor;

[0091] A spatiotemporal multiscale denoising module is configured to: input the tensor into a spatiotemporal multiscale denoising network to obtain a denoising result of a current frame;

[0092] Among them, the spatiotemporal multiscale denoising network includes a kernel prediction network and a spatiotemporal multiscale hybrid network;

[0093] The kernel prediction network is used to extract features of the tensor to obtain pixel-by-pixel spatial kernels, pixel-by-pixel temporal kernels, temporal mixing weights, and inter-layer mixing weights;

[0094] The spatiotemporal multiscale mixing network applies the pixel-by-pixel spatial kernel to the multiscale noise map, and uses the inter-layer mixing weights to perform weighted summation to obtain a spatial multiscale mixing result; at the same time, the pixel-by-pixel temporal kernel is applied to the warped reprojection of the previous frame to obtain a temporal filtering result; using the temporal mixing weights, the spatial multiscale mixing result and the temporal filtering result are weighted summed to obtain a spatiotemporal multiscale mixing result, and it is used as the denoising result of the current frame.

[0095] The loss function of the spatiotemporal multi-scale denoising network is the sum of spatial loss, temporal loss and motion occlusion loss.

[0096] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0097] Embodiment 3

[0098] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the denoising method for continuous frames of ray tracing rendering as described in the first embodiment above are implemented.

[0099] Embodiment 4

[0100] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the denoising method for continuous frames of ray tracing rendering as described in the first embodiment above are implemented.

[0101] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A denoising method for continuous frames of ray tracing rendering, characterized in that: include: Get the dual motion vectors of the current frame and the denoising result of the previous frame, and calculate the warped reprojection of the previous frame; Get the noise map and auxiliary features of the current frame, combine them with the distortion reprojection of the previous frame, and splice them into a tensor; Inputting the tensor into a spatiotemporal multiscale denoising network to obtain a denoising result of the current frame; wherein the spatiotemporal multiscale denoising network includes a kernel prediction network and a spatiotemporal multiscale hybrid network; The kernel prediction network is used to extract features of the tensor to obtain pixel-by-pixel spatial kernels, pixel-by-pixel temporal kernels, temporal mixing weights, and inter-layer mixing weights; The spatiotemporal multiscale hybrid network applies a pixel-by-pixel spatial kernel to a multiscale noise map and uses inter-layer hybrid weights for weighted summation to obtain a spatial multiscale hybrid result; at the same time, the pixel-by-pixel temporal kernel is applied to the warped reprojection of the previous frame to obtain a temporal filtering result; Using the time mixing weight, the spatial multi-scale mixing result and the time filtering result are weighted and summed to obtain the spatiotemporal multi-scale mixing result, which is used as the denoising result of the current frame.

2. A denoising method for continuous frames of ray tracing rendering as claimed in claim 1, characterized in that: The specific method for calculating the distortion reprojection of the previous frame is: Creating a two-dimensional tensor storing pixel coordinates and directly adding the two motion vectors to obtain a mapping relationship between pixel coordinates of the current frame and the previous frame; The pixel coordinate mapping relationship is used to quickly align the denoising result of the previous frame with the current frame to obtain a distorted reprojection of the previous frame.

3. A denoising method for continuous frames of ray tracing rendering as claimed in claim 1, characterized in that: The loss function of the spatiotemporal multi-scale denoising network adopts the sum of spatial loss, temporal loss and motion occlusion loss.

4. A denoising method for continuous frames of ray tracing rendering as claimed in claim 3, characterized in that: The specific calculation method of the motion occlusion loss is: Obtain motion vectors of the current frame and multiple adjacent frames, denoising results of the current frame, and a comparison chart; Calculating a time-domain progressive weighted mask of the current frame based on motion vectors of the current frame and a plurality of adjacent frames; Use Hadamard product to multiply the time domain progressive weighted mask with the denoising result of the current frame and the comparison map according to the matrix corresponding pixels; Based on the masked denoising result and the comparison image, the motion occlusion loss is calculated.

5. A denoising method for continuous frames of ray tracing rendering as claimed in claim 4, characterized in that: The time-domain progressive weighted mask of the current frame is a weighted sum of multiple masks; The mask is the difference between the dual motion vector of the current frame or the adjacent frame and the SVGF motion vector.

6. A denoising method for continuous frames of ray tracing rendering as claimed in claim 1, characterized in that: The auxiliary features include albedo, depth and normal of the current frame.

7. A denoising system for continuous frames of ray tracing rendering, characterized in that: include: A distortion reprojection calculation module is configured to: obtain a dual motion vector of a current frame and a denoising result of a previous frame, and calculate a distortion reprojection of the previous frame; A splicing module is configured to: obtain the noise map and auxiliary features of the current frame, combine them with the warped reprojection of the previous frame, and splice them into a tensor; A spatiotemporal multiscale denoising module is configured to: input the tensor into a spatiotemporal multiscale denoising network to obtain a denoising result of a current frame; Among them, the spatiotemporal multiscale denoising network includes a kernel prediction network and a spatiotemporal multiscale hybrid network; The kernel prediction network is used to extract features of the tensor to obtain pixel-by-pixel spatial kernels, pixel-by-pixel temporal kernels, temporal mixing weights, and inter-layer mixing weights; The spatiotemporal multiscale hybrid network applies the pixel-by-pixel spatial kernel to the multiscale noise map and uses the inter-layer hybrid weights for weighted summation to obtain a spatial multiscale hybrid result; at the same time, the pixel-by-pixel temporal kernel is applied to the warped reprojection of the previous frame to obtain a temporal filtering result; Using the time mixing weight, the spatial multi-scale mixing result and the time filtering result are weighted and summed to obtain the spatiotemporal multi-scale mixing result, which is used as the denoising result of the current frame.

8. A denoising system for continuous frame rendering by ray tracing as claimed in claim 7, characterized in that: The loss function of the spatiotemporal multi-scale denoising network is the sum of spatial loss, temporal loss and motion occlusion loss.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a denoising method for continuous frames of ray tracing rendering as described in any one of claims 1 to 6 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a denoising method for continuous frames of ray tracing rendering as described in any one of claims 1-6 are implemented.

Images