Ray tracing multi-frame noise reduction method, electronic device, chip and readable storage medium

Abstract

An embodiment of the present application provides a ray tracing multi-frame denoising method, which relates to the terminal field. By aligning the previous denoised image frame to the current noisy frame, and filling the edge area where the image information is not sampled due to the alignment process with the content of the current noisy frame, the previous denoised image frame and the current noisy frame are merged in a channel series manner and sent to the denoising network for denoising, thereby obtaining the denoising result of the current noisy frame. An embodiment of the present application also provides an electronic device, a chip and a computer-readable storage medium. The present application is based on iterative multi-frames as the input of the denoising network, aligning the denoising result of the previous frame to the current noisy frame, and then merging them in a channel series manner and sending them to the denoising network for denoising, thereby realizing denoising and multi-frame fusion simultaneously using one network inference.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a ray tracing multi-frame noise reduction method, electronic device, chip and computer-readable storage medium. Background Art

[0002] Rasterization and ray tracing (RT) are currently the mainstream graphics rendering technologies. Rasterization is relatively mature and well-integrated with hardware, making it the primary rendering technology currently supported by various graphics cards. Its advantages include fast rendering speeds, making it suitable for scenarios with high real-time requirements, such as gaming. However, its disadvantage is that the rendering effects lack realism. Ray tracing, based on certain physical principles, such as light propagation, reflection, refraction, and energy conservation, solves and renders light rays. Its advantages include realistic rendering effects, even to the point of being indistinguishable to the naked eye. However, its disadvantage is the extremely high computational effort required for rendering.

[0003] Currently, the mainstream implementation of ray tracing is path tracing (PT) combined with Monte Carlo sampling. Therefore, the industry often equates this implementation method with ray tracing. Path tracing treats a single pixel in screen space as a region, randomly selecting locations within that region to launch rays. These rays are then traced based on certain physical principles to ultimately obtain color information and assign it to the corresponding pixel. The sampling process with a certain degree of randomness in this approach is called Monte Carlo sampling, and firing a ray once for each pixel is called a single sample per pixel (SPP). Typically, performing a single Monte Carlo sampling (1 SPP) on the entire image does not guarantee accurate color information for each pixel, so the rendering results of a 1 SPP are often noisy. To obtain accurate color information, the random sampling process must be repeated numerous times, and all the sampling results must be superimposed. However, relying solely on increasing the number of SPPs to achieve high-quality rendering results requires significant computing power. For example, using an Nvidia RTX 2080Ti graphics card to render a 3D model with millions of polygons at 1080p resolution can take tens of minutes to achieve a reasonably good result, significantly falling short of real-time display. Summary of the Invention

[0004] In view of this, it is necessary to provide a ray tracing multi-frame denoising method, which can overcome the above problems, denoise and fuse multi-frames of low-sampling-rate rendering results, and improve image display quality.

[0005] A first aspect of an embodiment of the present application discloses a ray tracing multi-frame denoising method, comprising: in response to a denoising requirement of a video clip, aligning a first denoised image frame to a second image frame of the video clip; determining an edge missing area of the aligned first denoised image frame; filling the edge missing area with image information corresponding to the edge missing area in the second image frame to obtain an aligned image frame; channel-connecting the aligned image frame and the second image frame in series, and inputting the channels into a denoising network to denoise the second image frame to obtain a second denoised image frame; wherein the first denoised image frame is an image frame obtained by denoising the first image frame of the video clip using the denoising network, and the first image frame is the upper N frames of the second image frame, where N is a positive integer.

[0006] By adopting this technical solution, iterative multiple frames are used as the input of the denoising network, the denoising result of the previous frame is aligned to the current noisy frame, and then they are merged using channel series connection and sent to the denoising network for denoising processing to obtain the denoising result of the current noisy frame. Noise reduction and multi-frame fusion can be achieved simultaneously through one network inference.

[0007] In one possible implementation, a video clip is generated by rendering a three-dimensional model by a ray tracing pipeline, and the ray tracing pipeline also generates auxiliary information of the three-dimensional model when rendering the three-dimensional model. The channel concatenation includes one of the following: performing channel concatenation on the aligned image frame and the second image frame; performing channel concatenation on the aligned image frame, the second image frame and the auxiliary information of the three-dimensional model; wherein the auxiliary information includes at least one of the following: normal vector information, world coordinate information, color information, and texture information.

[0008] By adopting this technical solution, it is possible to achieve noise reduction and multi-frame fusion for video clips that also contain auxiliary information of three-dimensional models during rendering using a single network inference.

[0009] In a possible implementation, the ray tracing multi-frame denoising method further includes: selecting a first type of sample image frame from a plurality of sample video clips, wherein each image frame in the sample video clip corresponds to a first type of sample image frame and a second type of sample image frame, and the SPP of the first type of sample image frame is smaller than the SPP of the second type of sample image frame; aligning the first sample denoised image frame of the sample video clip to the second sample image frame of the first type of the sample video clip; determining an edge missing area of the aligned first sample denoised image frame; and filling the edge missing area with image information corresponding to the edge missing area in the second sample image frame of the first type to obtain a sample aligned image. Frame; channel-connecting the sample aligned image frame and the second sample image frame of the first type in series, and inputting them into a preset deep learning network to perform denoising training on the preset deep learning network, wherein the first sample denoised image frame is an image frame obtained by performing denoising on the first sample image frame of the first type using the preset deep learning network, and the first sample image frame of the first type is the upper M frames of the second sample image frame of the first type, where M is a positive integer; calculating the loss function between the second sample denoised image frame output by the preset deep learning network and the second sample image frame of the second type; and using the loss function to perform gradient backpropagation to update the model parameters of the preset deep learning network until the training of the preset deep learning network is completed to obtain the denoising network.

[0010] By adopting this technical solution, iterative multiple frames are used as training inputs of the deep learning network, the denoising result of the previous frame is aligned to the current noisy frame, and then they are merged using channel concatenation and sent to the deep learning network for denoising training to obtain the denoising result of the current noisy frame. The loss function is then calculated based on the denoising result and its corresponding high SPP label image, and the model parameters of the network are updated through gradient backpropagation to achieve the training of the denoising network.

[0011] In one possible implementation, each image frame in the sample video clip corresponds to multiple sample image frames of the first type and one sample image frame of the second type, and the SPPs of the multiple sample image frames of the first type are different from each other. The ray tracing multi-frame denoising method also includes: randomly selecting a sample image frame from the multiple sample image frames of the first type, and combining it with the sample image frame of the second type into a sample data set; wherein the sample image frames of the first type in the sample data set are used as training input data of a preset deep learning network, and the sample image frames of the second type in the sample data set are used as label data of the preset deep learning network.

[0012] By adopting this technical solution, a sample image frame is randomly selected from multiple sample image frames of the first type and combined with the sample image frames of the second type to form a sample data set, so that the trained model will have noise reduction capabilities for a relatively wide range of sampling numbers.

[0013] In one possible implementation, multiple sample video clips are generated by rendering a three-dimensional sample model. When rendering the three-dimensional sample model, auxiliary information of the three-dimensional sample model is also generated, and the sample alignment image frame and the second sample image frame of the first type are channel-connected in series, including: channel-connecting the sample alignment image frame, the auxiliary information of the three-dimensional sample model and the second sample image frame of the first type.

[0014] By adopting this technical solution, it is possible to train sample video clips that also contain auxiliary information of the three-dimensional model when rendering, so that the trained denoising network can perform denoising and multi-frame fusion on the image to be processed containing auxiliary information.

[0015] In one possible implementation, the ray tracing multi-frame denoising method further includes: preprocessing each image frame in the sample video clip; wherein the preprocessing includes replacing abnormal pixel values in the image frame with preset pixel values, and / or removing textures in the image frame.

[0016] By adopting this technical solution, the rendered image frames are preprocessed to avoid the failure of denoising network training caused by abnormal pixel values contained in the image frames.

[0017] In a possible implementation, each image frame in the sample video clip is preprocessed, including: replacing pixels with infinite pixel values in the image frame with zero; or performing nearest neighbor interpolation replacement on pixels with infinite pixel values in the image frame.

[0018] In one possible implementation, aligning a first sample denoised image frame of a sample video clip to a second sample image frame of a first type of the sample video clip includes: using coordinate transformation to determine position information of each pixel in the first sample denoised image frame of the sample video clip in the second sample image frame of the first type of the sample video clip; and using a preset image interpolation algorithm to interpolate pixels whose positions have changed in the first sample denoised image frame of the sample video clip.

[0019] By adopting this technical solution, the previous noise reduction image frame can be aligned to the current noisy image frame.

[0020] In one possible implementation, selecting a first type of sample image frame of a sample video clip from a plurality of sample video clips includes: caching a previous sample noise reduction image frame of each sample video clip; and randomly selecting a current first type of sample image frame of a sample video clip from the plurality of sample video clips.

[0021] By adopting this technical solution, a noise reduction cache mechanism and frame-level random switching are used to select the current image frame to be processed, so that the input received by the network during the training phase does not have to come from the same sample video clip each time, thereby solving the negative impact of network memory.

[0022] In one possible implementation, the image information corresponding to the edge missing area in the second sample image frame of the first type is filled into the edge missing area to obtain a sample aligned image frame, including: filling the image information corresponding to the edge missing area in the second sample image frame of the first type into the edge missing area; randomly selecting an image area from the aligned first sample denoised image frame, and filling the image information corresponding to the randomly selected image area in the second sample image frame of the first type into the randomly selected image area to obtain a sample aligned image frame.

[0023] By adopting this technical solution, the areas where no information was sampled in the previous denoised image frame due to alignment and some other random areas are filled with the content of the current noise frame, which can solve the problem of inconsistent residual noise in edge areas or other areas caused by image alignment.

[0024] In one possible implementation, the image information corresponding to the edge missing area in the second sample image frame of the first type is filled into the edge missing area to obtain a sample aligned image frame, including: filling the edge missing area with preset image information; randomly selecting an image area from the aligned first sample denoised image frame, and filling the preset image information into the randomly selected image area to obtain the sample aligned image frame.

[0025] By adopting this technical solution, the areas where no information was sampled in the previous denoised image frame due to alignment and some other random areas are filled with default image content, which can solve the problem of inconsistent residual noise in edge areas or other areas caused by image alignment.

[0026] In one possible implementation, the sample-aligned image frame and the second sample image frame of the first type are channel-concatenated and input into a preset deep learning network to perform noise reduction training on the preset deep learning network, including: channel-concatenating the sample-aligned image frame and the second sample image frame of the first type to obtain a concatenated image frame; randomly cutting an image area of a preset size from the concatenated image frame and inputting the image area into the preset deep learning network to perform noise reduction training on the preset deep learning network.

[0027] By adopting this technical solution, the deep learning network is trained by inputting image blocks randomly cut out of fixed sizes, which can avoid the increased complexity of training caused by image frames of different sizes.

[0028] In one possible implementation, the ray tracing multi-frame denoising method further includes: defining lens parameters of a scene lens, the lens parameters including a lens movement trajectory and a lens orientation; rendering a three-dimensional sample model based on the lens parameters using a ray tracing pipeline to obtain a plurality of sample video clips; wherein the three-dimensional sample model includes a static three-dimensional model or a dynamic three-dimensional model.

[0029] By adopting this technical solution, it is possible to render multiple sample video clips by means of lens roaming.

[0030] In one possible implementation, a ray tracing pipeline is used to render a three-dimensional sample model based on lens parameters to obtain multiple sample video clips, including: using a ray tracing pipeline to render a three-dimensional sample model based on lens parameters to obtain multiple sample videos; randomly sampling frames and / or randomly reversing the sample videos to combine them to obtain sample video clips.

[0031] By adopting this technical solution, data enhancement processing can be performed on multiple sample videos obtained by rendering to obtain multiple sample video clips.

[0032] In one possible implementation, the ray tracing multi-frame denoising method further includes: testing the denoising network using a test set; when the denoising network is determined to meet preset denoising requirements through the test set, denoising the video clip based on the denoising network; wherein each test data in the test set includes a first type of sample image frame and a second type of sample image frame.

[0033] By adopting this technical solution and using high SPP image frames as the image quality evaluation standard, the denoising results obtained by the denoising network are tested and evaluated to determine whether the denoising network meets the preset denoising requirements.

[0034] In a second aspect, an embodiment of the present application provides a ray tracing multi-frame denoising method, comprising: selecting a first type of sample image frame of a sample video clip from a plurality of sample video clips, wherein each image frame in the sample video clip corresponds to a first type of sample image frame and a second type of sample image frame, and the SPP of the first type of sample image frame is smaller than the SPP of the second type of sample image frame; aligning the first sample denoised image frame of the sample video clip to the second sample image frame of the first type of the sample video clip; determining an edge missing area of the aligned first sample denoised image frame; filling the edge missing area with image information corresponding to the edge missing area in the second sample image frame of the first type to obtain a sample aligned image frame; channeling the sample aligned image frame with the second sample image frame of the first type. The images are connected in series and input into a preset deep learning network to perform noise reduction training on the preset deep learning network, wherein the first sample denoised image frame is an image frame obtained by performing noise reduction processing on the first sample image frame of the first type using the preset deep learning network, and the first sample image frame of the first type is the upper M frames of the second sample image frame of the first type, where M is a positive integer; the loss function between the second sample denoised image frame output by the preset deep learning network and the second sample image frame of the second type is calculated; the loss function is used to perform gradient backpropagation to update the model parameters of the preset deep learning network until the training of the preset deep learning network is completed to obtain the noise reduction network; the noise reduction network is tested using a test set; when it is determined through the test set that the noise reduction network meets the preset noise reduction requirements, the image frame to be processed is subjected to noise reduction processing based on the noise reduction network.

[0035] By adopting this technical solution, iterative multiple frames are used as training inputs of the deep learning network, the denoising result of the previous frame is aligned to the current noisy frame, and then they are merged using a channel series method and sent to the deep learning network for denoising training to obtain the denoising result of the current noisy frame. The loss function is then calculated based on the denoising result and its corresponding high SPP label image, and the model parameters of the network are updated through gradient backpropagation to achieve the denoising network obtained through training, and the image frame to be processed is denoised based on the denoising network obtained through training.

[0036] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the ray tracing multi-frame denoising method as described in the first aspect or the second aspect.

[0037] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, so that the electronic device executes the ray tracing multi-frame denoising method as described in the first aspect or the second aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the ray tracing multi-frame denoising method as described in the first aspect or the second aspect.

[0039] In a sixth aspect, an embodiment of the present application provides a chip coupled to a memory in an electronic device, and the chip is used to control the electronic device to execute the ray tracing multi-frame denoising method as described in the first aspect or the second aspect.

[0040] In a seventh aspect, embodiments of the present application provide a device that implements the electronic device behavior described in the methods of the first or second aspects. The functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functionality described above.

[0041] It can be understood that the computer-readable storage medium described in the third aspect, the electronic device described in the fourth aspect, the computer program product described in the fifth aspect, the chip described in the sixth aspect, and the device described in the seventh aspect all correspond to the methods of the first aspect or the second aspect above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of an application environment for the ray tracing multi-frame denoising method provided in one embodiment of the present application;

[0043] Figure 2 A schematic flow chart of a ray tracing multi-frame denoising method provided in one embodiment of the present application;

[0044] Figure 3 A schematic diagram illustrating the principle of a ray tracing multi-frame denoising method provided in one embodiment of the present application;

[0045] Figure 4 A flowchart of a method for training a noise reduction network according to an embodiment of the present application is provided;

[0046] Figure 5 A schematic diagram illustrating the principle of generating training data for a noise reduction network according to an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the training principle of a noise reduction network provided in one embodiment of the present application;

[0048] Figure 7 A schematic diagram of the training principle of a preset deep learning network provided in one embodiment of the present application;

[0049] Figure 8A schematic diagram of the structure of a possible electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0051] For ease of understanding, some concepts related to the embodiments of the present application are exemplarily described for reference:

[0052] Rasterization: This refers to the process of converting vertex data into fragments. This converts raw image data into an image composed of a grid, where each element in a fragment corresponds to a pixel in the frame buffer. Rasterization is the process of converting geometric primitives into a two-dimensional image.

[0053] Path tracing: A rendering method proposed by Kajiya in 1986, it is the first unbiased rendering method. The basic idea of path tracing is to send a ray from the viewpoint. When the ray intersects the surface of an object, it continues to sample a direction based on the surface's material properties and sends another ray. This iterative process continues until the ray hits a light source (or escapes the scene). The Monte Carlo method is then used to calculate the ray's contribution as the pixel's color value. The Monte Carlo method is unbiased in solving the integral, and as long as sufficient time is given, the final image will converge to a correct result.

[0054] Ray tracing: A method for presenting three-dimensional (3D) images on a two-dimensional (2D) screen. Rays are emitted from the viewpoint through each pixel on the viewing plane, and the intersection of rays with objects is continuously determined, while taking into account optical phenomena such as reflection and refraction to render the three-dimensional scene.

[0055] Monte Carlo sampling: A random sampling process that uses the superposition of a large number of repeated random samples to make the results converge to the true solution.

[0056] Channel concatenation: Image data generally consists of three dimensions: height, width, and channels. By concatenating multiple image data in the channel dimension, a new data body is formed with the same height and width and the sum of the number of channels of the original multiple images.

[0057] Iterate multiple frames: Use the processed results of the previous frame and the current unprocessed frame as the input of the subsequent processing model, and obtain the processed result of the current frame after processing.

[0058] Neural network inference: The process of inputting data into a neural network and obtaining output through the network's forward operation.

[0059] Generalization: It is used to describe the scope of application of an algorithm. The wider the scope of application, the better the generalization; conversely, the worse the generalization.

[0060] Ghosting: refers to the situation where part of the rendering results of the previous frame still remain in the subsequent frames, causing the object to appear as a ghost in the picture.

[0061] In some animation, game, and movie scenes, the realism of 3D scenes can be enhanced through image rendering. Image rendering denoising generally uses an iterative multi-frame scheme for denoising. The existing scheme generally aligns the denoising result of the previous frame to the current noisy frame through coordinate space transformation, and then inputs the aligned previous frame and current noisy frame into the neural network model for denoising respectively. The two denoising results output by the model are then added and fused to obtain the denoising result of the current frame. However, this scheme requires two neural network inferences, which significantly increases the amount of calculation, and it is difficult to achieve a good fusion effect by using a simple addition method for multi-frame fusion. Therefore, it is necessary to provide a ray tracing multi-frame denoising method, which aligns the denoising result of the previous frame to the current noisy frame, and then uses a channel series method to merge them and feed them into a deep learning network, so as to achieve denoising and multi-frame fusion simultaneously using a single network inference.

[0062] Reference Figure 1 FIG2 is a schematic diagram of an application environment of a ray tracing multi-frame denoising method provided by an embodiment of the present application. The embodiment may include: an electronic device 100 and a server 200. The electronic device 100 may be a terminal capable of performing image rendering.

[0063] In some embodiments, the ray tracing multi-frame denoising method provided in the embodiments of the present application can be applied to the electronic device 100, and the electronic device 100 can render the three-dimensional model based on the ray tracing and rendering engine, and display the rendering result. The number of electronic devices 100 can be one or more, and each electronic device 100 can independently perform the model rendering operation. The electronic device 100 and the server 200 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. For example, the server 200 is a game server, and the electronic device 100 logs in to the server 200 via wireless communication.

[0064] In some embodiments, the server 200 may include at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. The server 200 may undertake the primary computing work, and the electronic device 100 may undertake the secondary computing work; or, the server 200 may undertake the secondary computing work, and the electronic device 100 may undertake the primary computing work; or, the electronic device 100 and the server 200 may perform collaborative computing using a distributed computing architecture. The device types of the electronic device 100 include, but are not limited to, at least one of: a smartphone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart watch, a laptop computer, or a desktop computer.

[0065] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0066] Reference Figure 2 FIG. 1 is a flow chart of a ray tracing multi-frame denoising method provided in an embodiment of the present application. The ray tracing multi-frame denoising method can be applied to an electronic device 100. In this embodiment, the ray tracing multi-frame denoising method may include:

[0067] 21. In response to a noise reduction requirement of the video clip, align the first noise-reduced image frame to the second image frame of the video clip.

[0068] In some embodiments, a video clip may be generated by rendering a three-dimensional model through a ray tracing pipeline. The three-dimensional model may refer to the three-dimensional model to be rendered this time. The three-dimensional model may correspond to at least one object, which may be a static object or a dynamic object. For example, the three-dimensional model is a three-dimensional model of a single object in a virtual scene (such as a game scene), or a three-dimensional model of multiple objects in a virtual scene. The ray tracing pipeline may render the three-dimensional model in a camera roaming manner and generate image frames. When the ray tracing pipeline continuously renders the three-dimensional model, a video clip may be generated.

[0069] Taking a game scene as an example, it includes one or more virtual objects. The lens parameters of the viewing angle lens can be predefined, and the ray tracing pipeline can continuously render the 3D model based on these lens parameters to produce a video clip. This video clip includes multiple frames of game screen. Lens parameters can include lens movement trajectory and lens orientation. To avoid game screen lag, the real-time performance of the game screen is relatively high. Relying on high-SPP (e.g., greater than 3000 SPP) image frames to achieve high-quality visual display requires significant computing power and a long processing time, which cannot meet the real-time display requirements of the game scene. Therefore, the video clips generated by the ray tracing pipeline when rendering the 3D model contain low-SPP (e.g., less than 50 SPP) image frames, resulting in poor image quality. By performing noise reduction on the video clips, high-quality visual display can be achieved, making the game scene appear realistic, even to the point of being difficult to distinguish with the naked eye.

[0070] In some embodiments, the first denoised image frame may be a denoised result obtained by performing denoising processing on the first image frame of a video clip. The first image frame is the Nth image frame preceding the second image frame, where N is a positive integer. For example, the first image frame may be the image frame preceding the second image frame, or the first image frame may be the image frame two frames preceding the second image frame. The following description uses the example of the first image frame being the image frame preceding the second image frame.

[0071] In a video frame sequence, the perspective change between two adjacent frames is typically small, so most pixels from the previous frame can be reused. However, due to the perspective change, the same physical elements in the two frames are often located at different pixel positions, making them unusable. Their pixel positions must be aligned before they can be reused. This can be achieved by using coordinate transformation to align the first denoised frame with the second.

[0072] For example, the process of aligning the first denoised image frame to the second image frame may include first finding the corresponding position of each pixel in the first denoised image frame in the second image frame through coordinate transformation, and then interpolating the pixels in the first denoised image frame that have changed position through image interpolation. The coordinate transformation can be implemented using motion vectors or matrix transformations.

[0073] In some embodiments, for the first frame of a video clip, the first noise reduction image frame may be replaced by the first frame.

[0074] 22. Determine the edge missing area of the first denoised image frame that has undergone the alignment process.

[0075] In some embodiments, when a first denoised image frame is aligned to a second image frame through coordinate transformation, the aligned first denoised image frame rotates relative to the pre-aligned first denoised image frame, causing portions of its edge regions to extend beyond the image display range and lack image information. These portions of the edge regions lacking image information are defined as edge-missing regions. The edge-missing regions can be located based on the size and rotation information of the first denoised image frame, facilitating subsequent image filling in these edge-missing regions.

[0076] 23. Fill the edge missing area with image information corresponding to the edge missing area in the second image frame to obtain an aligned image frame.

[0077] In some embodiments, when missing edge regions are determined in the first denoised image frame that has undergone alignment, image information corresponding to the missing edge regions in the second image frame can be used to fill these missing edge regions, thereby obtaining an aligned image frame. For example, if image region A1 in the second image frame corresponds to the missing edge region, the image information of image region A1 can be used to fill these missing edge regions. The first denoised image frame that has undergone alignment and completed the filling of the missing edge regions is defined as the aligned image frame.

[0078] 24. Connect the aligned image frame and the second image frame in series, and input the channels into a denoising network to perform denoising on the second image frame to obtain a second denoised image frame.

[0079] In some embodiments, image data generally consists of three dimensions: height, width, and channels. Channel-wise concatenation of the aligned image frame and the second image frame may refer to concatenating the aligned image frame and the second image frame in the channel dimension to form an image data volume with unchanged height and width and a number of channels equal to the sum of the number of channels of the two image frames. This allows the aligned image frame and the second image frame to be combined as input to a denoising network, and network inference is performed to obtain a denoised result for the second image frame, i.e., a second denoised image frame. Similarly, the above denoising process can be iteratively performed on all image frames in the video clip to obtain denoised results for all image frames.

[0080] In some embodiments, the noise reduction network can be obtained based on convolutional neural network training, and the training data can be multiple sample video clips. The specific training and testing methods are described in detail below.

[0081] In some embodiments, when rendering a 3D model, the ray tracing pipeline can also obtain auxiliary information about the 3D model. The auxiliary information can include at least one of the following: normal vector information, world coordinate information, color information, and texture information. In this case, the aligned image frame, the second image frame, and the auxiliary information of the 3D model can be connected in series and combined to serve as input to the denoising network to obtain a second denoised image frame.

[0082] The above-mentioned ray tracing multi-frame denoising method uses iterative multi-frames as the input of the denoising network, aligns the denoising result of the previous frame to the current noisy frame, and then merges them using channel concatenation to send them together into the denoising network for denoising processing, obtaining the denoising result of the current noisy frame. Noise reduction and multi-frame fusion can be achieved simultaneously through a single network inference.

[0083] Reference Figure 3 Figure 2 is a schematic diagram illustrating the principles of a ray tracing multi-frame denoising method according to an embodiment of the present application. A ray tracing pipeline continuously renders a 3D model to generate a video clip. The following description uses the M-1th, M-th, and M+1th frames in the video clip as examples, where M is a positive integer greater than 1. The denoising result for the M-1th frame is the M-1th denoised frame, and the denoising result for the Mth frame is the Mth denoised frame.

[0084] 31. Image alignment and edge filling: Align the M-1th denoised image frame to the Mth image frame, and fill the edge areas of the aligned M-1th denoised image frame that have not been sampled with image information due to the alignment process with the corresponding content of the Mth image frame to obtain an aligned image frame.

[0085] 32. Channel concatenation: concatenate the aligned image frame and the Mth image frame through channels.

[0086] 33. Noise reduction processing: The result obtained by channel concatenation is input into the noise reduction network to obtain the Mth denoised image frame.

[0087] It can be understood that when the Mth denoised image frame is obtained, the Mth denoised image frame can be aligned to the M+1th image frame, and the edge area of the aligned Mth denoised image frame that has not sampled image information due to the alignment process can be filled with the corresponding content in the M+1th image frame, and the channel concatenation and denoising process can be repeated to obtain the M+1th denoised image frame.

[0088] In some embodiments, the M-1th, Mth, and M+1th denoised image frames obtained through the denoising network can be used for network training or sent to a related hardware device (such as a display screen) for display. If used for network training, the denoised image frames can be used together with the high SPP image frames to calculate the loss function, and the network model parameters can be updated through gradient backpropagation to achieve the network training purpose.

[0089] Reference Figure 4 FIG. 1 is a flow chart showing the steps of a method for training a denoising network according to an embodiment of the present application.

[0090] 41. Select a sample image frame of a first type from a plurality of sample video clips.

[0091] In some embodiments, a ray tracing pipeline can continuously render one or more 3D sample models within a preset scene (e.g., a game scene) to generate multiple sample video clips. Each image frame in the sample video clip corresponds to a first type of sample image frame and a second type of sample image frame. The SPP of the first type of sample image frame is smaller than the SPP of the second type of sample image frame.

[0092] In some embodiments, random frame sampling and / or random frame reversal can be performed on each sample video clip to combine and obtain multiple new sample video clips, thereby implementing data augmentation processing on the sample video clips. For example, multiple image frames participating in training in a sample video clip can be randomly selected to obtain a video clip sequence, and the video clip sequence can be reversed to obtain a new sample video clip.

[0093] In some embodiments, during the training phase of the denoising network, a plurality of low-SPP image frames (defined as sample image frames of the first type) may be pre-generated as network training inputs, and a high-SPP image frame (defined as a sample image frame of the second type) may be pre-generated as a training label. The plurality of low-SPP image frames and the high-SPP image frame may have the same image content.

[0094] For example, for each image frame in a sample video clip, multiple low-SPP image frames and one high-SPP image frame can be repeatedly generated. Due to the randomness of Monte Carlo sampling, even if the rendering perspective remains unchanged, the noise contained in the multiple rendered low-SPP image frames will vary, which can serve as data enhancement. The number of samples of low-SPP image frames can also vary, so that the trained denoising network will have noise reduction capabilities over a relatively wide sampling range. For example, the number of samples of low-SPP image frames can be 1SPP, 2SPP, 4SPP, 8SPP, etc. Since high-SPP image frames are used as training labels to guide network learning, they should be sufficiently clear. For example, high-SPP image frames are 4096SPP image frames.

[0095] For example, each image frame in the sample video clip may correspond to six low SPP image frames, namely, 1SPP image frame, 2SPP image frame, 4SPP image frame, 8SPP image frame, 16SPP image frame, and 32SPP image frame. Each image frame in the sample video clip also corresponds to a 4096SPP image frame.

[0096] In some embodiments, during the training phase of the denoising network, one image frame can be randomly selected from multiple low-SPP image frames and combined with a high-SPP image frame to form a sample dataset for training a pre-defined deep learning network, such as a convolutional neural network including an encoding-decoding framework. In other embodiments, each low-SPP image frame can be combined with a high-SPP image frame to obtain multiple sample datasets, which are then used to train the pre-defined deep learning network. For example, a 1SPP image frame can be combined with a 4096SPP image frame to obtain a first sample dataset, and a 2SPP image frame can be combined with a 4096SPP image frame to obtain a second sample dataset.

[0097] In some embodiments, a preset deep learning network is subjected to multi-frame iterative training to obtain a noise reduction network, generally using multiple sample video clips as training data. The existing training scheme is to complete training on a sample video clip and then randomly switch to another sample video clip to continue training. This training method may cause the network to only see the content of a certain sample video clip within a certain period of time. Since the content of a sample video clip is often highly repetitive and similar, it will amplify the negative impact of the memory of network learning. The memory of network learning can refer to the network's strong ability to fit the data of repeated training, and the effect is improved, but at the cost of a significant decrease in the network's effect on other data. The memory problem of network learning will make it difficult for the network to converge to a sufficiently good level, affecting the network's final noise reduction effect and generalization.

[0098] For example, when the preset deep learning network receives video clips in the following order near the end of training: sample video clip C2, sample video clip C1, and sample video clip C3, the trained denoising network will have a better denoising effect on sample video clip C3, an average denoising effect on sample video clip C1, and a poor denoising effect on sample video clip C2. Similarly, the further away from the end of training, the worse the effect. During the application phase of the denoising network, if the data being inferred is rendered from a 3D model that the denoising network has never seen, the denoising effect will be even worse. This solution, by using frame-level random switching to select sample data sets, can avoid the negative impact of the network's learning memory.

[0099] Specifically, for multiple sample video clips participating in training (three sample video clips C1, C2, and C3 are used as an example for illustration; the number of sample video clips can be more or less than three), the previous denoised image frame of each sample video clip C1-C3 can be cached, and then the current image frame to be processed of one of the sample video clips is randomly obtained, and it is passed to the subsequent data preprocessing, data enhancement, and training processes together with the corresponding previous denoised image frame. The embodiment provided by the present application designs a caching mechanism for the previous denoised image frame, so that the sample input received by the network each time during the training phase does not need to come from the same sample video clip, thereby solving the negative impact of the memory of network learning.

[0100] For example, sample video segment C1 contains sample image frames: C1-1, C1-2, C1-3, C1-4, C1-5; sample video segment C2 contains sample image frames: C2-1, C2-2, C2-3, C2-4, C2-5; and sample video segment C3 contains sample image frames: C3-1, C3-2, C3-3, C3-4, C3-5. Existing training schemes select sample image frames for training by randomly switching to another sample video segment after training is completed on one sample video segment. That is, the order of selecting sample image frames may be: C1-1, C1-2, C1-3, C1-4, C1-5, C3-1, C3-2, C3-3, C3-4, C3-5, C2-1, C2-2, C2-3, C2-4, C2-5. The order of selecting sample image frames using the frame-level random switching technology of this scheme may be: C1-1, C1-2, C3-1, C3-2, C2-1, C1-3, C2-2, C2-3, C3-3, C1-4, C2-4, C3-4, C1-5, C3-5, C2-5.

[0101] In some embodiments, when the ray tracing pipeline renders a three-dimensional sample model to generate an image frame, some pixels in the image frame may not obtain valid values. For example, for auxiliary information such as the normal vector and world coordinates of the three-dimensional sample model, the values of these pixels only retain the initialized default values. The default value may be set to infinity (inf), and these image frames need to be preprocessed. Because the result of multiplying or adding inf with any non-zero number is inf, and the main computational operations of the deep learning network are multiplication and addition, once inf enters the network, the network parameters may be contaminated to inf after a gradient back propagation, causing network training failure. For pixel values of inf, zeroing or neighborhood interpolation can be used. Specifically, when the current image frame to be processed of one of the sample video clips is randomly obtained, the image frame can be preprocessed to avoid the occurrence of pixels with inf values. The preprocessed image frame and the previous denoised image frame corresponding to the image frame are then sent to the subsequent training process.

[0102] In some embodiments, part of the area in the image frame generated by the ray tracing pipeline for rendering the three-dimensional sample model may be a texture (background image). The texture does not contain Monte Carlo sampling noise and does not require noise reduction processing. Therefore, the texture can be cropped from the image frame before training to prevent it from participating in the subsequent training process.

[0103] In some embodiments, as Figure 5 Figure 2 shows a schematic diagram of the principles for generating training data for a denoising network. A 3D sample model within a game scene can be rendered to generate multiple sample video clips. The 3D sample model can be a static or dynamic model. Before rendering, lens parameters can be designed for the 3D sample model. This lens parameter design can include lens trajectory movement and lens orientation. Based on the designed lens parameters, two image frames each with 1SPP, 2SPP, 4SPP, 8SPP, 16SPP, and 32SPP (low SPP) can be rendered as noise data, and one image frame with 4096SPP (high SPP) can be rendered as a training label. If the network requires auxiliary information as training input, such as the model normal vector, color information, and texture information of the 3D sample model, this auxiliary information should be generated simultaneously with the rendering of the noise data. After rendering these types of SPP image data, outlier processing can be performed to check for outliers such as inf. If present, these outliers can be reset to zero. This outlier check can be completed before network training to avoid affecting the subsequent network training speed.

[0104] 42. Align a first sample denoised image frame of the sample video clip to a second sample image frame of the first type of the sample video clip.

[0105] In some embodiments, the first sample denoised image frame is an image frame obtained by performing denoising processing on a first sample image frame of a first type using a preset deep learning network. The first sample image frame of the first type is the image frame preceding the second sample image frame of the first type by M frames, where M is a positive integer. For example, the first sample image frame of the first type is the image frame preceding the second sample image frame of the first type, or is the image frame two frames preceding the second sample image frame of the first type.

[0106] In a sample video frame sequence, the perspective change between two adjacent sample frames is typically small, so most pixels in the previous sample frame can be reused. However, due to the perspective change, the same physical elements in the two sample frames are often located at different pixel positions, making them inaccessible. Their pixel positions must be aligned before they can be reused. Specifically, coordinate transformation can be used to align the first sample denoised image frame with the second sample image frame of the first type.

[0107] 43. Determine an edge missing region of the first sample denoised image frame that has undergone alignment processing.

[0108] In some embodiments, when a first sample denoised image frame is aligned to a second sample image frame of the first type through coordinate transformation, the aligned first sample denoised image frame rotates relative to the pre-aligned first sample denoised image frame, causing portions of its edge regions to exceed the image display range and lack image information. The missing edge regions can be located based on the size and rotation information of the first sample denoised image frame to facilitate subsequent image filling in these missing edge regions.

[0109] 44. Fill the edge missing region with image information corresponding to the edge missing region in the second sample image frame of the first type to obtain a sample aligned image frame.

[0110] In some embodiments, the image information corresponding to the missing edge region in the second sample image frame of the first type can be used to fill the missing edge region to obtain a sample-aligned image frame. For example, image region B1 in the second sample image frame of the first type corresponds to the missing edge region, and the image information of image region B1 is then filled into the missing edge region. The first sample denoised image frame that has undergone the alignment process and completed the filling of the missing edge region is defined as the sample-aligned image frame.

[0111] 45. Channel-concatenate the sample aligned image frame and the second sample image frame of the first type, and input the resulting image frames into a preset deep learning network to train the preset deep learning network for noise reduction processing.

[0112] In some embodiments, the sample aligned image frame and the second sample image frame of the first type can be channel-concatenated to form a sample image data body with unchanged height and width and the number of channels being the sum of the number of channels of the two image frames. The sample image data body and the second sample image frame of the second type can be used together as training data for a preset deep learning network, and the second sample image frame of the second type is used as a training label.

[0113] In some embodiments, for the first sample denoised image frame that has completed filling of missing edge regions, one or more image regions may be randomly selected from the first sample denoised image frame and filled with the content of the second sample image frame of the first type to obtain a sample-aligned image frame. In other words, the first sample denoised image frame that has undergone alignment processing and completed filling of missing edge regions and random regions is defined as the sample-aligned image frame.

[0114] In some embodiments, operations such as channel shuffling and color randomization can be performed synchronously on the second sample image frames of the first type, the sample-aligned image frames, and the second sample image frames of the second type to achieve data enhancement. Synchronously performing channel shuffling (e.g., randomly adjusting the channel order of the image) and color randomization (e.g., randomly performing grayscale transformations on each channel of the image, such as linear transformation, piecewise linear transformation, gamma transformation, etc.) on the second sample image frames of the first type, the sample-aligned image frames, and the second sample image frames of the second type can mean that the three channel shuffling and color randomization operations must be performed simultaneously and using the same parameters.

[0115] In some embodiments, for the sample image data body obtained by channel concatenation and the second sample image frame of the second type, data enhancement processing such as random slicing (for example, image blocks randomly cut out of a fixed size), random rotation (for example, randomly rotating the image at a certain angle), and random flipping (for example, randomly flipping the image left and right or flipping the upper limit) can also be performed simultaneously. In order to adapt to the up and down sampling modules in the preset deep learning network, the image frame size obtained by random slicing can be optionally an exponential power of 2, where the exponent should be greater than the number of up and down sampling of the network. For example, if the number of up and down sampling on the network is 3, the width and height of the image frame obtained by random slicing can be set to 64*64, or 128*128, etc.

[0116] In some embodiments, when a ray tracing pipeline renders a 3D sample model, it can also obtain auxiliary information about the 3D sample model. The auxiliary information can include at least one of the following: normal vector information, world coordinate information, color information, and texture information. In this case, the sample alignment image frame, the second sample image frame of the first type, and the auxiliary information of the 3D sample model can be channel-concatenated.

[0117] 46. Calculate a loss function between a second sample denoised image frame output by a preset deep learning network and a second sample image frame of the second type.

[0118] In some embodiments, when the sample image data volume and the second sample image frame of the second type are used together as training data for a preset deep learning network, noise reduction training can be implemented for the preset deep learning network. A forward pass through the network yields a noise reduction result for the second sample image frame of the first type, i.e., the second sample denoised image frame. The loss function is then calculated using the second sample denoised image frame and the second sample image frame of the second type. The loss function can be calculated using existing calculation methods and is not limited herein.

[0119] In some embodiments, the size of the output of the preset deep learning network may be the same as the size of the second sample image frame of the second type.

[0120] 47. Use the loss function to perform gradient backpropagation to update the model parameters of the preset deep learning network until the training of the preset deep learning network is completed to obtain the denoising network.

[0121] In some embodiments, when the loss function is calculated using the second sample denoised image frame and the second sample image frame of the second type, the loss function can be used to perform gradient backpropagation to update the network parameters of the preset deep learning network until the loss function basically no longer decreases, thereby completing the training of the preset deep learning network and obtaining the denoising network.

[0122] In some embodiments, after training a preset deep learning network to obtain a denoising network, the denoising network can be tested using a test set. The test data in the test set can be pre-constructed, and each test data in the test set can include a low-SPP image frame and a high-SPP image frame. The high-SPP image frame is used as a criterion for image quality assessment. For example, a test data in the test set includes a 2SPP image frame P1 and a 4096SPP image frame P2. The 2SPP image frame P1 is input into the denoising network to obtain a denoised image frame P11 corresponding to the 2SPP image frame. Using image frame P2 as the evaluation criterion, the denoised image frame P11 is subjected to an image quality assessment to determine the denoising effect of the denoising network.

[0123] In some embodiments, the denoising network can be used to determine whether the denoising effect meets the preset requirements based on the image quality evaluation results of each test data in the test set. When the denoising effect of the denoising network is determined to meet the preset requirements, the denoising network can be used to Figure 2 In the ray tracing multi-frame denoising method shown in FIG. , for example, when the image quality assessment results of more than 95% of the test data in the test set are passed, it is determined that the denoising effect of the denoising network meets the preset requirements.

[0124] In some embodiments, when it is determined that the noise reduction effect of the noise reduction network does not meet the preset requirements, the network parameters can be adjusted, and training and testing can be re-performed until it is determined that the noise reduction effect of the noise reduction network meets the preset requirements.

[0125] Reference Figure 6 , which is a schematic diagram of the training principle of the noise reduction network provided in one embodiment of the present application.

[0126] The ray tracing pipeline continuously renders a 3D sample model to generate a sample video clip. The following description uses the P-1th sample image frame, the Pth sample image frame, and the P+1th sample image frame as examples, where P is a positive integer greater than 1. The denoising result of the P-1th sample image frame is the P-1th sample denoised image frame, and the denoising result of the Pth sample image frame is the Pth sample denoised image frame. The P-1th sample image frame, the Pth sample image frame, and the P+1th sample image frame are all low-SPP image frames, while the Pth labeled image frame is a high-SPP image frame. The Pth sample image frame and the Pth labeled image frame contain the same image content, differing only in their SPP values.

[0127] 61. Performing a first data augmentation on the sample video clip: Performing a first data augmentation process on the rendered sample video clip. This first data augmentation process can be used to select training data, such as selecting the Pth sample image frame as the current training data. This first data augmentation process can include random resampling, random reversing, and random frame-level switching of the sample video clip.

[0128] 62. Image Alignment and Edge Padding: Align the P-1th sample denoised image frame to the Pth sample image frame and perform edge padding. This edge padding may involve filling the edge areas of the aligned P-1th sample denoised image frame where image information was not sampled due to the alignment process with the content of the Pth sample image frame. The P-1th sample denoised image frame with the edge padding completed may be defined as a sample-aligned image frame.

[0129] 63. The second data enhancement process may include: i. randomly selecting one or more image areas from the P-1th sample denoised image frame after completing the edge area filling, and filling the image areas with the content of the Pth sample image frame; ii. synchronously performing operations such as channel shuffling and random color adjustment on the P-1th sample denoised image frame, the Pth sample image frame, and the Pth label image frame that have undergone the alignment and random area filling processes.

[0130] In some embodiments, the image information for filling the edge area and random area of the aligned P-1th sample denoised image frame may not come from the Pth sample image frame, but may be filled with default image information. The default image information may be pre-specified according to actual network training requirements.

[0131] 64. Channel concatenation: Perform channel concatenation on the P-1th sample denoised image frame and the Pth sample image frame in the second data enhancement processing result to form an image data body with unchanged height and width and a number of channels equal to the sum of the number of channels of the two image frames.

[0132] 65. Third data enhancement processing: Perform third data enhancement processing on the image data volume obtained by channel concatenation and the Pth labeled image frame. The third data enhancement processing may include performing random slicing, random rotation, random flipping, etc.

[0133] 66. Noise reduction processing: The processing result of the third data enhancement processing is input into the preset deep learning network to obtain the Pth sample denoised image frame.

[0134] 67. Calculate the loss function: Calculate the loss function between the P-th sample denoised image frame and the P-th labeled image frame output by the preset deep learning network.

[0135] 68. Gradient back propagation: Use the loss function to perform gradient back propagation to update the model parameters of the preset deep learning network until the loss function basically stops decreasing, completing the training of the preset deep learning network and obtaining the denoising network.

[0136] It can be understood that the Pth sample denoised image frame obtained by the preset deep learning network can be aligned to the P+1th sample image frame, and the above processing method is repeated to obtain the denoising result of the P+1th sample image frame.

[0137] Reference Figure 7 FIG2 is a schematic diagram showing the principle of training a preset deep learning network according to an embodiment of the present application. The preset deep learning network is a network of an encoding-decoding model framework.

[0138] 71. Data input: Input the training data into the preset deep learning network. The training data can be the processing result from the third data enhancement process mentioned above.

[0139] 72. Noise Reduction: The pre-set deep learning network performs noise reduction on the training data through a forward pass to generate noise-reduced data. The loss function is then calculated using the noise-reduced data and the labeled data. The labeled data is the high SPP image frame.

[0140] 73. Gradient back propagation: Use the loss function to perform gradient back propagation to update the network parameters of the preset deep learning network until the loss function basically stops decreasing. The trained preset deep learning network is the denoising network.

[0141] In some embodiments, the preset deep learning network may include an encoding-decoding framework consisting of multiple convolutional layers, multiple downsampling layers, and multiple upsampling layers. The number of channels of the multiple convolutional layers can be 32, 64, 128, 256, etc. For example, convolutional layer -32 indicates that the number of channels of the convolutional layer is 32.

[0142] refer to Figure 8 , is a schematic diagram of the hardware structure of the electronic device 100 provided in the embodiment of the present application. Figure 5 As shown, electronic device 100 may include a processor 1001, a memory 1002, a communication bus 1003, and a display screen 1004. Memory 1002 is used to store one or more computer programs 1005. One or more computer programs 1005 are configured to be executed by processor 1001. The one or more computer programs 1005 may include instructions, and the instructions may be used to implement the above-mentioned ray tracing multi-frame denoising method and / or denoising network training method in electronic device 100.

[0143] It is understood that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0144] The processor 1001 may include one or more processing units. For example, the processor 1001 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a DSP, a CPU, a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0145] Processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 1001 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 1001. If processor 1001 needs to use the same instruction or data again, it can directly access it from this memory. This avoids duplicate accesses, reduces the waiting time of processor 1001, and thus improves system efficiency.

[0146] In some embodiments, the processor 1001 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface.

[0147] In some embodiments, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0148] This embodiment also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the ray tracing multi-frame denoising method and / or the denoising network training method in the above-mentioned embodiment.

[0149] This embodiment also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the ray tracing multi-frame denoising method and / or the denoising network training method in the above-mentioned embodiment.

[0150] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the ray tracing multi-frame denoising method and / or the denoising network training method in the above-mentioned method embodiments.

[0151] Among them, the first electronic device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0152] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0157] The above description is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A ray tracing multi-frame denoising method, characterized in that: include: In response to a noise reduction requirement of a video clip, aligning a first noise-reduced image frame to a second image frame of the video clip; determining an edge missing region of the first denoised image frame that has undergone alignment processing; Filling the edge missing area with image information corresponding to the edge missing area in the second image frame to obtain an aligned image frame; Connecting the aligned image frame and the second image frame in series, and inputting the resultant image frame into a noise reduction network, so as to perform noise reduction processing on the second image frame to obtain a second noise-reduced image frame; The first denoised image frame is an image frame obtained by performing denoising processing on the first image frame of the video clip using the denoising network, and the first image frame is the upper N frames of the second image frame, where N is a positive integer.

2. The ray tracing multi-frame denoising method according to claim 1, wherein: The video clip is generated by rendering a three-dimensional model using a ray tracing pipeline, wherein the ray tracing pipeline also generates auxiliary information of the three-dimensional model when rendering the three-dimensional model, and the channel series includes one of the following: performing channel concatenation on the aligned image frame and the second image frame; performing channel concatenation on the aligned image frame, the second image frame, and the auxiliary information of the three-dimensional model; The auxiliary information includes at least one of the following: normal vector information, world coordinate information, color information, and texture information.

3. The ray tracing multi-frame denoising method according to claim 1 or 2, wherein: The method further comprises: selecting a first-type sample image frame of a sample video clip from a plurality of sample video clips, wherein each image frame in the sample video clip corresponds to a first-type sample image frame and a second-type sample image frame, and a sample per pixel (SPP) of the first-type sample image frame is smaller than the SPP of the second-type sample image frame; aligning a first sample denoised image frame of the sample video clip to a second sample image frame of the first type of the sample video clip; determining an edge missing region of the first sample denoised image frame that has undergone alignment processing; Filling the edge missing region with image information corresponding to the edge missing region in the second sample image frame of the first type to obtain a sample aligned image frame; Performing channel-wise concatenation on the sample aligned image frame and the second sample image frame of the first type, and inputting the concatenated frames into a preset deep learning network to perform noise reduction training on the preset deep learning network, wherein the first sample noise-reduced image frame is an image frame obtained by performing noise reduction processing on the first sample image frame of the first type using the preset deep learning network, and the first sample image frame of the first type is the upper M frames of the second sample image frame of the first type, where M is a positive integer; Calculating a loss function between a second sample denoised image frame output by the preset deep learning network and a second sample image frame of a second type; The loss function is used to perform gradient backpropagation to update the model parameters of the preset deep learning network until the training of the preset deep learning network is completed to obtain the denoising network.

4. The ray tracing multi-frame denoising method according to claim 3, wherein: Each image frame in the sample video segment corresponds to a plurality of sample image frames of the first type and a sample image frame of the second type, and the SPPs of the plurality of sample image frames of the first type are different from each other. The method further includes: Randomly selecting a sample image frame from the plurality of sample image frames of the first type, and combining the sample image frame with the sample image frame of the second type into a sample data set; The first type of sample image frames in the sample data set are used as training input data for the preset deep learning network, and the second type of sample image frames in the sample data set are used as label data for the preset deep learning network.

5. The ray tracing multi-frame denoising method according to claim 3, wherein: The plurality of sample video segments are obtained by rendering a three-dimensional sample model, wherein auxiliary information of the three-dimensional sample model is also generated when rendering the three-dimensional sample model, and the channel-concatenating the sample alignment image frame with the second sample image frame of the first type includes: The sample alignment image frame, the auxiliary information of the three-dimensional sample model, and the second sample image frame of the first type are channel-connected.

6. The ray tracing multi-frame denoising method according to claim 3, wherein: The method further comprises: Preprocessing each image frame in the sample video clip; The preprocessing includes replacing abnormal pixel values in the image frame with preset pixel values and / or removing a texture in the image frame.

7. The ray tracing multi-frame denoising method according to claim 6, wherein: The preprocessing of each image frame in the sample video clip includes: Replacing pixels in the image frame whose pixel values are infinite with zero; or Perform nearest neighbor interpolation replacement on pixels with infinite pixel values in the image frame.

8. The ray tracing multi-frame denoising method according to claim 3, wherein: The aligning the first sample denoised image frame of the sample video clip to the second sample image frame of the first type of the sample video clip includes: Determine, by using coordinate transformation, position information of each pixel in the first sample denoised image frame of the sample video clip in the second sample image frame of the first type of the sample video clip; A preset image interpolation algorithm is used to perform interpolation processing on pixels with position changes in the first sample noise reduction image frame of the sample video clip.

9. The ray tracing multi-frame denoising method according to claim 3, wherein: The selecting of a first type of sample image frame of a sample video clip from a plurality of sample video clips comprises: caching a previous sample noise reduction image frame of each sample video segment; A current sample image frame of the first type of a sample video segment is randomly selected from the multiple sample video segments.

10. The ray tracing multi-frame denoising method according to claim 3, wherein: Filling the edge missing area with image information corresponding to the edge missing area in the second sample image frame of the first type to obtain a sample aligned image frame includes: Filling the edge missing area with image information corresponding to the edge missing area in the second sample image frame of the first type; An image area is randomly selected from the first sample denoised image frame that has undergone edge filling processing, and image information corresponding to the randomly selected image area in the second sample image frame of the first type is filled into the image area to obtain the sample aligned image frame.

11. The ray tracing multi-frame denoising method according to claim 3, wherein: Filling the edge missing area with image information corresponding to the edge missing area in the second sample image frame of the first type to obtain a sample aligned image frame includes: Filling the edge missing area with default image information; An image region is randomly selected from the first sample denoised image frame that has undergone edge filling processing, and the default image information is filled into the image region to obtain the sample aligned image frame.

12. The ray tracing multi-frame denoising method according to claim 3, wherein: The channel-concatenating the sample aligned image frame and the second sample image frame of the first type, and inputting the channel-concatenated channel into a preset deep learning network to perform noise reduction training on the preset deep learning network, includes: Perform channel-wise concatenation on the sample aligned image frame and the second sample image frame of the first type to obtain a concatenated image frame; An image area of a preset size is randomly cut from the series-connected image frames and input into the preset deep learning network to perform noise reduction training on the preset deep learning network.

13. The ray tracing multi-frame denoising method according to claim 5, wherein: The method further comprises: Defining lens parameters of the scene lens, wherein the lens parameters include lens movement trajectory and lens direction; Rendering the three-dimensional sample model based on the lens parameters using the ray tracing pipeline to obtain the plurality of sample video clips; The three-dimensional sample model includes a static three-dimensional model or a dynamic three-dimensional model.

14. The ray tracing multi-frame denoising method according to claim 13, wherein: The rendering of the three-dimensional sample model based on the lens parameters by using the ray tracing pipeline to obtain the plurality of sample video clips includes: Rendering the three-dimensional sample model based on the lens parameters using the ray tracing pipeline to obtain a plurality of sample videos; The sample video is subjected to random frame sampling and / or random frame reversal to obtain the sample video segment.

15. The ray tracing multi-frame denoising method according to claim 3, wherein: The method further comprises: Testing the denoising network using a test set; When it is determined through the test set that the noise reduction network meets the preset noise reduction requirement, performing noise reduction processing on the video clip based on the noise reduction network; Each test data in the test set includes a sample image frame of the first type and a sample image frame of the second type.

16. A ray tracing multi-frame denoising method, characterized in that: include: Selecting a first type of sample image frame of a sample video clip from a plurality of sample video clips, wherein each image frame in the sample video clip corresponds to a first type of sample image frame and a second type of sample image frame, and an SPP of the first type of sample image frame is smaller than an SPP of the second type of sample image frame; aligning a first sample denoised image frame of the sample video clip to a second sample image frame of the first type of the sample video clip; determining an edge missing region of the first sample denoised image frame that has undergone alignment processing; Filling the edge missing region with image information corresponding to the edge missing region in the second sample image frame of the first type to obtain a sample aligned image frame; Performing channel-wise concatenation on the sample aligned image frame and the second sample image frame of the first type, and inputting the concatenated frames into a preset deep learning network to perform noise reduction training on the preset deep learning network, wherein the first sample noise-reduced image frame is an image frame obtained by performing noise reduction processing on the first sample image frame of the first type using the preset deep learning network, and the first sample image frame of the first type is the upper M frames of the second sample image frame of the first type, where M is a positive integer; Calculating a loss function between a second sample denoised image frame output by the preset deep learning network and a second sample image frame of a second type; Using the loss function to perform gradient backpropagation to update the model parameters of the preset deep learning network until the training of the preset deep learning network is completed to obtain a denoising network; Testing the denoising network using a test set; When it is determined through the test set that the denoising network meets the preset denoising requirement, denoising processing is performed on the image frame to be processed based on the denoising network.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed on an electronic device, enable the electronic device to perform the ray tracing multi-frame denoising method according to any one of claims 1 to 16.

18. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the ray tracing multi-frame denoising method according to any one of claims 1 to 16.

19. A chip coupled to a memory in an electronic device, characterized in that: The chip is used to control the electronic device to execute the ray tracing multi-frame noise reduction method according to any one of claims 1 to 16.

Images