Image noise reduction method and device, equipment and storage medium
By acquiring and processing N-frame noise images in ray tracing technology, using rendering buffer data and kernel functions for image noise reduction, the problem of image noise in ray tracing is solved, and a higher quality image output is achieved.
Patent Information
- Application Number
- CN202510012432.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2025-05-06
AI Technical Summary
In ray tracing technology, due to the limitations of computing resources and time, it is impossible to accurately capture all light propagation paths and energy distributions, resulting in image noise and reducing image quality.
By acquiring N-frame noise images, image noise reduction data is determined based on the rendering buffer data, including at least two-level kernel functions and noise weight coefficients, image grading noise reduction is performed, and the timing accumulation of noise images are integrated.
Effectively reduce image noise, improve image quality, take into account timing stability, avoid flickering, and retain high-frequency texture and edge areas.
Smart Images
Figure CN119941558A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to an image noise reduction method, device, equipment and storage medium. Background Art
[0002] In recent years, with the booming development of the gaming industry, people's standards for gaming experience have also been increasing, and image quality, as one of the key indicators, has attracted much attention from the public. As a result, ray tracing technology has been favored by people for its realistic rendering effects. This technology is a technology used in computer graphics to generate realistic images. Its basic principle is to generate images by reversely tracing the light emitted from the eyes, that is, starting from each pixel on the screen, tracing in the opposite direction of the light, finding the surface point of the object that intersects with the line of sight, and calculating the light intensity at the point as the color of the pixel, thereby generating an image based on the color of each pixel.
[0003] In ray tracing, in order to determine the color of each pixel, it is necessary to emit light from the eye and trace the interaction between the light and the object. However, in actual operation, due to the limitation of computing resources and time, it is impossible to emit an infinite number of rays, and only a limited number of sampling can be performed using the Monte Carlo method. However, when the number of samples is small or involves complex lighting scenes and object material properties, the aforementioned method cannot accurately capture all light propagation paths and energy distribution, resulting in missing or inaccurate light in some areas, making the generated image noisy and reducing the image quality. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an image noise reduction method, device, equipment and storage medium, which can reduce image noise and improve image quality.
[0005] In a first aspect, an embodiment of the present application provides an image noise reduction method, comprising:
[0006] Obtaining N noise images, where the noise images are images rendered by a ray tracing algorithm, the N noise images include a first noise image and a second noise image, where the second noise image is a noise image of a frame before the first noise image, and N is an integer greater than 1;
[0007] Determine image denoising data based on N frames of noisy images and rendering buffer data of objects in the first noisy image; the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients;
[0008] According to the noise weight coefficient, the first noise image and the second noise image are fused to obtain a time series cumulative noise image;
[0009] The time-series accumulated noise image is subjected to hierarchical image denoising by at least two levels of kernel functions to obtain a first denoised image of the first noise image.
[0010] In a second aspect, an embodiment of the present application provides an image noise reduction device, comprising:
[0011] An acquisition module is used to acquire N noise image frames, where the noise image is an image rendered by a ray tracing algorithm, and the N noise image frames include a first noise image and a second noise image, where the second noise image is a noise image of a frame before the first noise image, and N is an integer greater than 1;
[0012] A determination module, configured to determine image denoising data based on N frames of noisy images and rendering buffer data of objects in the first noisy image; the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients;
[0013] A fusion module, used for fusing the first noise image and the second noise image according to the noise weight coefficient to obtain a time series cumulative noise image;
[0014] The denoising module is used to perform image hierarchical denoising on the time-series accumulated noise image through at least two levels of kernel functions to obtain a first denoised image of the first noise image.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the steps of the image denoising method shown in the first aspect are implemented.
[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the image denoising method shown in the first aspect are implemented.
[0017] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run a program or instruction to implement the steps of the image noise reduction method shown in the first aspect.
[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the image denoising method shown in the first aspect.
[0019] In an embodiment of the present application, N frames of noise images are obtained after being rendered by a ray tracing algorithm, the N frames of noise images include a first noise image and a second noise image, and the second noise image is a noise image of a frame before the first noise image; based on the N frames of noise images and the rendering buffer data of the objects in the first noise image, image denoising data is determined, wherein the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients; according to the noise weight coefficient, the first noise image and the second noise image are fused to obtain a time-series cumulative noise image; and the time-series cumulative noise image is subjected to image hierarchical denoising through at least two levels of kernel functions to obtain a first denoised image of the first noise image. In this way, by fusing two consecutive frames of noise images to obtain a time-series cumulative noise image, the stability of the time series can be taken into account, and the flickering phenomenon that may appear in the first denoised image can be greatly reduced. In addition, the time-series cumulative noise image is subjected to image graded denoising through at least two levels of kernel functions, which can make the low-frequency texture more coordinated and unified, and avoid the appearance of color blocks, blur and the like in the first denoised image. In this way, in the spatial dimension, the high-frequency texture and edge area can be completely retained, and in the time dimension, the stability of the time series can be guaranteed, and there will be no obvious flickering, which can effectively reduce image noise and improve image quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A flowchart of an image noise reduction method provided in an embodiment of the present application;
[0021] Figure 2 A schematic diagram of a process of an image noise reduction method provided in an embodiment of the present application;
[0022] Figure 3 A schematic diagram of a flow chart of a convolutional neural network processing input splicing data in an image denoising method provided in an embodiment of the present application;
[0023] Figure 4 A schematic diagram of a hierarchical noise reduction process of an image noise reduction method provided in an embodiment of the present application;
[0024] FIG5( a) is a schematic diagram of a first noise image of an image denoising method provided in an embodiment of the present application;
[0025] FIG5( b ) is a schematic diagram of a first denoised image of an image denoising method provided in an embodiment of the present application;
[0026] FIG5( c ) is a second schematic diagram of a first noise image of an image noise reduction method provided in an embodiment of the present application;
[0027] FIG5( d ) is a second schematic diagram of a first denoised image of an image denoising method provided in an embodiment of the present application;
[0028] Figure 6 A schematic diagram of the structure of an image noise reduction device provided in an embodiment of the present application;
[0029] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0030] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0033] In order to solve the problems in the related technology, the embodiments of the present application provide an image denoising method, apparatus, device and storage medium, which can comprehensively improve the effect of image denoising while controlling resource consumption. In the spatial dimension, high-frequency textures and edge areas can be completely preserved. In the temporal dimension, the stability of the video can be ensured without obvious flickering, thereby reducing image noise and improving image quality.
[0034] Based on this, the following Figure 1 to Figure 5(a) -(d) The image denoising method provided in the embodiment of the present application is described in detail through specific embodiments and their application scenarios.
[0035] First, combine Figure 1 An image noise reduction method provided in an embodiment of the present application is described in detail.
[0036] Figure 1A flowchart of an image noise reduction method provided in an embodiment of the present application.
[0037] like Figure 1 As shown, the image noise reduction method provided in the embodiment of the present application can be applied to electronic devices. Based on this, the image noise reduction method can include the following steps:
[0038] Step 110, obtaining N frames of noise images, where the noise images are images rendered by a ray tracing algorithm, and the N frames of noise images include a first noise image and a second noise image, where the second noise image is a noise image of a frame before the first noise image, and N is an integer greater than 1; Step 120, determining image denoising data based on the N frames of noise images and the rendering buffer data of the objects in the first noise image; the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients; Step 130, performing fusion processing on the first noise image and the second noise image according to the noise weight coefficient to obtain a time-series cumulative noise image; Step 140, performing image hierarchical denoising on the time-series cumulative noise image through at least two levels of kernel functions to obtain a first denoised image of the first noise image.
[0039] In this way, by fusing two consecutive frames of noise images to obtain a time-series cumulative noise image, the stability of the time series can be taken into account, and the flickering phenomenon that may appear in the first denoised image can be greatly reduced. In addition, the time-series cumulative noise image is subjected to image graded denoising through at least two levels of kernel functions, which can make the low-frequency texture more coordinated and unified, and avoid the appearance of color blocks, blur and the like in the first denoised image. In this way, in the spatial dimension, the high-frequency texture and edge area can be completely retained, and in the time dimension, the stability of the time series can be guaranteed, and there will be no obvious flickering, which can effectively reduce image noise and improve image quality.
[0040] The above steps are described in detail below, as shown below.
[0041] First, step 110 is involved. In some embodiments of the present application, step 110 may specifically include steps 1101 to 1103 .
[0042] Step 1101 , rendering a first image in a video by a ray tracing algorithm to obtain a first noise image, where the first noise image is a three-dimensional effect image of the first image.
[0043] Step 1102, acquiring a second image from the video according to the displacement of pixels of the object in the first image between the first image and each frame of the video, where the second image is an image of a frame before the first image in the video.
[0044] Step 1103: Determine the image after the second image is rendered by the ray tracing algorithm as the second noise image.
[0045] Exemplarily, the first image in the video can be rendered by a ray tracing algorithm to obtain a first noise image, which is recorded as Color. At this time, the rendering buffer data of the geometry buffer (G-Buffer) can be obtained in the rendering pipeline. Among them, the rendering buffer data may include surface normal vector data for reflecting the shape of the object, object surface reflectivity data for characterizing the reflective characteristics of the object surface, and motion data for characterizing the object between at least two frames of images. Further, the surface normal vector data may include surface normal data (Normal) corresponding to the pixel of the object, the object surface reflectivity data may include diffuse color data, i.e., albedo, and the motion data may be used in the G-Buffer to record the motion information of the object at the pixel level, which is a two-dimensional vector data, including but not limited to motion vectors (Motion Vectors, MV), velocity vectors (Velocity Vectors), and motion blur (Motion Blur). Here, in the embodiment of the present application, the motion data is taken as a motion vector as an example for explanation. Specifically, the motion vector records the displacement of the pixel between one frame of image and another frame of image. In this way, the rendering buffer data can be recorded as G-Buffer (Albedo, Normal, MV).
[0046] Based on this, a second image can be selected from the video based on MV, that is, the second image can be screened from the video, the module of displacement of pixels of the object in the first image between the first image and the second image is the shortest, and the image rendered by the ray tracing algorithm is determined as the second noise image.
[0047] Next, referring to step 120 , in some embodiments of the present application, the rendering buffer data also includes first timing data related to the second noise image and used for timing propagation. Based on this, step 120 may specifically include step 1201 and step 1202 .
[0048] Step 1201 , performing data splicing processing on the pixel values of the first noise image, the pixel values of the second noise image, the surface normal vector data, the object surface reflectivity data and the first time series data to obtain input spliced data.
[0049] Step 1202: Process the input spliced data through a convolutional neural network to obtain image denoising data.
[0050] For example, Figure 2As shown, the embodiment of the present application is to perform data splicing processing on the pixel value of Color, the pixel value of Previous Color, Normal, Albedo and the first time series data (Previous Feature) through a concat operation to obtain input splicing data (input_features), which is used as the input of a convolutional neural network (Unet-like architecture). The output of the convolutional neural network is divided into 4 parts, namely, the second time series data for time series propagation, namely Feature, and at least two levels of kernel functions (Kernels) for denoising. In the example of the present application, an example is given in which Kernels are kernels of 5 different sizes, which are used for subsequent denoising of images of different levels and sizes, and are used to fuse the noise weight coefficients (Color Weight) of Previous Color and Color, and are used to fuse the noise reduction weight coefficients (OutputWeight) of the first denoised image and the second denoised image.
[0051] It should be noted that, before executing step 1201 , the first noise image and the second noise image may be aligned. Based on this, the image denoising method may further include steps 1203 and 1204 .
[0052] Step 1203: align the object in the second noise image with the object in the first noise image according to the displacement of the object in the first noise image between the first noise image and the second noise image, so as to obtain an aligned second noise image.
[0053] Step 1204, replace the pixel values of the second noise image with the pixel values of the aligned second noise image, so as to perform data splicing processing with the pixel values of the second noise image, the surface normal vector data, the object surface reflectivity data and the first time series data to obtain input spliced data.
[0054] It should be noted that if the second noise image is the first frame of N frames of noise images, there is no previous frame, and then the three variables of the pixel value of the second noise image (Previous Color), Previous Feature and Previous Output are all replaced by 0.
[0055] Therefore, by adopting the MV method to align the object in the first noisy image with the object in the second noisy image, the accuracy of noise reduction can be improved.
[0056] In some embodiments of the present application, the image denoising data also includes a denoising weight coefficient, which is used to fuse the first denoised image and the second denoised image with the second noise image. Based on this, the above step 1202 may specifically include steps 12021 to 12025.
[0057] Step 12021, through the convolutional neural network, the input spliced data is sequentially downsampled at least once and upsampled at least once to obtain a sampling result, wherein the sampling result includes a first sampling number of the downsampling process, a second sampling number of the upsampling process, a sequence of the first noise image and the second noise image on the timeline, a first image noise change amount between the first noise image and the second noise image, and an estimated denoised image corresponding to the first noise image.
[0058] For example, Figure 3 As shown in the figure, the input feature input_features of the Unet-like architecture needs to go through 4 times of downsampling Maxpool and 4 times of bilinear interpolation upsampling Upsample. Among them, the feature map generated by Decoder layer4 will be generated by a 3*3 convolution to generate the Kernel: K4 required for denoising. K3, K2, and K1 are also generated by a similar method. Among them, the feature map generated by Decoder layer0 is generated by a 3*3 convolution. In addition to generating the Kernel: K0 required for denoising, it also generates Feature, Output Weight, and Kernel Weight.
[0059] Step 12022: Determine the order of the kernel function based on the first sampling number or the second sampling number.
[0060] Exemplarily, the order of the kernel function may be obtained by adding 1 to the first sampling number.
[0061] Step 12023: Generate first time series data based on the sequence of the first noise image and the second noise image on the timeline.
[0062] Step 12024: determine the normalized value of the noise variation of the first image as the noise weight coefficient.
[0063] Step 12025: determine the normalized value of the second image noise variation as the noise reduction weight coefficient, and the second image noise variation is the image noise variation between the first noisy image and the estimated noise reduction image.
[0064] It should be noted that the above convolutional neural network can be trained through the following process.
[0065] The training data set used in the embodiment of the present application is to render the noisy original image Color of 1spp, 2spp, 4spp, 8spp, 16spp, 32spp, and 2048spp in the same scene, the same position, and the same time, and export Gbuffer (including Albedo, Normal, MV) at the same time. Among them, the Color data of 32spp and below is used as the training input, and the 2048spp data is used as the ground truth (GT). Here, since there is still a small amount of noise in 2048spp, in order to ensure the accuracy of the data set, it can be processed by the denoising algorithm of the open source image denoising library (Open Image Denoise, OIDN) before use.
[0066] In order to enhance the generalization performance, the embodiment of the present application may use data enhancement in the training phase of the convolutional neural network, wherein Normal and MV represent vectors, and their directions must be marked when performing data enhancement.
[0067] The loss function (Loss) involved in the convolutional neural network in the embodiment of the present application may include three parts, namely, a spatial loss function (spatial Loss), a perceptual loss function (percept Loss) and a temporal loss function (temporal Loss), which can be specifically calculated by the following formula (1):
[0068]
[0069] in, is the image denoising data at time t, is the GT at time t, the value of ε is 1e-5, and MSE is the mean-square error. Refers to the mean of the feature maps of "relu1_2", "relu2_2", "relu3_3", and "relu4_3" obtained by inferring the image through the VGG-16 model. It refers to backward warp, which uses the MV at time t to warp the image at time t-1 to align with time t.
[0070] Therefore, the embodiment of the present application can constrain the changes between frames by using the time domain loss. Specifically, the change of the current frame compared to the previous frame is required to be similar to the change of GT. If the change between the two is too large, a corresponding penalty will be given in the loss item to improve the temporal stability. In addition, the effect is improved through a variety of data enhancement methods, and in terms of the design of Loss, while using pixel-by-pixel L1-like Loss, the effect is also enhanced with the help of perceptual Loss.
[0071] Furthermore, with respect to step 130 , in some embodiments of the present application, the noise weight coefficient includes a first noise weight coefficient and a second noise weight coefficient. Based on this, step 130 may specifically include steps 1301 to 1303 .
[0072] Step 1301: Adjust the pixel values of a first noise image according to a first noise weight coefficient to obtain a first adjusted noise image.
[0073] Step 1302: Adjust the pixel values of the second noise image according to the second noise weight coefficient to obtain a second adjusted noise image.
[0074] Step 1303: perform image fusion on the first adjusted noise image and the second adjusted noise image to obtain a time series accumulated noise image.
[0075] For example, Color and the second noise image are recorded as (Previous Color) and fused according to the following formula (2) to obtain a time series accumulated noise image (Accumulated Color):
[0076] AccumulatedColor=(1-ColorWeight)*PreviousColor+ColorWeight*Color(2)
[0077] Therefore, the embodiment of the present application can perform image fusion processing on two noisy images, namely the first noise image and the second noise image, so that the first noise image of the current frame can use the information of the second noise image of the previous frame, and the accumulation operation can be continuously performed with the rendering. This not only effectively enhances the stability in the time domain, but also improves the presentation effect of the current first frame.
[0078] Then, step 140 is involved. In some embodiments of the present application, at least two levels of kernel functions include a first level kernel function and a second level kernel function. Step 140 may specifically include steps 1401 to 1403.
[0079] Step 1401 , using a first level kernel function, bilinear difference downsampling of a first pixel multiple is performed on the time series accumulated noise image to obtain first chromaticity coordinate information of the time series accumulated noise image.
[0080] Step 1402, using a second-level kernel function, bilinear difference downsampling of the time-series accumulated noise image at a second pixel multiple is performed to obtain second chromaticity coordinate information of the time-series accumulated noise image, wherein the second pixel multiple is greater than the first pixel multiple.
[0081] Step 1403: Generate a first denoised image based on the first chromaticity coordinate information and the second chromaticity coordinate information.
[0082] For example, Figure 4 As shown, the Accumulated Color and Kernels are sent to the Filter module for denoising. First, the Acuumulated Color is downsampled by 1, 2, 4, 8, and 16 times of bilinear interpolation, and then combined with the Kernel to generate the first denoised image, which can be obtained by the following formula (3):
[0083]
[0084] Among them, (x, y) is the coordinate of the midpoint of the result image, (u, v) is the 5*5 area around (x, y), and K uvxy is the predicted Kernel value of the corresponding coordinate, A uv is the value of Acuumulated Color at the (u,v) coordinate, F xy is the value of the denoising result at the (x, y) coordinate. In this way, the chromaticity coordinate information obtained at each level can be accumulated layer by layer to obtain the comprehensive chromaticity coordinate information, and the first denoised image (Filtered Color) is generated based on the chromaticity coordinate information.
[0085] Therefore, the embodiment of the present application can perform multi-layer downsampling of the noise image in sequence, and then use the kernel to perform noise reduction processing on each layer of the noise image, and then perform upsampling and accumulation operations. This method can not only ensure that the image can be clearer and more coordinated while retaining high-frequency textures and edges, making the overall effect more coordinated and unified, but also remove the color block problem. In addition, since direct pixel generation is avoided, low-frequency textures can be more coordinated and unified, and linear calculations are performed on the noise image using kernel functions, image noise can be reduced and image quality can be improved when computing resources are limited.
[0086] In addition, in some embodiments of the present application, the image denoising data in the embodiments of the present application also includes a denoising weight coefficient. Based on this, after step 140, the image denoising method may further include step 150:
[0087] According to the noise reduction weight coefficient, the first noise reduction image and the second noise reduction image of the second image are fused to obtain a time-series cumulative noise reduction image corresponding to the first noise image.
[0088] Furthermore, the noise reduction weight coefficient includes a first noise reduction weight coefficient and a second noise reduction weight coefficient. Based on this, step 150 may specifically include steps 1501 to 1503, as shown below.
[0089] Step 1501: adjusting the pixel values of a first denoised image according to a first denoising weight coefficient to obtain a first adjusted denoised image.
[0090] For example, reference may still be made to Figure 2 , the first denoised image can be recorded as the pixel point of (Filtered Color), and the first denoising weight coefficient can be recorded as (Output Weight). At this time, the product of Filtered Color*Output Weight can be used as the pixel point of the first adjusted denoised image, so as to generate the first adjusted denoised image according to the pixel point of the first adjusted denoised image.
[0091] Step 1502: adjusting the pixel values of the second denoised image according to the second denoising weight coefficient to obtain a second adjusted denoised image.
[0092] Exemplarily, the second denoised image can be recorded as the pixel points of (Previous Output), and the second denoising weight coefficient can be recorded as (1-Output Weight). At this time, the product of (1-Output Weight)*Previous Output can be used as the pixel points of the second adjusted denoised image, thereby generating the first adjusted denoised image according to the pixel points of the second adjusted denoised image.
[0093] Step 1503: perform image fusion on the first adjusted denoised image and the second adjusted denoised image to obtain a time-series cumulative denoised image corresponding to the first noisy image.
[0094] For example, the first adjusted denoised image and the second adjusted denoised image may be fused by the following formula (4) to obtain a time-series cumulative denoised image (Output):
[0095] Output=(1-OutputWeight)*PreviousOutput+OutputWeight*FilteredColor(4)
[0096] Therefore, in the embodiment of the present application, the first noise image can be as shown in Figures 5(a) and 5(c), and the time-accumulated denoised image can be as shown in Figures 5(b) and 5(d). In this way, the low-frequency texture can be more coordinated and unified, avoiding the appearance of color blocks, blur and other phenomena in the first denoised image. In this way, in the spatial dimension, the high-frequency texture and edge area can be completely retained. In the time dimension, the stability of the timing can be guaranteed, and there will be no obvious flickering, which can effectively reduce image noise and improve image quality.
[0097] It should be noted that, in order to use the data of this time for the next execution of the image denoising method in the embodiment of the present application, that is, after step 150, the image denoising method may further include step 160:
[0098] The first noise image, the second time series data related to the first noise image and used for time series propagation, the first denoised image and the time series accumulated denoised image of the first noise image are used to determine the third denoised image corresponding to the third noise image. The third noise image is a noise image of a frame after the first noise image in the N frames of noise images.
[0099] For example, reference may still be made to Figure 2 As shown, Color, Feature and Output need to be stored for use in the next frame to achieve the effect of time domain accumulation.
[0100] Therefore, the image denoising method provided in the embodiment of the present application can be transferred between frames with the help of Features. Specifically, the neural network of the current frame will generate corresponding features, which will be used as input for the neural network of the next frame to enhance the temporal stability.
[0101] It should be noted that the image denoising method provided in the embodiment of the present application will always have limitations on hardware and computing resources. Therefore, as long as the scene involves ray tracing technology, there will be a need to use ray tracing to reduce noise. Therefore, the image denoising method provided in the embodiment of the present application can be applied to many ray tracing rendering scenes such as games, animations, and movies to improve the realistic picture presentation of ray tracing.
[0102] The image denoising method provided in the embodiment of the present application may be executed by an image denoising device. In the embodiment of the present application, an image denoising device performing image denoising is taken as an example to illustrate the image denoising method provided in the embodiment of the present application.
[0103] Based on the same inventive concept, the present application also provides an image noise reduction device. Figure 6 Provide detailed explanation.
[0104] Figure 6 A schematic diagram of the structure of an image noise reduction device provided in an embodiment of the present application.
[0105] like Figure 6 As shown, the image noise reduction device 60 can be applied to electronic equipment, and the image noise reduction device 60 can specifically include:
[0106] An acquisition module 601 is used to acquire N noise image frames, where the noise image is an image rendered by a ray tracing algorithm, and the N noise image frames include a first noise image and a second noise image, where the second noise image is a noise image of a frame before the first noise image, and N is an integer greater than 1;
[0107] A determination module 602 is used to determine image denoising data based on the N frames of noisy images and the rendering buffer data of the object in the first noisy image; the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients;
[0108] A fusion module 603 is used to fuse the first noise image and the second noise image according to the noise weight coefficient to obtain a time series cumulative noise image;
[0109] The denoising module 604 is configured to perform image hierarchical denoising on the time-series accumulated noise image by using at least two levels of kernel functions to obtain a first denoised image of the first noise image.
[0110] The image noise reduction device 60 in the embodiment of the present application is described in detail below, as shown below.
[0111] In some embodiments of the present application, the fusion module 603 can also be used to, when the image denoising data also includes a denoising weight coefficient, fuse the first denoised image and the second denoised image of the second noise image according to the denoising weight coefficient to obtain a time-series cumulative denoised image corresponding to the first noise image.
[0112] In some embodiments of the present application, the image denoising device 60 in the embodiment of the present application may further include an adjustment module, which is used to adjust the pixel value of the first denoised image according to the first denoising weight coefficient to obtain a first adjusted denoised image when the denoising weight coefficient includes the first denoising weight coefficient and the second denoising weight coefficient; and adjust the pixel value of the second denoised image according to the second denoising weight coefficient to obtain a second adjusted denoised image.
[0113] The fusion module 603 may be specifically configured to perform image fusion on the first adjusted denoised image and the second adjusted denoised image to obtain a time-series cumulative denoised image corresponding to the first noisy image.
[0114] In some embodiments of the present application, the image denoising device 60 in the embodiment of the present application may further include a rendering module, which is used to render the first image in the video by a ray tracing algorithm to obtain a first noise image, where the first noise image is a three-dimensional effect image of the first image;
[0115] The acquisition module 601 may also be used to acquire a second image from the video according to the displacement of pixels of the object in the first image between the first image and each frame of the video, where the second image is an image of a frame before the first image in the video;
[0116] The determination module 602 may also be configured to determine an image after the second image is rendered by a ray tracing algorithm as a second noise image.
[0117] In some embodiments of the present application, the determination module 602 may be specifically configured to, when the rendering buffer data further includes first time series data related to the second noise image and used for time series propagation, perform data splicing processing on pixel values of the first noise image, pixel values of the second noise image, surface normal vector data, object surface reflectivity data, and the first time series data to obtain input spliced data;
[0118] The input spliced data is processed through the convolutional neural network to obtain image denoising data.
[0119] In some embodiments of the present application, the image noise reduction device 60 in the embodiment of the present application may further include an alignment module, which is used to align the object in the second noise image with the object in the first noise image according to the displacement of the object in the first noise image between the first noise image and the second noise image, so as to obtain an aligned second noise image;
[0120] The image denoising device 60 in the embodiment of the present application may further include a stitching module for replacing the pixel values of the second noise image with the pixel values of the aligned second noise image, so as to perform data stitching processing with the pixel values of the second noise image, surface normal vector data, object surface reflectivity data and first time series data to obtain input stitching data.
[0121] In some embodiments of the present application, the determination module 602 may be specifically used to, when the image denoising data further includes a denoising weight coefficient, and the denoising weight coefficient is used to fuse the first denoised image and the second denoised image with the second noise image, sequentially perform at least one downsampling process and at least one upsampling process on the input spliced data through a convolutional neural network to obtain a sampling result, wherein the sampling result includes a first sampling number of the downsampling process, a second sampling number of the upsampling process, a sequence of the first noise image and the second noise image on a timeline, a first image noise change amount between the first noise image and the second noise image, and an estimated denoised image corresponding to the first noise image;
[0122] Determining the order of the kernel function based on the first sampling number or the second sampling number;
[0123] Generate first time series data based on the sequence of the first noise image and the second noise image on the timeline;
[0124] The normalized value of the noise variation of the first image is determined as a noise weight coefficient;
[0125] The normalized value of the second image noise variation is determined as the noise reduction weight coefficient, and the second image noise variation is the image noise variation between the first noisy image and the estimated noise reduction image.
[0126] In some embodiments of the present application, the fusion module 603 may be specifically configured to, when the noise weight coefficient includes a first noise weight coefficient and a second noise weight coefficient, adjust the pixel value of the first noise image according to the first noise weight coefficient to obtain a first adjusted noise image;
[0127] According to the second noise weight coefficient, pixel values of the second noise image are adjusted to obtain a second adjusted noise image;
[0128] The first adjusted noise image and the second adjusted noise image are fused to obtain a time series accumulated noise image.
[0129] In some embodiments of the present application, the noise reduction module 604 may be specifically configured to, in a case where at least two levels of kernel functions include a first level kernel function and a second level kernel function, perform bilinear difference downsampling of a first pixel multiple on the time series accumulated noise image by the first level kernel function to obtain first chromaticity coordinate information of the time series accumulated noise image;
[0130] Performing bilinear difference downsampling of a second pixel multiple on the time series accumulated noise image through a second-level kernel function to obtain second chromaticity coordinate information of the time series accumulated noise image, wherein the second pixel multiple is greater than the first pixel multiple;
[0131] A first noise reduction image is generated based on the first chromaticity coordinate information and the second chromaticity coordinate information.
[0132] In some embodiments of the present application, the determination module 602 can also be used to use the first noise image, the second time series data related to the first noise image and used for time series propagation, the first denoised image and the time series accumulated denoised image of the first noise image as a third denoised image corresponding to the third noise image, where the third noise image is a noise image of a frame after the first noise image in the N frames of noise images.
[0133] The image noise reduction device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0134] The image noise reduction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0135] The device coordination device provided in the embodiment of the present application can achieve Figure 1 to Figure 5(a) -The various processes implemented in the embodiment of the image denoising method shown in -(d) achieve the same technical effect, and will not be described again here to avoid repetition.
[0136] Based on this, the image denoising device provided in the embodiment of the present application can be used to obtain N frames of noise images rendered by a ray tracing algorithm, the N frames of noise images including a first noise image and a second noise image, the second noise image being a noise image of a frame before the first noise image; based on the N frames of noise images and the rendering buffer data of the objects in the first noise image, determine the image denoising data, wherein the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients; according to the noise weight coefficient, the first noise image and the second noise image are fused to obtain a time-series cumulative noise image; through at least two levels of kernel functions, the time-series cumulative noise image is subjected to image hierarchical denoising to obtain a first denoised image of the first noise image. In this way, by fusing two consecutive frames of noise images to obtain a time-series cumulative noise image, the stability of the time series can be taken into account, and the flickering phenomenon that may appear in the first denoised image can be greatly reduced. In addition, the time-series cumulative noise image is subjected to image graded denoising through at least two levels of kernel functions, which can make the low-frequency texture more coordinated and unified, and avoid the appearance of color blocks, blur and the like in the first denoised image. In this way, in the spatial dimension, the high-frequency texture and edge area can be completely retained, and in the time dimension, the stability of the time series can be guaranteed, and there will be no obvious flickering, which can effectively reduce image noise and improve image quality.
[0137] Optional, such as Figure 7 As shown, an embodiment of the present application further provides an electronic device 70, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned image denoising method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0138] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
[0139] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
[0140] The electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, a processor 810 and other components.
[0141] Those skilled in the art will appreciate that the electronic device 800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0142] Among them, in the embodiment of the present application, the processor 810 is used to obtain N frames of noise images, the noise image is an image rendered by a ray tracing algorithm, the N frames of noise images include a first noise image and a second noise image, the second noise image is a noise image of a frame before the first noise image, and N is an integer greater than 1. The processor 810 can also be used to determine image denoising data based on the N frames of noise images and the rendering buffer data of the object in the first noise image; the rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients. The processor 810 can also be used to perform fusion processing on the first noise image and the second noise image according to the noise weight coefficient to obtain a time-series cumulative noise image. The processor 810 can also be used to perform image hierarchical denoising on the time-series cumulative noise image through at least two levels of kernel functions to obtain a first denoised image of the first noise image.
[0143] The electronic device 800 is described in detail below, as shown below.
[0144] In some embodiments of the present application, the fusion module 603 can also be used to, when the image denoising data also includes a denoising weight coefficient, fuse the first denoised image and the second denoised image of the second noise image according to the denoising weight coefficient to obtain a time-series cumulative denoised image corresponding to the first noise image.
[0145] In some embodiments of the present application, the processor 810 may also be configured to, when the denoising weight coefficient includes a first denoising weight coefficient and a second denoising weight coefficient, adjust the pixel value of the first denoised image according to the first denoising weight coefficient to obtain a first adjusted denoised image; and adjust the pixel value of the second denoised image according to the second denoising weight coefficient to obtain a second adjusted denoised image;
[0146] The first adjusted denoised image and the second adjusted denoised image are fused to obtain a time-series cumulative denoised image corresponding to the first noisy image.
[0147] In some embodiments of the present application, the processor 810 may be specifically configured to render the first image in the video through a ray tracing algorithm to obtain a first noise image, where the first noise image is a three-dimensional effect image of the first image;
[0148] Acquire a second image from the video according to the displacement of pixels of the object in the first image between the first image and each frame of the video, where the second image is an image of a frame before the first image in the video;
[0149] An image obtained by rendering the second image using a ray tracing algorithm is determined as a second noise image.
[0150] In some embodiments of the present application, the processor 810 may be specifically configured to, when the rendering buffer data further includes first time series data related to the second noise image and used for time series propagation, perform data splicing processing on pixel values of the first noise image, pixel values of the second noise image, surface normal vector data, object surface reflectivity data, and the first time series data to obtain input spliced data;
[0151] The input spliced data is processed through the convolutional neural network to obtain image denoising data.
[0152] In some embodiments of the present application, the processor 810 may also be configured to align an object in the second noise image with an object in the first noise image according to a displacement of the object in the first noise image between the first noise image and the second noise image, to obtain an aligned second noise image;
[0153] The pixel values of the second noise image are replaced with the pixel values of the aligned second noise image to perform data splicing processing with the pixel values of the first noise image, the surface normal vector data, the object surface reflectivity data and the first time series data to obtain input splicing data.
[0154] In some embodiments of the present application, the processor 810 may be specifically configured to, when the image denoising data further includes a denoising weight coefficient, and the denoising weight coefficient is used to fuse the first denoised image and the second denoised image with the second noise image, sequentially perform at least one downsampling process and at least one upsampling process on the input spliced data through a convolutional neural network to obtain a sampling result, wherein the sampling result includes a first sampling number of the downsampling process, a second sampling number of the upsampling process, a sequence of the first noise image and the second noise image on a timeline, a first image noise change amount between the first noise image and the second noise image, and an estimated denoised image corresponding to the first noise image;
[0155] Determining the order of the kernel function based on the first sampling number or the second sampling number;
[0156] Generate first time series data based on the sequence of the first noise image and the second noise image on the timeline;
[0157] The normalized value of the noise variation of the first image is determined as a noise weight coefficient;
[0158] The normalized value of the second image noise variation is determined as the noise reduction weight coefficient, and the second image noise variation is the image noise variation between the first noisy image and the estimated noise reduction image.
[0159] In some embodiments of the present application, the processor 810 may be specifically configured to, when the noise weight coefficient includes a first noise weight coefficient and a second noise weight coefficient, adjust the pixel value of the first noise image according to the first noise weight coefficient to obtain a first adjusted noise image;
[0160] According to the second noise weight coefficient, pixel values of the second noise image are adjusted to obtain a second adjusted noise image;
[0161] The first adjusted noise image and the second adjusted noise image are fused to obtain a time series accumulated noise image.
[0162] In some embodiments of the present application, the processor 810 may be specifically configured to, in a case where at least two levels of kernel functions include a first level kernel function and a second level kernel function, perform bilinear difference downsampling of a first pixel multiple on the time series accumulated noise image by the first level kernel function to obtain first chromaticity coordinate information of the time series accumulated noise image;
[0163] Performing bilinear difference downsampling of a second pixel multiple on the time series accumulated noise image through a second-level kernel function to obtain second chromaticity coordinate information of the time series accumulated noise image, wherein the second pixel multiple is greater than the first pixel multiple;
[0164] A first noise reduction image is generated based on the first chromaticity coordinate information and the second chromaticity coordinate information.
[0165] In some embodiments of the present application, the processor 810 can also be used to use the first noise image, the second time series data related to the first noise image and used for time series propagation, the first denoised image and the time series accumulated denoised image of the first noise image as a third denoised image corresponding to the third noise image, where the third noise image is a noise image of a frame after the first noise image in the N frames of noise images.
[0166] It should be understood that the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of a static image or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 806 may include a display panel, and the display panel may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also referred to as a touch screen. The touch panel 8071 may include two parts, a touch detection device and a touch display. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as a volume display button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0167] The memory 809 can be used to store software programs and various data. The memory 809 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 809 can include a volatile memory or a non-volatile memory, or the memory 809 can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0168] The processor 810 may include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless display signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 810.
[0169] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image denoising method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0170] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0171] In addition, an embodiment of the present application further provides a chip, which includes a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image denoising method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0172] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0173] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image denoising method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0174] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0175] In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.
[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0177] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. An image denoising method, characterized in that: include: Acquire N noise images, where the noise images are images rendered by a ray tracing algorithm, and the N noise images include a first noise image and a second noise image, where the second noise image is a noise image of a frame before the first noise image, and N is an integer greater than 1; Determine image denoising data based on the N frames of noisy images and the rendering buffer data of the object in the first noisy image; The rendering buffer data includes surface normal vector data for reflecting the shape of the object and object surface reflectivity data for characterizing the reflective characteristics of the object surface, and the image denoising data includes at least two levels of kernel functions and noise weight coefficients; According to the noise weight coefficient, the first noise image and the second noise image are fused to obtain a time series accumulated noise image; The time-series accumulated noise image is subjected to hierarchical image denoising by at least two levels of kernel functions to obtain a first denoised image of the first noise image.
2. The method according to claim 1, characterized in that The image denoising data also includes a denoising weight coefficient; the method also includes: According to the denoising weight coefficient, a fusion process is performed on the first denoised image and a second denoised image of the second noise image to obtain a time-series cumulative denoised image corresponding to the first noise image.
3. The method according to claim 2, characterized in that The denoising weight coefficient includes a first denoising weight coefficient and a second denoising weight coefficient; the fusion processing of the first denoised image and the second denoised image of the second noise image according to the denoising weight coefficient to obtain a time-series cumulative denoised image corresponding to the first noise image includes: adjusting pixel values of the first denoised image according to the first denoising weight coefficient to obtain a first adjusted denoised image; adjusting the pixel values of the second denoised image according to the second denoising weight coefficient to obtain a second adjusted denoised image; The first adjusted denoised image and the second adjusted denoised image are fused to obtain a time-series cumulative denoised image corresponding to the first noisy image.
4. The method according to claim 1, characterized in that: The step of acquiring N frames of noisy images comprises: Rendering the first image in the video by using the ray tracing algorithm to obtain the first noise image, where the first noise image is a three-dimensional effect image of the first image; Acquire a second image from the video according to the displacement of pixels of the object in the first image between the first image and each frame of the image in the video, where the second image is an image of a frame before the first image in the video; An image obtained by rendering the second image using the ray tracing algorithm is determined as the second noise image.
5. The method according to claim 1, characterized in that The rendering buffer data also includes first time series data related to the second noise image and used for time series propagation; the image denoising data is determined based on the N frames of noise images and the rendering buffer data of the objects in the first noise image, including: Performing data splicing processing on the pixel values of the first noise image, the pixel values of the second noise image, the surface normal vector data, the object surface reflectivity data, and the first time series data to obtain input splicing data; The input spliced data is processed by a convolutional neural network to obtain the image denoising data.
6. The method according to claim 5, characterized in that Before performing data splicing processing on the pixel values of the first noise image, the pixel values of the second noise image, the surface normal vector data, the object surface reflectivity data and the first time series data to obtain input spliced data, the method further includes: aligning the object in the second noise image with the object in the first noise image according to the displacement of the object in the first noise image between the first noise image and the second noise image, to obtain an aligned second noise image; The pixel values of the second noise image are replaced with the pixel values of the aligned second noise image to perform data splicing processing with the pixel values of the first noise image, the surface normal vector data, the object surface reflectivity data and the first time series data to obtain input spliced data.
7. The method according to claim 5, characterized in that The image denoising data further includes a denoising weight coefficient, and the denoising weight coefficient is used to fuse the first denoised image with a second denoised image with the second noise image; The step of processing the input spliced data by a convolutional neural network to obtain the image denoising data includes: By means of the convolutional neural network, the input spliced data is sequentially subjected to at least one downsampling process and at least one upsampling process to obtain a sampling result, wherein the sampling result includes a first sampling number of the downsampling process, a second sampling number of the upsampling process, a sequence of the first noise image and the second noise image on a timeline, a first image noise variation amount between the first noise image and the second noise image, and an estimated denoised image corresponding to the first noise image; Determining the order of the kernel function based on the first sampling number or the second sampling number; Generate the first time series data based on the sequence of the first noise image and the second noise image on the timeline; Determine a normalized value of the noise variation of the first image as the noise weight coefficient; A normalized value of a second image noise variation is determined as the noise reduction weight coefficient, wherein the second image noise variation is an image noise variation between the first noisy image and the estimated noise reduction image.
8. The method according to claim 1, characterized in that The noise weight coefficient includes a first noise weight coefficient and a second noise weight coefficient; The step of fusing the first noise image and the second noise image according to the noise weight coefficient to obtain a time series accumulated noise image includes: adjusting pixel values of the first noise image according to the first noise weight coefficient to obtain a first adjusted noise image; adjusting pixel values of the second noise image according to the second noise weight coefficient to obtain a second adjusted noise image; The first adjusted noise image and the second adjusted noise image are fused to obtain the time series accumulated noise image.
9. The method according to claim 1, characterized in that: The at least two levels of kernel functions include a first level kernel function and a second level kernel function; The step of performing image hierarchical denoising on the time series accumulated noise image by using at least two levels of kernel functions to obtain a first denoised image of the first noise image includes: Performing bilinear difference downsampling of a first pixel multiple on the time series accumulated noise image by using the first-level kernel function to obtain first chromaticity coordinate information of the time series accumulated noise image; Performing bilinear difference downsampling of a second pixel multiple on the time-series accumulated noise image by using the second-level kernel function to obtain second chromaticity coordinate information of the time-series accumulated noise image, wherein the second pixel multiple is greater than the first pixel multiple; The first noise reduction image is generated based on the first chromaticity coordinate information and the second chromaticity coordinate information.
10. The method according to claim 2, characterized in that The method further comprises: The first noise image, the second time series data related to the first noise image and used for time series propagation, the first denoised image and the time series accumulated denoised image of the first noise image are used to determine a third denoised image corresponding to the third noise image, and the third noise image is a noise image of a frame after the first noise image in the N frames of noise images.