Method and device for reducing noise of rendered image

Abstract

The invention is suitable for the technical field of image processing, provides a noise reduction method and device for a rendered image, and can improve the image quality of the rendered image. The noise reduction method for the rendered image comprises the following steps: acquiring a first rendered image and first rendering data of each first pixel point in the first rendered image; acquiring second rendering data of each second pixel point, wherein each second pixel point is a pixel point at a position corresponding to each first pixel point in a previous frame of rendered image of the first rendered image; for each first pixel point, adjusting the pixel value of the first pixel point according to the difference between the first rendering data and the second rendering data to obtain the pixel adjustment value of the first pixel point; and inputting the pixel adjustment value of each first pixel point in the first rendered image and the first rendered data corresponding to the pixel adjustment value into a preset neural network for processing to obtain a denoised image of the first rendered image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method and device for denoising a rendered image. Background Art

[0002] In 3D medical image visualization tasks, the Monte Carlo method is mainly used to solve the rendering equation in order to speed up the rendering process. However, the Monte Carlo solution relies on a large number of samples, that is, the higher the number of samples per pixel (SPP), the higher the rendering quality, but the longer the processing time of a frame of image, resulting in image interaction jamming and difficulty in achieving real-time rendering effects. The lower the SPP, the more noise will be generated in the rendered image, resulting in a worse rendering effect.

[0003] In order to improve rendering efficiency, some solutions propose to perform Monte Carlo solvers based on low SPP to obtain rendered images with large noise, and then combine neural networks to further reduce the noise of the rendered images. However, due to the large difference in noise distribution in each frame of the rendered image, it is difficult to reconstruct image details, resulting in unsatisfactory noise reduction effects. Summary of the invention

[0004] The present application provides a method and device for denoising a rendered image, which can improve the picture quality of the rendered image.

[0005] In a first aspect, the present application provides a method for denoising a rendered image, the method comprising:

[0006] Acquire a first rendered image and first rendering data of each first pixel in the first rendered image;

[0007] Acquire second rendering data of each second pixel point, where each second pixel point is a pixel point at a position corresponding to each first pixel point in a rendering image of a previous frame of the first rendering image;

[0008] For each of the first pixel points, adjusting the pixel value of the first pixel point according to the difference between the first rendering data and the second rendering data to obtain a pixel adjustment value of the first pixel point;

[0009] The pixel adjustment value of each of the first pixels in the first rendered image and the first rendering data corresponding thereto are input into a preset neural network for processing to obtain a denoised image of the first rendered image.

[0010] In a possible implementation, the rendering data includes one or more of depth, viewing angle vector, raycast mixed color, gradient map, and ray bounce number map.

[0011] In a possible implementation manner, adjusting the pixel value of the first pixel point according to the first rendering data and the second rendering data to obtain the pixel adjustment value of the first pixel point includes:

[0012] Determine a target cumulative SPP and an accumulated SPP according to a difference between the first rendering data and the second rendering data, wherein the target cumulative SPP and the accumulated SPP are related to a historical frame that can be referenced by the first pixel point;

[0013] Determine a ratio of the target cumulative SPP to the accumulated SPP as the weight coefficient;

[0014] The pixel value of the first pixel and the pixel adjustment value of the second pixel are weighted according to the weight coefficient to obtain the pixel adjustment value of the first pixel.

[0015] In a possible implementation manner, determining the target accumulated SPP and the accumulated SPP according to the difference between the first rendering data and the second rendering data includes:

[0016] If the difference between the first rendering data and the second rendering data is greater than a preset first threshold, it is determined that the target accumulated SPP is the same as the accumulated SPP.

[0017] In a possible implementation, the method further includes:

[0018] The accumulated error of the first pixel is cleared.

[0019] In a possible implementation manner, determining the target accumulated SPP and the accumulated SPP according to the difference between the first rendering data and the second rendering data includes:

[0020] If the difference between the first rendering data and the second rendering data is less than or equal to a first threshold, obtaining a cumulative error of the second pixel;

[0021] Determine a cumulative error of the first pixel point according to the difference and the cumulative error of the second pixel point;

[0022] If the accumulated error of the first pixel point is greater than a second threshold, determining that a value obtained by reducing the actual accumulated SPP according to a first preset ratio is the accumulated SPP, and a difference between a value obtained by reducing the accumulated SPP according to a second preset ratio and the first preset value is the target accumulated SPP;

[0023] If the accumulated error of the first pixel point is less than or equal to the second threshold, the actual accumulated SPP is determined to be the accumulated SPP, and a value obtained by reducing the actual accumulated SPP according to a third preset ratio is determined to be the target accumulated SPP.

[0024] In a possible implementation, the neural network includes a U-net network and a kernel prediction network, the output features of at least two levels of decoders in the U-net network are input into the kernel prediction network, the U-net network is used to determine feature information of at least two filter kernels of different scales corresponding to the first pixel point, the kernel prediction network is used to denoise the input information according to the feature information of the filter kernels of at least two different scales to obtain a denoised image of the first rendered image, and the input information includes pixel adjustment values ​​of each of the first pixels in the first rendered image and the first rendering data corresponding thereto.

[0025] In a possible implementation, the denoising the input information according to the filter kernel feature information of the at least two different scales to obtain a denoised image of the first rendered image includes:

[0026] Obtaining the at least two filter kernels of different scales according to feature information of the at least two filter kernels of different scales;

[0027] Using at least two filter kernels of different scales to filter the input information respectively to obtain at least two feature maps;

[0028] The at least two feature maps are sequentially fused to obtain a denoised image of the first rendered image.

[0029] In a second aspect, the present application provides an image processing device, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any possible implementation of the first aspect is implemented.

[0030] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in the first aspect or any possible implementation of the first aspect is implemented.

[0031] In a fourth aspect, the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method as described in the first aspect or any possible implementation of the first aspect is implemented.

[0032] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0033] The denoising method for rendered images provided in the present application is first based on the time domain denoising method, and uses the difference between the second rendering data of each second pixel point in the previous frame of the first rendered image and the first rendering data of the first pixel point at the corresponding position in the first rendered image to adjust the pixel value of each first pixel point to complete the preliminary denoising of the first rendered image. After that, the pixel adjustment value of each first pixel point obtained by adjustment and the first rendering data are spliced ​​as input information and input into the neural network for denoising to obtain a denoised image. By combining the time domain denoising and deep learning methods, the time domain stability between adjacent rendered images is guaranteed while improving the denoising effect. And the method provided in the present application adjusts the pixel value for each pixel point, thereby avoiding the sudden change of the picture caused by the change of noise distribution in the rendered image, and improving the picture quality of the rendered image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A schematic diagram of a flow chart of a method for denoising a rendered image provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a network structure of a neural network provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a network structure of a core prediction network provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] For ease of understanding, some concepts related to the embodiments of the present application are exemplarily described for reference:

[0041] Image rendering refers to the process of converting three-dimensional (3D) coordinates into two-dimensional (2D) coordinates, and converting 2D coordinates into actual colored pixels, so as to achieve the purpose of presenting a three-dimensional image on a two-dimensional screen.

[0042] SPP refers to the number of samples collected when calculating the pixel value of a pixel during the rendering process. For example, taking ray tracing as an example, a ray of light is emitted from a viewpoint, and the ray of light collides with the scene after passing through the target pixel. The data collected at the collision point, such as depth, viewing angle vector, raycast mixed color, gradient map, ray bounce count map, etc., is a sample. By emitting light from different viewpoints, different samples can be collected. The larger the SPP, the more samples are used to calculate the target pixel, the more accurate the pixel value of the target pixel, and the higher the picture quality of the rendered image.

[0043] Since the hardware conditions of most image processing devices cannot support real-time rendering of high SPP, further denoising the rendered image obtained based on low SPP (for example, SPP=1) to obtain high-quality rendered images is a major research direction in the industry.

[0044] To this end, the present application provides a denoising method, which performs secondary denoising on the rendered image by combining time domain denoising with deep learning, thereby improving the denoising effect while ensuring the time domain stability between denoised images.

[0045] The noise reduction method provided in this application is exemplarily described below.

[0046] See also Figure 1 , is a flow chart of a method for denoising a rendered image provided by the present application, the denoising method is applied to an image processing device, the image processing device can be a mobile phone, a computer, a camera, and any terminal device that can realize image rendering. Figure 1 As shown, the noise reduction method provided by the present application may include:

[0047] S101, obtaining a first rendered image, first rendering data of each first pixel in the first rendered image, and second rendering data of each second pixel, wherein each second pixel is a pixel at a position corresponding to each first pixel in a rendered image of a frame previous to the first rendered image.

[0048] In an embodiment of the present application, the first rendered image is a non-first frame rendered image in a sequence of rendered images to be processed. The rendered image sequence is an image sequence obtained by rendering the three-dimensional data corresponding to the target three-dimensional scene. The three-dimensional data can be medical data, such as Digital Imaging and Communications in Medicine (DICOM) data, or game model data, movie data, animation data, map data, etc., and the present application does not limit this.

[0049] The specific method of rendering the target three-dimensional data to obtain the rendered image sequence to be processed is not limited in this application.

[0050] Rendering data refers to sample data collected during the rendering of target three-dimensional data. For example, rendering data may include one or more sample data such as depth, viewing angle vector, raycast mixed color, gradient map, and light bounce times map collected based on the collision point of light in the three-dimensional scene. In one example, when SPP=N (N is an integer greater than 1) is set during rendering, each pixel point can collect N groups of rendering data. Then, during denoising, N groups of rendering data can be obtained, or one group or part of the N groups of rendering data can be obtained. The specific setting can be based on the actual computing resources, computing efficiency, and denoising effect requirements.

[0051] It is worth noting that the denoising method provided in the present application can be used to denoise a rendered image sequence that has been rendered, and can also be used in a real-time rendering scene, that is, the denoising process can be started every time a frame of rendered image is rendered, and the denoising method provided in the present application can be used to perform denoising on the currently obtained rendered image.

[0052] When performing denoising on a rendered image sequence that has been rendered, denoising can be performed frame by frame in the playback order of the rendered image sequence, or in the reverse order of the playback order of the rendered image sequence. Then, the so-called previous frame of the first rendered image refers to the denoised adjacent frame of the rendered image before the first rendered image in the denoising order.

[0053] S102 , for each first pixel, adjusting the pixel value of the first pixel according to the difference between the first rendering data and the second rendering data to obtain a pixel adjustment value of the first pixel.

[0054] Considering that there is a certain continuity between frames in a moving scene, the denoising result of the previous frame of the rendered image can be reused in the next frame of the rendered image to reduce the noise in the first rendered image. Based on this, the embodiment of the present application proposes to use the correspondence between frames to adjust the pixel value of each pixel in the first rendered image respectively, reduce the influence of large-area noise, and avoid sudden changes in the denoised image due to changes in the noise distribution in the first rendered image.

[0055] The following takes a first pixel point in the first rendered image as an example to exemplarily explain the process of adjusting the pixel value of the first pixel point according to the difference between the first rendering data and the second rendering data. It is worth noting that the pixel value of each pixel point in the first rendered image can be adjusted, that is, each pixel point in the first rendered image is a first pixel point; the pixel points in a specified area in the first rendered image can also be adjusted, that is, each pixel point in the specified area in the first rendered image is a first pixel point, and this application does not impose any restrictions on this.

[0056] In some cases, for example, when a new object appears in the scene or the shooting shakes violently, the difference between the previous and next frames is large, so the denoising result of the previous frame is obviously not suitable for reuse in the current frame. In one example, it is possible to directly determine whether to use the pixel adjustment value of the second pixel to adjust the pixel value of the first pixel based on the difference between the first rendering data and the second rendering data.

[0057] For example, in the embodiment of the present application, a first threshold is set as a judgment condition. When the difference between the first rendering data and the second rendering data is greater than the preset first threshold, it indicates that the picture difference between the previous and next two frames is large, and the pixel adjustment value of the second pixel cannot be used to adjust the pixel value of the first pixel. Therefore, it can be determined that the pixel adjustment value of the first pixel is the pixel value of the first pixel.

[0058] When the difference between the first rendering data and the second rendering data is less than or equal to the first threshold, it indicates that the images of the two frames before and after are continuous, and the denoising result of the second pixel can be reused on the first pixel, so the pixel value of the first pixel and the pixel adjustment value of the second pixel can be weighted and summed according to the preset weight coefficient to obtain the pixel adjustment value of the first pixel. For example, the weight coefficients of the first pixel and the second pixel are 0.5, or 0.8 and 0.2, or 0.3 and 0.7, etc., respectively, and this application does not limit this.

[0059] Assume that the first rendered image is the i-th frame image in the rendered image sequence, and the pixel value of the first pixel is represented by x i , the pixel adjustment value of the second pixel is expressed as y i-1 , the weight coefficient of the first pixel is expressed as a i , then the pixel adjustment value y of the first pixel i The calculation can be expressed as:

[0060] y i =(1-α i )*y i-1 +α i *x i .

[0061] It can be understood that when calculating the pixel adjustment value of the first pixel, the pixel adjustment value of the second pixel is reused according to the weight coefficient, that is, the rendering data of the second pixel is reused, and the pixel adjustment value of the second pixel may reuse the rendering data of the pixel at the corresponding position in the i-2th frame rendering image. Therefore, it can be understood that although a smaller SPP is used during rendering, the SPP of the first rendered image is increased in the time domain by reusing the corresponding rendering data of the historical frame during the denoising process, thereby achieving a certain denoising effect and improving the picture quality.

[0062] In another example, the weight coefficient of the first pixel point can also be adaptively adjusted according to the difference between the first rendering data and the second rendering data, and the pixel value of the first pixel point and the pixel adjustment value of the second pixel point are weightedly summed according to the determined weight coefficient to obtain the pixel adjustment value of the first pixel point.

[0063] In one embodiment of the present application, the weight coefficient of the first pixel can be determined based on the SPP accumulated by the first pixel during the denoising process. For example, the target accumulated SPP (hereinafter referred to as ) and the accumulated SPP (hereinafter referred to as ), the ratio of the target cumulative SPP to the accumulated SPP is defined as the weight coefficient of the first pixel, that is, The target accumulated SPP and the accumulated SPP of the first pixel may be determined according to the difference between the first rendering data and the second rendering data.

[0064] Among them, the target cumulative SPP and the accumulated SPP are both related to the historical frames that the first pixel needs to refer to (i.e., multiple frames of rendered images that are arranged in the denoising order and denoised before the first rendered image), indicating the SPP that the first pixel needs to accumulate during the denoising process. That is, in this application, the SPP that the first pixel needs to accumulate during the denoising process can be adjusted according to the difference between the first rendering data and the second rendering data, thereby adjusting the corresponding weight coefficient.

[0065] Exemplarily, an embodiment of the present application provides a method for determining a target cumulative SPP and an accumulated SPP, which may be as follows:

[0066] S102a, if the difference between the first rendering data and the second rendering data (hereinafter may be represented as ε i ) is greater than a preset first threshold (hereinafter referred to as β hard ), it is determined that the target cumulative SPP is the same as the accumulated SPP.

[0067] When the difference between the first rendering data and the second rendering data is greater than a preset first threshold, it indicates that the first rendering image is significantly different from the previous rendering image. At this time, the denoising result of the previous rendering image is not suitable for reuse in the first rendering image. Therefore, it can be determined that the target cumulative SPP and the accumulated SPP are the same, so that the weight coefficient of the first pixel is 1, and the corresponding weight coefficient of the second pixel is 0, that is, the denoising result of the second pixel is discarded, so that the pixel adjustment value of the first pixel is still the pixel value of the first pixel.

[0068] For example, the target cumulative SPP and the accumulated SPP are both set to a first preset value (hereinafter referred to as n min ). That is, when When the weight coefficient of the first pixel is 1, the corresponding weight coefficient of the second pixel is 0, i =x i .

[0069] S102b, if the difference between the first rendering data and the second rendering data is less than or equal to the first threshold, obtain the accumulated error of the second pixel (hereinafter represented as ), and according to the difference between the first rendering data and the second rendering data (hereinafter represented as ε i ) and the accumulated error of the second pixel point determines the accumulated error of the first pixel point (hereinafter represented as ),Right now

[0070] It can be understood that, based on the same principle, the cumulative error of the second pixel It is calculated based on the difference between the rendering data of the second pixel point and the rendering data of the pixel point at the corresponding position in the i-2th frame rendering image, and the cumulative error of the pixel point at the corresponding position in the i-2th frame rendering image.

[0071] Optionally, if the first rendered image is not the last frame of the rendered image sequence, then after the above step 102a, when the pixel value of the first pixel point does not refer to the denoising result of the historical frame, the cumulative error of the first pixel point can be cleared to zero, so that the rendered image after the first rendered image starts the cumulative calculation with the first rendered image.

[0072] It can be understood that the farther away the historical frame from the first rendered image is, the weaker the continuity between the historical frame and the first rendered image is, and the greater the difference between the historical frame and the first rendered image is. As the SPP accumulates in the time domain, the error between the rendering data of the pixel points at the relative position in the historical frame and the rendering data of the first pixel point is also accumulating. Then when the accumulated error of the first pixel point is large, it may affect the denoising effect of the pixel adjustment value of the first pixel point. Therefore, when weighting, the weight coefficient of the second pixel point can be reduced and the weight coefficient of the first pixel point can be increased to reduce the impact of the accumulated error of the first pixel point on the pixel adjustment value.

[0073] S102c, if the accumulated error of the first pixel is less than or equal to the second threshold (hereinafter referred to as β soft ), the actual accumulated SPP is determined to be the accumulated SPP, and the value of the actual accumulated SPP reduced according to the third preset ratio is the target accumulated SPP.

[0074] The actual accumulated SPP is the total SPP currently accumulated. Assuming that M (1≤M≤N) sets of rendering data are used in the denoising process, the actual accumulated SPP of the first pixel is the accumulated SPP of the second pixel (hereinafter can be expressed as ) plus M, For example, in order to improve the denoising efficiency, a set of rendering data is retained in the denoising process to calculate the weight coefficient, then the actual accumulated SPP of the first pixel is the accumulated SPP of the second pixel plus 1.

[0075] Assuming that the third preset ratio is represented by d, when hour, d is less than or equal to A positive number.

[0076] In the embodiment of the present application, the second threshold is set as the upper limit of the cumulative error of the first pixel point. When the cumulative error of the first pixel point is less than or equal to the second threshold, it means that the currently accumulated error is within the allowable range, so the denoising result of the second pixel point can be reused as much as possible to obtain the pixel adjustment value of the first pixel point. That is, the third preset ratio can be set as large as possible, so that the weight coefficient of the first pixel point is as small as possible, and the weight coefficient of the second pixel point is as large as possible.

[0077] For example, Can be set to 1, that is, the third preset ratio is Then when hour,

[0078] S102d, if the accumulated error of the first pixel is greater than a preset second threshold (hereinafter referred to as β soft), then determine the actual cumulative SPP (hereinafter can be expressed as ) The value obtained by reducing the accumulated SPP according to the first preset ratio is the accumulated SPP, the difference between the value obtained by reducing the accumulated SPP according to the second preset ratio and the first preset value is the target accumulated SPP, and the target accumulated SPP is less than the first preset value.

[0079] In an embodiment of the present application, when the accumulated error of the first pixel point is greater than the second threshold, it means that the currently accumulated error exceeds the allowable range, which may affect the accuracy of the pixel adjustment value of the first pixel point. It is necessary to increase the weight coefficient of the first pixel point, reduce the weight coefficient of the second pixel point, and reuse the denoising result of the second pixel point in small amounts to obtain the pixel adjustment value of the first pixel point.

[0080] Assuming that the first preset ratio is represented by b and the second preset ratio is represented by c, when hour, The first preset ratio and the second preset ratio may be equal, for example, b=c=2; the first preset ratio and the second preset ratio may also be different.

[0081] S103: Inputting the pixel adjustment value of the first pixel point in the first rendered image and the first rendering data into a preset neural network for processing to obtain a denoised image of the first rendered image.

[0082] It is worth noting that in the above step S102, after adjusting the pixel values ​​of each first pixel in the first rendered image using the denoising result of the previous frame of the rendered image, the time domain accumulated image may have a pixel adjustment value of each first pixel reused the denoising result of the second pixel, and there may be some first pixels that do not reuse the denoising result of the second pixel due to the large difference between the rendering data, so these first pixels still have large noise. In addition, if the first rendered image only adjusts the pixel values ​​of the pixels in the designated area (such as the middle area), and the pixels in the non-designated area (such as the marginalized area) are not adjusted, then the non-designated area in the time domain accumulated image still has large noise. Therefore, after the pixel value adjustment, the time domain accumulated image still has the problem of uneven noise distribution. For this reason, the embodiment of the present application proposes to further reduce the noise of the time domain accumulated image using a neural network to obtain the final denoised image of the first rendered image.

[0083] In one example, the pixel adjustment value of each first pixel point and its corresponding first rendering data can be spliced ​​along the channel direction to obtain the input information of the neural network. It can be understood that if there is a non-specified area in the first rendered image, then the input information also includes the pixel value and rendering data of each pixel point in the non-specified area in the first rendered image.

[0084] like Figure 2 As shown, a network structure of a neural network provided in an embodiment of the present application includes a U-net network and a prediction module. Input information is input into the U-net network and the prediction module respectively. The U-net network includes multiple layers ( Figure 2 The encoders (taking 4 levels as an example) and the decoders corresponding to the encoders are connected one by one. The encoders and the corresponding decoders are connected by jump. At least two levels ( Figure 2 The output of the decoder (taking the last 3 levels as an example) is connected to the prediction module.

[0085] In an embodiment of the present application, the U-net network is used to determine the feature information of at least two filter kernels of different scales corresponding to the first pixel point. Compared with the method of directly reconstructing pixel values ​​through the U-net network, the filter kernel is predicted through the U-net network in the embodiment of the present application, so that the U-net network can normalize the data when processing input information to avoid extreme values, thereby reducing the risk of overfitting of the U-net network, producing an implicit regularization effect, and obtaining a more stable denoising effect.

[0086] Among them, the encoder can use a convolutional layer or a network structure connecting a convolutional layer with a downsampling layer, and form a downsampling path through multiple levels of encoders to gradually reduce the size of the feature map to extract context information of different scales. The decoder can use a transposed convolutional layer or a network structure connecting a convolutional layer with an upsampling layer, and form an upsampling path through multiple levels of decoders to gradually restore the size of the feature map, extract detailed features, and obtain feature information of corresponding size and filtering. The jump connection transmits the feature map output by the encoder to the decoder of the corresponding level to achieve the fusion of feature maps of different sizes to improve the prediction accuracy of the decoder.

[0087] The prediction module is used to perform denoising processing on the input information according to feature information of at least two filter kernels of different scales output by the U-net network to obtain a denoised image of the first rendered image.

[0088] Among them, the prediction module can be any prediction neural network that has the function of denoising the input noise image based on the feature information of filter kernels of different scales, for example, a network structure based on a long short-term memory neural network, or a network structure based on a combination of a generator and a discriminator.

[0089] Optionally, the prediction module may also be a kernel prediction network, including a fusion unit and at least two prediction paths. The at least two prediction paths are respectively used to obtain at least two filter kernels of different scales according to feature information of at least two filter kernels of different scales, and then filter the input information using at least two filter kernels of different scales to obtain at least two feature maps; the fusion unit is used to fuse the at least two feature maps in sequence to obtain a denoised image.

[0090] For example, based on Figure 2 The neural network shown in FIG. 1 takes the prediction module including three prediction paths as an example, Figure 3 As shown, it is a schematic diagram of the network structure of a kernel prediction network provided in an embodiment of the present application. The prediction path includes a convolutional coding layer, a decoding layer and a filtering unit. After the convolutional coding layer receives the feature information of the filter kernel output by the U-net network, it encodes the feature information to obtain a single-channel coding map, and then the decoding layer processes each pixel local area in the single-channel coding map with a convolution kernel of a preset size (for example, 10*10) to obtain a filter kernel of a corresponding size; the filtering unit performs filtering processing on the input information according to the filter kernel output by the decoding layer to obtain a feature map of the corresponding scale, and inputs the feature map into the fusion unit.

[0091] It can be understood that in the process of filter kernel prediction by the kernel prediction network, its input information is the feature information of the filter kernel, and the feature information is encoded into a single-channel coding map during processing. The operator structure is simple, thus greatly improving the inference speed of the kernel prediction network.

[0092] The fusion unit receives three feature maps of different scales (assuming that they are represented as features Figure 1 ,feature Figure 2 and Features Figure 3 ) after which the pair is characterized Figure 1 ,feature Figure 2 and Features Figure 3 The denoised image is obtained by fusion.

[0093] Exemplary, characteristic Figure 1 ,feature Figure 2 and Features Figure 3 The scale of increases successively, then the fusion unit can first Figure 1 and Features Figure 2 Fusion to obtain features Figure 4 . Then for the features Figure 4 and Features Figure 3 The denoised image is obtained by fusion. Assuming that when the fusion unit fuses two feature maps, the small-scale feature map of the two feature maps is represented as x c , the large-scale feature map is represented as x f , then in one example, the fusion unit can fuse the feature maps based on the following method:

[0094] o=x f -α(UDx f )+α(Ux c )

[0095] Where α is a trainable weight, U is 2x2 upsampling and D is 2x2 downsampling.

[0096] In one example, when training the neural network provided in an embodiment of the present application, a hybrid loss function combining the symmetric mean absolute percentage error (SMAPE) and the high-frequency error norm (HFEN) can be used as the objective function for training. The training set used includes training samples and label values. Among them, the training samples use the original data from data sets of multiple different organizations, and are obtained through different rendering methods (including different rendering algorithms and different rendering parameter settings). For example, the original data is processed according to the above step S102 and spliced ​​with the corresponding rendering parameters to obtain the training sample. The label value corresponding to the training sample is based on the image obtained by the rendering method of SPP (for example, SPP=5000).

[0097] Through verification, the neural network provided by the embodiment of the present application can effectively improve the quality of the rendered picture, and at the same time, the structural similarity index (SSIM), peak signal-to-noise ratio (PSNR) and time domain stability indicators are significantly improved. And because the neural network has a faster reasoning speed, it is suitable for real-time rendering scenes and can ensure the real-time requirements of real-time rendering scenes.

[0098] Of course, in addition to the network structure provided in the above-mentioned embodiment of the present application, other network structures that can complete the denoising task can also be used, for example, convolutional neural networks of various structures, including deep convolutional neural networks, U-Net networks, recursive networks, etc., which are not limited to this application.

[0099] After the denoised image is obtained, the denoised image can be color mapped and sent to a display for display. Alternatively, the denoised image can be stored for subsequent display when needed.

[0100] In summary, the denoising method provided in the embodiment of the present application first uses the denoising result of the previous frame of the rendered image to adjust the pixel values ​​of each first pixel in the first rendered image respectively through time domain denoising to obtain a time domain cumulative image, and then further improves the image quality of the time domain cumulative image based on a neural network. By combining time domain denoising with deep learning, the rendered image is reconstructed, and the denoising effect is effectively improved while ensuring the denoising speed, thereby improving the picture quality of the rendered image.

[0101] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application provides an image processing device, and the device embodiment corresponds to the above method embodiment. For ease of reading, the present device embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the device in this embodiment can correspond to and implement all the contents in the above method embodiment.

[0102] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application includes:

[0103] An acquisition unit 401 is used to acquire a first rendered image and first rendering data of each first pixel in the first rendered image, and to acquire second rendering data of each second pixel, where each second pixel is a pixel at a position corresponding to each first pixel in a rendered image of a previous frame of the first rendered image;

[0104] a pixel adjustment unit 402, configured to adjust, for each of the first pixel points, a pixel value of the first pixel point according to a difference between the first rendering data and the second rendering data, to obtain a pixel adjustment value of the first pixel point;

[0105] The denoising unit 403 is used to input the pixel adjustment value of each first pixel point in the first rendered image and the first rendering data corresponding thereto into a preset neural network for processing to obtain a denoised image of the first rendered image.

[0106] The image processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0107] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned image processing device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0108] Based on the same inventive concept, an embodiment of the present application also provides an image processing device. Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the image processing device provided in this embodiment includes: a memory 501 and a processor 502, the memory 501 is used to store computer programs; the processor 502 is used to execute the method described in the above method embodiment when calling the computer program.

[0109] The image processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0110] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in the above method embodiment is implemented.

[0111] The embodiment of the present application also provides a computer program product. When the computer program product is run on an image processing device, the image processing device implements the method described in the above method embodiment when executing the computer program product.

[0112] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media can include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0114] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0117] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0118] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0119] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0120] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0121] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for denoising a rendered image, characterized in that: The method comprises: Acquire a first rendered image and first rendering data of each first pixel in the first rendered image; Acquire second rendering data of each second pixel point, where each second pixel point is a pixel point at a position corresponding to each first pixel point in a rendering image of a previous frame of the first rendering image; For each of the first pixel points, adjusting the pixel value of the first pixel point according to the difference between the first rendering data and the second rendering data to obtain a pixel adjustment value of the first pixel point; The pixel adjustment value of each of the first pixels in the first rendered image and the first rendering data corresponding thereto are input into a preset neural network for processing to obtain a denoised image of the first rendered image.

2. The method according to claim 1, characterized in that The rendering data includes one or more of depth, viewing angle vector, raycast mixed color, gradient map, and ray bounce number map.

3. The method according to claim 1, characterized in that The step of adjusting the pixel value of the first pixel point according to the first rendering data and the second rendering data to obtain the pixel adjustment value of the first pixel point includes: Determine a target cumulative SPP and an accumulated SPP according to a difference between the first rendering data and the second rendering data, wherein the target cumulative SPP and the accumulated SPP are related to a historical frame that can be referenced by the first pixel point; Determine a ratio of the target cumulative SPP to the accumulated SPP as a weight coefficient; The pixel value of the first pixel and the pixel adjustment value of the second pixel are weighted according to the weight coefficient to obtain the pixel adjustment value of the first pixel.

4. The method according to claim 3, characterized in that The determining, according to the difference between the first rendering data and the second rendering data, a target accumulated SPP and an accumulated SPP comprises: If the difference between the first rendering data and the second rendering data is greater than a preset first threshold, it is determined that the target accumulated SPP is the same as the accumulated SPP.

5. The method according to claim 4, characterized in that The method further comprises: The accumulated error of the first pixel is cleared.

6. The method according to claim 3, characterized in that The determining, according to the difference between the first rendering data and the second rendering data, a target accumulated SPP and an accumulated SPP comprises: If the difference between the first rendering data and the second rendering data is less than or equal to a first threshold, obtaining a cumulative error of the second pixel; Determine a cumulative error of the first pixel point according to the difference and the cumulative error of the second pixel point; If the accumulated error of the first pixel point is greater than a second threshold, determining that a value obtained by reducing the actual accumulated SPP according to a first preset ratio is the accumulated SPP, and a difference between a value obtained by reducing the accumulated SPP according to a second preset ratio and the first preset value is the target accumulated SPP; If the accumulated error of the first pixel point is less than or equal to the second threshold, the actual accumulated SPP is determined to be the accumulated SPP, and a value obtained by reducing the actual accumulated SPP according to a third preset ratio is determined to be the target accumulated SPP.

7. The method according to any one of claims 1 to 6, characterized in that: The neural network includes a U-net network and a kernel prediction network. The output features of decoders of at least two levels in the U-net network are input into the kernel prediction network. The U-net network is used to determine feature information of filter kernels of at least two different scales corresponding to the first pixel point. The kernel prediction network is used to denoise the input information according to the feature information of the filter kernels of at least two different scales to obtain a denoised image of the first rendered image. The input information includes pixel adjustment values ​​of each of the first pixels in the first rendered image and the first rendering data corresponding thereto.

8. The method according to claim 7, characterized in that The denoising process is performed on the input information according to the filter kernel feature information of the at least two different scales to obtain a denoised image of the first rendered image, including: Obtaining the at least two filter kernels of different scales according to feature information of the at least two filter kernels of different scales; Using at least two filter kernels of different scales to filter the input information respectively to obtain at least two feature maps; The at least two feature maps are sequentially fused to obtain a denoised image of the first rendered image.

9. An image processing device, characterized in that: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images