Image noise reduction method, device, computer equipment and medium

By denoising the noise reduction synthesis and inter-frame information multiplexed images in the rendered video stream, the problem of low noise reduction quality of rendered video streams in the prior art is solved, and the noise reduction quality of rendered images is improved and efficiency.

CN114663314BActive Publication Date: 2024-12-06HANGZHOU QUNHE INFORMATION TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210321004.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-29
Publication Date
2024-12-06
Estimated Expiration
2042-03-29

AI Technical Summary

Technical Problem

The low noise reduction quality of existing rendered video streams leads to poor efficiency and effectiveness of rendered images.

Method used

By obtaining multi-frame rendered images in the rendered video stream, each frame of the image includes multiple layer information, performing noise reduction synthesis processing, obtaining the target object of the target rendered image and the adjacent frame rendered image, and adjusting the pixel points according to the position of the target object Color value to achieve image noise reduction.

Benefits of technology

It improves the noise reduction quality and noise reduction efficiency of the rendered image, reduces the time overhead of the algorithm, reduces the noise reduction error, and greatly utilizes layer information and inter-frame information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114663314B_ABST
    Figure CN114663314B_ABST
Patent Text Reader

Abstract

The embodiments of the present application belong to the technical field of computer graphics, and relate to an image denoising method, apparatus, computer equipment and medium thereof. The method comprises obtaining a rendered video stream, wherein the rendered video stream comprises multiple frames of rendered images, each frame of the rendered image comprises multiple layer information, performing denoising synthesis processing on each layer information in each frame of the rendered image to obtain a pre-denoised rendered image, obtaining a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image, and adjusting the color value of a pixel point of the target object in the target rendered image according to a position of the target object to obtain a denoised target rendered image. On the one hand, it is beneficial to reduce the denoising error occurring in the concentrated denoising process of multiple frame rendered images, and on the other hand, it realizes the reuse of information between frames, and improves the denoising quality and denoising efficiency of the rendered image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics, and in particular to an image denoising method, apparatus, computer equipment and medium thereof. Background Art

[0002] Rendering an image means rendering a scene image by adjusting parameters such as light, color, and angle, so that the generated scene image can vividly display the rendered design effect. In the graphics pipeline, rendering is the last important step, through which the final display effect of the 3D model and animation is obtained.

[0003] Today's video rendering technology is developing rapidly, and high requirements are placed on the efficiency and effect of rendering. Usually, a single-frame image rendering is divided into a scene rendering stage and a post-processing stage. The scene rendering stage mainly obtains a rough rendering image through the rendering calculation of a certain number of samples. This image usually contains a certain amount of noise, while the post-processing stage uses a series of methods to process this rough rendering effect image into a high-quality final rendering result image. The sequence frame denoising commonly used on the market is generally one-way denoising, resulting in poor denoising quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image denoising method, apparatus, computer equipment and medium thereof to solve the problem of low denoising quality of rendered video streams.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an image noise reduction method, including:

[0006] Obtain a reference pseudo-color image of the scene image;

[0007] Obtaining a rendered image of the scene image according to the reference pseudo-color image and preset rendering engine information;

[0008] Convert the rendered image to a false color image;

[0009] Calculate the similarity value between the pseudo-color image and the reference pseudo-color image;

[0010] Evaluates the rendering quality of the rendered image based on the similarity value.

[0011] In order to solve the above technical problems, the embodiment of the present application further provides an image noise reduction device, comprising:

[0012] An acquisition module is used to acquire a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, and each frame of the rendered image includes multiple layers of information;

[0013] A denoising and synthesis module, used for performing denoising and synthesis processing on each layer information in each frame of rendered image to obtain a pre-denoised rendered image;

[0014] A determination module, used for acquiring a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image;

[0015] The adjustment module is used to adjust the color value of the pixel point of the target object in the target rendered image according to the position of the target object to obtain the target rendered image after noise reduction.

[0016] In order to solve the above technical problems, an embodiment of the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above image denoising method are implemented.

[0017] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image denoising method are implemented.

[0018] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0019] By acquiring a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, each frame of the rendered image includes multiple layer information, performing denoising and synthesis processing on each layer information in each frame of the rendered image to obtain a pre-denoised rendered image, obtaining a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image, and adjusting the color value of the pixel point of the target object in the target rendered image according to the position of the target object to obtain the denoised target rendered image. On the one hand, by denoising the layer information of each frame of the rendered image, it is beneficial to reduce the denoising error occurring in the centralized denoising process of the multi-frame rendered images. On the other hand, the position and pixel points of the target rendered image and the adjacent frame rendered image are utilized to realize the reuse of inter-frame information, that is, the layer information is greatly utilized, while ensuring the rendering quality, the time overhead of the algorithm is also reduced, and the denoising quality and denoising efficiency of the rendered image are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0022] Figure 2 is a flow chart of an embodiment of the image noise reduction method of the present application;

[0023] Figure 3 is a schematic diagram of decoupling a rendered image to obtain different layer information in an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the process of noise reduction synthesis of layer information in an embodiment of the present application;

[0025] Figure 5 This is another example diagram of decoupling the rendered image to obtain different layer information in the embodiment of the present application;

[0026] Figure 6 is an example diagram of filters with different radii in an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of an inter-frame rendered image in a rendering flow in the implementation of the present application;

[0028] Figure 8 is a schematic diagram of an embodiment of the image noise reduction device of the present application;

[0029] Fig. 9 It is a basic structural block diagram of the computer device of this application. DETAILED DESCRIPTION

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0033] Based on this, the present application provides an image denoising method to solve the above technical problems.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, network 104 and server 105. Network 104 is used to provide a medium for communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0037] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0038] It should be noted that the image noise reduction method provided in the embodiment of the present application is executed by a server / terminal device, and accordingly, the image noise reduction device is generally arranged in the server / terminal device.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Continue to refer Figure 2 , Figure 2 A flowchart of an embodiment of an image noise reduction method includes:

[0041] S201: Obtain a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, and each frame of the rendered image includes multiple layers of information.

[0042] In an embodiment of the present application, the rendered video stream is a sequence frame rendered image of a video stream that is initially rendered by a rendering engine software, that is, multiple frames of rendered images in the rendered video stream are sorted according to a time sequence. Among them, the video stream can be uploaded to the terminal by a user, so that the rendering engine software on the terminal decodes and renders the video stream. The acquisition of the rendered video stream can also be extracted from a database to the terminal, or the user uploads the rendered video stream to the terminal via Bluetooth or WiFi, etc. The method of rendering the video stream is not limited here.

[0043] Furthermore, after decoding the obtained rendering video stream to obtain multiple frames of rendering images, the layer information of each frame of rendering image is extracted through a preset renderer, and the layer information includes various layer types and information corresponding to each layer type. Among them, the layer types include but are not limited to normal layer, noise level layer, object ID layer, diffuse color layer, specular color layer, refraction color layer and shadow layer.

[0044] Specifically, the normal layer is used to obtain the normal information of each pixel in the current frame rendering image; the noise level layer is used to obtain the noise level information of each pixel in the current frame rendering image. The brighter the color, the higher the noise level; the object ID layer is used to obtain the object ID identification of each pixel in the current frame rendering image, that is, the object identification label. The object identification label is independent for each object in the scene. Each independent object The ID identifier can use a separate color identifier, which can be used to identify the area where each object is located during the denoising process; the diffuse color layer is used to obtain the diffuse color layer of each pixel in the current frame rendering image. The diffuse color layer represents the diffuse inherent color of the pixel and does not include the diffuse color of the light; the specular color layer is used to obtain the specular color layer of each pixel in the current frame rendering image. The specular color layer represents the specular inherent color of the pixel and does not include the specular color of the light; the refraction color layer is used to obtain the refraction color layer of each pixel in the current frame rendering image. The refraction color layer represents the refraction inherent color of the pixel and does not include the refraction color of the light; the shadow layer is used to obtain the refraction color layer of each pixel in the current frame rendering image. The shadow layer represents the refraction inherent color of the pixel and does not include the refraction color of the light.

[0045] S202: Perform denoising and synthesis processing on each layer information in each frame of the rendered image to obtain a pre-denoised rendered image.

[0046] In order to convert each frame of the rendered image of the rendered video stream from a rough rendering effect into a high-quality rendering result, the embodiment of the present application performs pre-denoising processing on each frame of the rendered image. The pre-denoising processing includes denoising the layer information of each frame of the rendered image and synthesizing the denoised layer information, thereby improving the rendering quality of each frame of the rendered image after pre-denoising.

[0047] In some implementations, performing denoising and synthesis processing on each layer information in each frame of rendered image to obtain a pre-denoised rendered image includes:

[0048] Decouple each frame of rendered image by layers to obtain multiple layer information of each frame of rendered image;

[0049] Performing noise reduction processing on each layer information to obtain multiple layer information after noise reduction in each frame of rendered image;

[0050] The denoised multiple layer information is synthesized to obtain a pre-denoised rendered image.

[0051] Since the inter-frame information of the rendered image is mixed with various changing light sources, occlusion information changes between objects, and even objects moving in and out, it is easy to cause errors when performing denoising on the rendered image. Therefore, performing frame denoising on each rendered image can minimize the probability of denoising errors between frames.

[0052] Exemplarily, the embodiment of the present application adopts a preset single-frame noise reduction algorithm, that is, a noise reduction synthesis processing method is adopted to achieve a high-quality noise reduction effect for each frame of rendered image.

[0053] In some implementations, the single-frame noise reduction algorithm may be SVGF (Spatiotemporal Variance-Guided Filter), which is not limited here. The SVGF method is mainly used in games and is a high-quality real-time noise reduction method.

[0054] In some implementations, performing layer decoupling on each frame of rendered image to obtain multiple layer information of each frame of rendered image includes:

[0055] Get multiple layers of information of the rendered image according to the preset renderer.

[0056] Among them, each renderer has the function of rendering a channel map. The channel map refers to a map that only renders certain attribute information of the scene. The channel map is the layer information described in the embodiment of the present application.

[0057] Specifically, the parameters required by the layer type in the layer information to be decoupled are set in a preset renderer. After the parameters are set, the rendered image is decoupled through the renderer. After the decoupling is completed, multiple layer information corresponding to each frame of the rendered image and the image corresponding to each layer information are output. The specific layer information can be referred to the above embodiment and will not be described in detail here.

[0058] Furthermore, by decoupling the renderer, the layer information of each frame renderer is performed synchronously, that is, the decoupling of the layer information is consistent, the resolution and size of the output layer information are consistent, and the format of the layer information can be png or jpg, such as Figure 3 As shown, Figure 3 This is a schematic diagram of decoupling the rendered image to obtain different layer information.

[0059] For example, Figure 4 As shown, Figure 4 The figure is a flow chart of the denoising synthesis of layer information in the embodiment of the present application. In the process of denoising synthesis of a single frame, firstly, the rough rendered image is layer decoupled, the decoupled layer information is divided into a refraction color layer and a non-refraction color layer, and the layer information of the refraction part and the layer information of the non-refraction part are denoised respectively, and finally, the denoising result of the refraction color layer and the denoising result of the non-refraction color layer are further synthesized to obtain the pre-denoised rendered image. Among them, Figure 4 The noise reduction algorithm shown in can use ATrous (dilated convolution) filters.

[0060] Furthermore, layer decoupling is beneficial for subsequent denoising of multiple layers of materials in the rendered image. For example, denoising of both diffuse reflection layer and refraction color layer information of the object in the rendered image can be achieved.

[0061] It should be noted that the layer decoupling process sometimes decouples more than two layers of information, that is, the specular reflection layer and even the self-luminous layer in the rendered image will be decoupled, such as Figure 5 As shown, Figure 5 This is another example diagram of decoupling the rendered image to obtain different layer information in the embodiment of the present application. It can be seen that Figure 5 After layer decoupling, the rendered image in the above example obtains three layers of information, namely, diffuse reflection layer, refraction layer and specular reflection layer. The more layer information obtained by decoupling, the greater the time overhead of the denoising algorithm of the layer information. Therefore, in the embodiment of the present application, two types of layer information representing the rendered image are generally selected for denoising synthesis to improve the efficiency of denoising synthesis.

[0062] In some implementations, performing noise reduction processing on each layer information to obtain multiple layers of noise-reduced information in each frame of rendered image includes:

[0063] Get the pixel points of each layer information;

[0064] Perform filtering processing on each pixel point in the layer information to obtain a plurality of filtered layer information corresponding to each layer information, wherein each filtered layer information includes filtered pixel information;

[0065] The filtered pixel information in each filtered layer information is averaged to obtain the denoised layer information.

[0066] In some implementations, filtering each pixel in the layer information to obtain a plurality of filtered layer information corresponding to each layer information includes:

[0067] Get multiple filters;

[0068] Each pixel point in each layer information is input into a plurality of filters respectively, so as to output a plurality of filtered layer information corresponding to each layer information.

[0069] Exemplarily, the filters in the embodiments of the present application are Atrous filters of different radii. The Atrous filter is a type of atrous convolution that introduces a parameter called "dilation rate" into the convolution layer. This parameter defines the spacing between values ​​when the convolution kernel processes data. Therefore, it has a wider receptive field than general convolution, which can be intuitively expressed as a pixel in the screen space that can obtain more information through the Atrous filter, and at the same time, the filtering efficiency will not be reduced a lot due to accessing too many surrounding pixels.

[0070] like Figure 6 As shown, Figure 6 It is an example diagram of filters with different radii in the embodiment of the present application. Figure 6 It shows the pixels visited by the Atrous filter when the Atrous filter radius is 1, 2, and 4, that is, Figure 6 The solid dots in .

[0071] In some implementations, the filtered pixel information includes the weight of the filtered pixel point, the weight value represents the mapped color value, and the filtered pixel information in each filtered layer information is averaged to obtain the denoised layer information, including:

[0072] According to the position of the same filtering pixel point, the weights of multiple filtering pixels are obtained;

[0073] The weights of multiple filtered pixels are averaged, and the averaged weights are used as the noise reduction weights of the filtered pixel positions. The weight values ​​of the denoised layer information are constructed through the noise reduction weights.

[0074] Exemplarily, each pixel in each layer information is filtered five times by an ATrous filter with a radius ranging from 1 to 5, and five filtered layer information corresponding to one layer information is obtained. The filtered layer information is the layer information obtained by different filter processing, and the weight of each filtered pixel in each filtered layer information is dynamically evaluated, and the weights of the filtered pixel positions at the same position on the filtered layer information obtained after five filtering processes are added and averaged to obtain the final weight of the pixel, that is, the denoising weight. The denoising weight is used to represent the color value of the pixel in the layer information after denoising. For example, taking the refraction color layer as an example, the target pixel at the target position of the refraction color layer is filtered five times, and the weight values ​​obtained by the filtering process are 2, 2, 2, 4 and 2 respectively. At this time, the pixel value of the target pixel is (2+2+2+4+2) / 5=2.4.

[0075] It can be seen from the filtering process that the Atrous filter performs a weighted evaluation on the contribution of each pixel, and the weighted evaluation is generally performed based on the pixel of the layer information and the filtered pixel information filtered in the Atrous filter. The layer information that has been filtered includes the target normal information, target shadow information, inherent color information, and noise level information inherent in the rendered image. Among them, the target normal information, target shadow information, and target noise level information are obtained from the normal layer, shadow layer, and noise level layer in the rendered image, respectively, while the inherent color information is obtained in different ways depending on the type of layer decoupling, for example, the inherent color information obtained from the diffuse color layer, and so on.

[0076] The target normal information is used to identify the normal information of each pixel, and the target shadow information is used for the refraction color layer of each pixel. After obtaining the layer information through the renderer, the pixel information of each layer information can be read.

[0077] Furthermore, the weight evaluation of the filtered pixel is calculated based on the target normal information N, the target shadow information S, the intrinsic color information F, and the noise level A.

[0078] Exemplarily, a quaternion function f can be used to represent the weight of the filtered pixel point Weight = f(Ns-Nd, Ss-Sd, Fs-Fd, As), where Ns-Nd represents the difference between the normal information Ns of the current pixel point to be filtered and the normal information Nd of the surrounding pixels participating in the filtering, Ss-Sd represents the difference between the shadow information Ss of the current pixel point to be filtered and the shadow information Sd of the surrounding pixels, Fs-Fd represents the difference between the inherent color information Fs of the current pixel point to be filtered and the inherent color information Fd of the surrounding pixels, and As is the noise level of the current pixel point to be filtered. Therefore, the function f(Ns-Nd, Ss-Sd, Fs-Fd, As) can determine the weight of the filtered pixel point according to the difference between the normal information of the current pixel point to be filtered and the surrounding pixels. The greater the difference, the smaller the weight, and vice versa. Similarly, the weight size fed back by the shadow information and the inherent color information is deduced by analogy. Furthermore, the greater the noise level A of the pixel to be filtered currently is, the greater the weight of the pixel and the surrounding pixels is, and vice versa.

[0079] It should be noted that the noise level information needs to be continuously updated during the filtering process of the ATrous filter with a radius ranging from 1 to 5, because the noise level information may not be accurate at the beginning, and the noise level information will become more and more accurate during the continuous updating process.

[0080] Furthermore, after completing the denoising operation for each layer of information, the denoised multiple layer information are synthesized to finally obtain a pre-denoised rendered image. The synthesis method is to average the weight of each pixel in each denoised layer information, so that each frame image in the rendered video stream can achieve high-quality denoising effect. At the same time, the denoising method of the filter is used to improve the denoising efficiency, and the denoising time can be achieved within 10ms.

[0081] S203: acquiring a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image;

[0082] In the embodiment of the application, the target rendered image of the current frame and the information of the two adjacent rendered frames are used to realize the information reuse of the rendered images between frames, so as to ensure that the previous and next frame images of the target rendered image are rendered. Figure 7 As shown, Figure 7Schematic diagram of inter-frame rendered images in a rendering stream in the implementation of the present application. From the inter-frame sequence of the pre-denoised rendering stream, the pre-denoised rendered image of the 23rd frame is selected as the adjacent frame rendered image for inter-frame information multiplexing, that is, the adjacent frame rendered image can be a rendered image several frames before or after the currently rendered frame (target rendered image), and a common target object is determined from the target rendered image and the adjacent frame rendered image. The target object can be an object in the rendered image, for example Figure 3 chairs, cups, windows, etc. By determining the specific target objects, the subsequent noise reduction can be more targeted, thereby improving the quality of noise reduction.

[0083] S204: adjusting the color values ​​of the pixels of the target object in the target rendered image according to the position of the target object, so as to obtain a denoised target rendered image.

[0084] Furthermore, information of adjacent frames of rendered images before and after the target rendered image is reused on the target rendered image to implement noise reduction and anti-aliasing operations.

[0085] Since the rendered video stream itself is moving, the center point of the target object of the previous frame of the rendered image is located at screen pixel a, and the center of the target object of the next frame of the rendered image may move to screen pixel b, so the reprojection operation is used to identify the position change of the same target object. Among them, the reprojection uses the object ID information (objectID) of the target object and the world coordinate position information (position) of the target object, through which the current pixel point of the target object can be calculated to be projected to the screen coordinate position of the adjacent frame.

[0086] Furthermore, since the occlusion relationship of multiple target objects may change after the movement, the previously saved world coordinate position information of the target objects can be used to determine the occlusion relationship of multiple target objects. After obtaining the pixel position of the surrounding (i.e., adjacent frame rendering image) target object, it is necessary to further make an effective judgment on the target object to minimize the errors such as ghosting. The effective judgment is mainly based on the object ID information and color threshold of the target object to determine whether it is a valid target object. If the color threshold of the pixel of the surrounding (i.e., adjacent frame rendering image) target object does not meet the color threshold of the target object, it is judged as an invalid color value, that is, the weight of the pixel is not assigned to the pixel at the same position as the pixel in the target rendering image; if it is a valid color value, it is necessary to evaluate the weight of the pixel and the layer information of the target rendering image. The weight evaluation is mainly based on the normal information and shadow information. Finally, all the reusable weights around the target rendering image are weighted averaged, and the average weight value obtained, that is, the average weight value reflects the target color value. The color value of the pixel of the target object in the target rendering image is adjusted by the target color value, which is the color value of the target rendering image after denoising.

[0087] In some implementations, adjusting the color value of a pixel of the target object in the target rendered image according to the position of the target object to obtain a denoised target rendered image includes:

[0088] Respectively obtaining a first position of the target object in the adjacent frame rendering image and a second position of the target object in the target rendering image;

[0089] When the target object is determined to be in a non-blocked state according to the first position and the second position, weighted averaging is performed on the weights of the pixel points to which the target object belongs;

[0090] The weighted average processing result is used to adjust the color value of the pixel of the target object to obtain a denoised target rendered image.

[0091] The first position of the target object in the rendered image of the adjacent frame and the second position of the target object in the rendered image of the target are determined by depth information (i.e., the distance from the world coordinate to the camera). If the target object exists in the first position and the second position and is not blocked by other objects in the rendered image, the weight of the pixel points to which the target object belongs is evaluated and then weighted average processing is performed. The weight obtained by the weighted average processing, i.e., the target color value mapped by the weight at this time, is used to adjust the color value of the pixel points of the target object. If the target object exists in the first position and / or the second position and is blocked by other objects in the rendered image, it means that the target object is blocked. At this time, the color value (weight) of the target object will be discarded and will not be included in the weighted average calculation.

[0092] Exemplarily, the position of the pixel point belonging to the target object can be determined by projecting a pixel P of the sth frame onto the image plane of the dth frame. Then, the object O to which the pixel P of the sth frame belongs can be obtained through the object ID information layer. Through O, the position and direction of object O in the sth frame (saved in advance) and the position and direction of O in the dth frame can be known. The camera projection algorithm is used to obtain the pixel point corresponding to the projection of the pixel P of the sth frame onto the image plane of the dth frame.

[0093] Further, the weight of the pixel points belonging to the target object is evaluated based on the normal information N, the shadow information S, the object ID, and the frame difference F. For example, the weight of the pixel points belonging to the target object is Weight=f(Ns-Nd, Ss-Sd, IDs, IDd, F), where f is a four-element function, Ns represents the normal information of the pixel points of the current frame, Nd represents the normal information of the pixel points of the surrounding frames projected to the current frame, Ns-Nd represents the difference in the normal information of the two, Ss-Sd represents the shadow information of the pixel points of the surrounding frames projected to the current frame, IDs and IDd are used to determine whether the mapped pixel points of the surrounding frames and the pixel points of the target frame are on the same object. If not, the weight is 0, and F represents the frame difference, that is, the current frame number minus the frame number where the projected pixel is located. For example, if the current frame is the 5th frame and the frame number where the projected pixel is located is the 9th frame, then F=4, and the larger the frame difference F, the smaller the weight.

[0094] Furthermore, after obtaining the weights of the pixel points of the target object determined by the rendered images of several adjacent frames, the weights are weighted averaged to finally obtain the denoised target rendered image, and the image rendering quality is improved by multiplexing the inter-frame information. For example, the pixel points of the target object of the current frame are the 5th frame in the rendered video stream, and the pixel points of the same target object of the 3rd, 4th and 6th frames are respectively taken. After weight evaluation, the weight values ​​corresponding to the 3rd, 4th and 6th frames are obtained as 2, 3, and 4, respectively. At this time, the weight of the pixel points of the target object of the current frame is (2+3+4) / 3=3.

[0095] By acquiring a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, each frame of the rendered image includes multiple layer information, performing denoising and synthesis processing on each layer information in each frame of the rendered image to obtain a pre-denoised rendered image, obtaining a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image, and adjusting the color value of the pixel point of the target object in the target rendered image according to the position of the target object to obtain the denoised target rendered image. On the one hand, by denoising the layer information of each frame of the rendered image, it is beneficial to reduce the denoising error occurring in the centralized denoising process of the multi-frame rendered images. On the other hand, the position and pixel points of the target rendered image and the adjacent frame rendered image are utilized to realize the reuse of inter-frame information, that is, the layer information is greatly utilized, while ensuring the rendering quality, the time overhead of the algorithm is also reduced, and the denoising quality and denoising efficiency of the rendered image are improved.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0097] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0098] Further references Figure 8 , as a response to the above Figure 2 The present application provides a schematic diagram of an embodiment of an image noise reduction device, and the embodiment of the device is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0099] like Figure 8As shown, the image noise reduction device described in this embodiment includes: an acquisition module 81, a noise reduction synthesis module 82, a determination module 83 and an adjustment module 84. Among them:

[0100] An acquisition module 81 is used to acquire a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, and each frame of the rendered image includes multiple layers of information;

[0101] A noise reduction synthesis module 82 is used to perform noise reduction synthesis processing on each layer information in each frame of the rendered image to obtain a pre-noise-reduced rendered image;

[0102] A determination module 83 is used to obtain a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image;

[0103] The adjustment module 84 is used to adjust the color values ​​of the pixels of the target object in the target rendered image according to the position of the target object, so as to obtain a denoised target rendered image.

[0104] In some embodiments, the noise reduction synthesis module 82 includes:

[0105] A decoupling unit, used for performing layer decoupling on each frame of rendered image to obtain multiple layer information of each frame of rendered image;

[0106] A denoising unit, used for performing denoising processing on each layer information to obtain multiple layers of denoised information in each frame of rendered image;

[0107] The synthesis unit is used to synthesize the denoised multiple layer information to obtain a pre-denoised rendered image.

[0108] In some embodiments, the noise reduction unit includes:

[0109] Get sub-units, used to obtain pixel points of each layer information;

[0110] A filtering subunit, configured to filter each pixel in the layer information to obtain a plurality of filtering layer information corresponding to each layer information, wherein each filtering layer information includes filtering pixel information;

[0111] The denoising subunit is used to perform mean processing on the filtered pixel information in each filtered layer information to obtain the denoised layer information.

[0112] In some embodiments, the filtering subunit is further configured to:

[0113] Get multiple filters;

[0114] Each pixel point in each layer information is input into a plurality of filters respectively, so as to output a plurality of filtered layer information corresponding to each layer information.

[0115] In some embodiments, the filtering subunit is further configured to:

[0116] According to the position of the same filtering pixel point, the weights of multiple filtering pixels are obtained;

[0117] The weights of the plurality of filtered pixel points are averaged, and the averaged weights are used as noise reduction weights of the filtered pixel points, and the weight values ​​of the layer information after noise reduction are constructed through the noise reduction weights.

[0118] In some embodiments, the adjustment module 84 includes:

[0119] A position acquisition unit, used to respectively acquire a first position of the target object in the adjacent frame rendering image and a second position of the target object in the target rendering image;

[0120] A weighted averaging unit, configured to perform weighted averaging processing on the weights of the pixel points to which the target object belongs when the target object is determined to be in a non-occluded state according to the first position and the second position;

[0121] The adjustment unit is used to adjust the color value of the pixel point of the target object using the obtained weighted average processing result to obtain a target rendered image after noise reduction.

[0122] In some embodiments, the decoupling unit is further configured to:

[0123] Get multiple layers of information of the rendered image according to the preset renderer.

[0124] Regarding the image noise reduction device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0125] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0126] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 with components 91-93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0127] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0128] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as the program code of the image noise reduction method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0129] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of the image noise reduction method.

[0130] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0131] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an image denoising program, and the image denoising program can be executed by at least one processor so that the at least one processor performs the steps of the image denoising method as described above.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0133] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An image denoising method, characterized in that: include: Acquire a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, and each frame of rendered image includes multiple layers of information; Performing denoising and synthesis processing on each layer information in each frame of rendered image to obtain a pre-denoised rendered image; Acquire a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image; According to the position of the target object, the color value of the pixel of the target object in the target rendered image is adjusted to obtain a denoised target rendered image; The step of performing noise reduction and synthesis processing on each layer information in each frame of the rendered image to obtain a pre-noised rendered image includes: Decouple each frame of rendered image by layers to obtain multiple layer information of each frame of rendered image; Performing noise reduction processing on each layer information to obtain multiple layer information after noise reduction in each frame of rendered image; Performing synthesis processing on the plurality of layer information after denoising to obtain a pre-denoised rendered image; The performing denoising on each layer information to obtain multiple denoised layer information in each frame of rendered image includes: Get the pixel points of each layer information; Perform filtering processing on each pixel point in the layer information to obtain a plurality of filtered layer information corresponding to each layer information, wherein each filtered layer information includes filtered pixel information; The filtered pixel information in each filtered layer information is averaged to obtain the denoised layer information.

2. The image denoising method according to claim 1, characterized in that: The filtering process for each pixel in the layer information to obtain a plurality of filtered layer information corresponding to each layer information includes: Get multiple filters; Each pixel point in each layer information is input into a plurality of filters respectively, so as to output a plurality of filtered layer information corresponding to each layer information.

3. The image denoising method according to claim 1, characterized in that: The filtering pixel information includes the weight of the filtering pixel points, and the performing mean processing on the filtering pixel information in each filtering layer information to obtain the layer information after noise reduction includes: According to the position of the same filtering pixel point, the weights of multiple filtering pixels are obtained; The weights of the plurality of filtered pixel points are averaged, and the averaged weights are used as noise reduction weights of the filtered pixel points, and the weight values ​​of the layer information after noise reduction are constructed through the noise reduction weights.

4. The image denoising method according to claim 1, characterized in that: The step of adjusting the color values ​​of the pixels of the target object in the target rendered image according to the position of the target object to obtain the target rendered image after noise reduction includes: Respectively obtaining a first position of the target object in an adjacent frame rendered image and a second position of the target object in a target rendered image; When the target object is determined to be in a non-blocked state according to the first position and the second position, weighted averaging is performed on the weights of the pixel points to which the target object belongs; The obtained weighted average processing result is used to adjust the color value of the pixel point of the target object to obtain a denoised target rendered image.

5. The image denoising method according to claim 1, characterized in that: The layer decoupling of each frame of rendered image to obtain multiple layer information of each frame of rendered image includes: Get multiple layer information of the rendered image according to the preset renderer.

6. An image noise reduction device, characterized in that: include: An acquisition module, used for acquiring a rendered video stream, wherein the rendered video stream includes multiple frames of rendered images, and each frame of rendered image includes multiple layers of information; A denoising and synthesis module is used to perform denoising and synthesis processing on each layer information in each frame of rendered image to obtain a pre-denoised rendered image, including: performing layer decoupling on each frame of rendered image to obtain multiple layer information of each frame of rendered image; performing denoising processing on each layer information to obtain multiple layer information after denoising in each frame of rendered image; performing synthesis processing on the multiple layer information after denoising to obtain a pre-denoised rendered image; The denoising process is performed on each layer information to obtain multiple denoised layer information in each frame of rendered image, including: obtaining pixel points of each layer information; filtering each pixel point in the layer information to obtain multiple filtered layer information corresponding to each layer information, wherein each filtered layer information includes filtered pixel information; averaging the filtered pixel information in each filtered layer information to obtain the denoised layer information; A determination module, configured to obtain a target rendered image, an adjacent frame rendered image of the target rendered image, and a target object existing in the target rendered image and the adjacent frame rendered image from the pre-denoised rendered image; The adjustment module is used to adjust the color value of the pixel point of the target object in the target rendered image according to the position of the target object, so as to obtain the target rendered image after noise reduction.

7. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image denoising method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image noise reduction method according to any one of claims 1 to 5 are implemented.