Image noise reduction method and device, computing equipment and computer readable storage medium

Abstract

The embodiment of the invention provides an image noise reduction method and device, computing equipment and a computer readable storage medium. The image noise reduction method comprises the steps of obtaining a current image frame of a to-be-processed video and a noise reduction result graph of a previous image frame; filtering the current image frame to obtain a filtering result graph of the current image frame; and according to the current image frame, the filtering result graph of the current image frame and the noise reduction result graph of the previous image frame, obtaining image coding characteristics, and decoding the image coding characteristics to obtain the noise reduction result graph of the current image frame. The noise reduction result of the previous image frame is learned when noise reduction processing is carried out on the current image frame, time sequence modeling is realized by adopting a recursion mechanism, the noise reduction result of the image frame is optimized by combining filtering processing and time sequence modeling, efficient suppression of noise in the image frame is realized, meanwhile, more detail information can be reserved, and the noise reduction result of the image frame can be optimized.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of image processing technology, and more particularly to an image denoising method, apparatus, computing device, and computer-readable storage medium. Background Art

[0002] With the rapid development of computers, network technology and multimedia technology, various images or videos are involved in many aspects of people's work and life. Compared with text, images or videos carry richer and more expressive information, and thus are increasingly attracting the attention and love of users. Various images and videos emerge in an endless stream, and the degree of visualization and videoization is getting higher and higher.

[0003] Noise reduction is a technique for reducing unwanted random noise in video frames. It is an important image quality processing task in video scenes. Noise can come from a variety of sources, such as sensor noise, low light conditions, or artifacts caused by video compression. Noise reduction can significantly improve the visual quality of image frames, enhance coding efficiency, and improve the results of subsequent image quality tasks.

[0004] In the prior art, image processing technology is usually used to reduce the noise of image frames in the video, but there are limitations in the noise reduction range and the preservation of image details. The noise reduction accuracy is low, which affects the noise reduction effect, and the quality of the image frames obtained by noise reduction is poor. Therefore, a more accurate and higher quality image noise reduction solution is urgently needed. Summary of the invention

[0005] In view of this, an embodiment of the present specification provides an image noise reduction method. One or more embodiments of the present specification also relate to an image noise reduction device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, there is provided an image noise reduction method, comprising: Get the current image frame of the video to be processed and the noise reduction result image of the previous image frame; Performing filtering processing on the current image frame to obtain a filtering result image of the current image frame; An image coding feature is obtained according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and the image coding feature is decoded to obtain the noise reduction result map of the current image frame.

[0007] According to a second aspect of the embodiments of this specification, there is provided an image noise reduction device, including: An acquisition module is configured to acquire a current image frame of a video to be processed and a noise reduction result image of a previous image frame; A determination module is configured to perform filtering processing on the current image frame to obtain a filtering result image of the current image frame; The denoising processing module is configured to obtain image coding features according to the current image frame, the filtering result map of the current image frame and the denoising result map of the previous image frame, and decode the image coding features to obtain the denoising result map of the current image frame.

[0008] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned image denoising method are implemented.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned image denoising method are implemented.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned image denoising method when executed by a processor.

[0011] The embodiment of the present specification provides an image denoising method, which implements the introduction of filtering processing before the feature encoding of the current image frame to implement the denoising processing, determines the filtering result map of the current image frame, and then obtains the image coding feature according to the current image frame, the filtering result map of the current image frame and the denoising result map of the previous image frame, and decodes the image coding feature to obtain the denoising result map of the current image frame. In this way, filtering processing is first performed before the image encoding of the current image frame, and the filtering capability and the image feature encoding capability are combined to achieve efficient suppression of the noise in the image frame, while more detail information can be retained; and the denoising result output by the previous image frame can be used as the input of the denoising processing of the current image frame, and the denoising result of the previous image frame can be learned when the denoising processing of the current image frame is performed, and a recursive mechanism is used to implement time series modeling, and the denoising result of the image frame is optimized by combining filtering processing and time series modeling, so as to improve the accuracy of the denoising processing, obtain better denoising image quality, ensure the denoising effect, and improve the quality of the image frame obtained by denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of an image noise reduction method provided by an embodiment of this specification; Figure 2 is a structural schematic diagram of a noise reduction model provided by an embodiment of this specification; Figure 3 is an algorithm diagram of an image noise reduction method provided by an embodiment of this specification; Figure 4 is a schematic diagram of a network structure of an image noise reduction method provided by an embodiment of this specification; Figure 5 is a processing flow chart of an image noise reduction method provided by an embodiment of this specification; Figure 6 This is a schematic diagram of an algorithm implementation of an image noise reduction method provided by an embodiment of this specification; Figure 7 is a structural schematic diagram of an image noise reduction device provided by an embodiment of this specification; Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0013] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0014] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0016] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0017] First, the terms involved in one or more embodiments of this specification are explained.

[0018] Infinite Impulse Response (IIR) filter: is a type of filter commonly used in digital signal processing. The design of IIR filters is based on a feedback structure, which means that the output depends not only on the current and past input samples, but also on past output samples. Unlike finite impulse response (FIR) filters, IIR filters are characterized by their unit impulse response that can theoretically continue infinitely. This feature enables IIR filters to achieve higher selectivity while using fewer coefficients. Since IIR filters can use relatively few coefficients to achieve good frequency response, they usually save more computing resources than FIR filters of the same performance.

[0019] Deep learning: It is a subfield of machine learning. It imitates the working method of the human brain and uses a multi-layer neural network model to learn high-level abstract representations of data. In recent years, this technology has achieved remarkable success in many fields such as image processing, speech processing, natural language processing, and recommendation systems, and has become an important driving force for the development of artificial intelligence.

[0020] Recurrent Neural Network (RNN): is a neural network model used to process sequence data. Unlike feedforward neural networks, RNN has internal memory capabilities and can capture and utilize temporal dynamic information in sequences. This feature makes RNN very suitable for processing text, voice, video, time series and other data with time order.

[0021] It should be noted that noise reduction is a technology that reduces unwanted random noise in image frames in videos. It is an important image quality processing task in video scenes. Noise can come from a variety of sources, such as sensor noise, low light conditions, or artifacts caused by video compression. Noise reduction can significantly improve the visual quality of image frames in videos, enhance video coding efficiency, and improve the effects of subsequent visual image quality tasks (such as algorithms such as low-quality video enhancement).

[0022] In one possible implementation, noise reduction usually relies on image processing techniques, such as time domain and spatial domain filters. Although these methods can reduce noise to a certain extent, they have limitations in terms of noise reduction and maintaining video details. IIR filter is a commonly used digital filter, widely used in signal processing and control systems. The noise reduction algorithm based on IIR filter smoothes the noise between image frames in the video by applying recursive filtering in the time domain. The specific method is to apply IIR filter to each frame of the video, and use the information of the current frame and several previous frames to reduce noise. The basic idea is to enhance the smoothness of the signal through the weighted average method.

[0023] IIR filters are more effective in processing specific types of noise, such as Gaussian noise, and can largely retain image texture details while reducing noise. However, they are less effective for other types of noise, such as color noise, compression noise, and other nonlinear noises. At the same time, they respond slowly to fast-changing signals in the video, such as the edges of moving objects, and may introduce a smoothing effect, resulting in motion ghosting, which affects the quality of image noise reduction.

[0024] In another possible implementation, deep learning technology, especially RNN (Recurrent Neural Network), shows great potential in the field of noise reduction. RNN can process sequence data and capture temporal correlation through its memory mechanism, which is particularly important for the noise reduction of image frames in videos, because video is a kind of time series data, and there is temporal continuity and correlation between its frames. RNN can effectively capture the temporal dependency between the image frames of the video and use the temporal context information to more accurately distinguish between noise and useful signals. At the same time, RNN can retain information about long-term dependencies through its hidden states, which is of great significance for processing noise in long video sequences.

[0025] Although RNN can effectively utilize the inter-frame information of previous and next image frames in the video, accurately distinguish noise from useful information and thus achieve noise reduction, it is prone to detail loss caused by inaccurate motion estimation when processing image frames with fast or complex motion. In particular, in areas with rich high-frequency details, insufficient detail retention can cause image blur, resulting in a decrease in the quality of the image frame after noise reduction.

[0026] Therefore, in order to solve the above technical problems, an image denoising solution is provided in the embodiments of this specification, which introduces an IIR filter as a pre-processing step before the RNN, and utilizes the real-time recursive denoising capability of the IIR filter and the temporal correlation modeling capability of the RNN to achieve efficient suppression of video noise while retaining more detail information. In this way, combining the IIR filter with the RNN recursive neural network for cascade image denoising processing can take into account the advantages of both RNN and IIR filters, and is superior to the traditional method of using RNN or IIR filters alone in terms of noise suppression and detail retention. While reducing noise, it can greatly retain image details while eliminating motion smearing problems, and obtain better denoised image quality.

[0027] In this specification, an image denoising method is provided. This specification also relates to an image denoising device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0028] See also Figure 1 , Figure 1 A flowchart of an image noise reduction method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0029] Step 102: Obtain the current image frame of the video to be processed and the noise reduction result image of the previous image frame.

[0030] It should be noted that the image denoising method provided in the embodiments of this specification can be applied to the client or to the cloud server. If applied to the client, the video to be processed can be obtained through the cloud server or other devices to perform denoising on each image frame in the video to be processed on the client; if applied to the cloud server, the video to be processed can be obtained locally from the client, other servers or the cloud server to perform denoising on each image frame in the video to be processed on the cloud server. Among them, the cloud server can be connected to one or more clients through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client here can include but is not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client can interact with the user through a graphical user interface to implement operations such as uploading, managing, and denoising the video to be processed, thereby implementing the image denoising method provided in the embodiments of this specification.

[0031] Specifically, a video is essentially a series of still image frames displayed in rapid succession. The number of frames displayed per second is usually called the frame rate (e.g., 24fps, 30fps, or 60fps, etc.). These image frames work together to produce a visual effect of motion. The video to be processed is a video that needs to be denoised. The video to be processed includes multiple image frames. When performing denoising on the video to be processed, the image frames can be denoised frame by frame. In video processing, the current image frame and the previous image frame refer to two consecutive image frames in the video to be processed. The current image frame refers to a frame of image being processed or analyzed, representing a time point in the sequence of the video to be processed; the previous image frame refers to a frame of image immediately before the current image frame, providing temporal context information about how the content of the picture changes.

[0032] In actual implementation, after performing noise reduction processing on any image frame in the video to be processed to obtain a noise reduction result map, the noise reduction result map of the image frame can be stored. Specifically, when performing noise reduction processing on the current image frame in the video to be processed, the current image frame, that is, the original image frame being processed or analyzed, can be obtained; and the noise reduction result map of the image frame before the current image frame can be read from the cache.

[0033] If the current image frame is the first image frame in the video to be processed, and there is no previous image frame, the original image of the first image frame can be directly used as the denoising result image after denoising processing, that is, for the first image frame of the video to be processed, the output image of the denoising processing is equal to the input image.

[0034] It should be noted that, for the current image frame of the video to be processed, the current image frame is any image frame after the second one (including the second image frame) in the video to be processed. Based on the noise reduction result map of the current image frame and the previous image frame, the current image frame can be jointly subjected to noise reduction processing. The noise reduction result output by the previous image frame is used as the input of the noise reduction processing of the current image frame. A recursive mechanism is adopted to optimize the noise reduction result of the image frame to obtain better noise reduction image quality.

[0035] Step 104: Filter the current image frame to obtain a filtering result image of the current image frame.

[0036] In actual implementation, a filter can be used to determine the filtering result map of the current image frame. The filter can be an IIR filter, so that the recursive noise reduction capability of the IIR filter can be used for smoothing. Of course, other filtering algorithms, filtering models or tools can also be used to filter the current image frame to obtain the filtering result map of the current image frame, such as FIR (finite impulse response) filter, median filter, mean filter, Gaussian filter, bilateral filter, etc., and this specification embodiment does not limit this.

[0037] In an optional implementation of this embodiment, filtering is performed on the current image frame to obtain a filtering result map of the current image frame, including: Obtain a filtering result image of a previous image frame of the video to be processed; The filtering result graphs of the current image frame and the previous image frame are filtered by the filter to determine the filtering result graph of the current image frame.

[0038] Among them, in the noise reduction processing, the image frame can be filtered. The filtering result map of the previous image frame and the filtering result map of the current image frame refer to the output image after applying a certain filter to two consecutive frames in the processed video. The filtering result map of the previous image frame can be obtained in real time using the filter processing, or it can be read from the cache.

[0039] In actual implementation, for any image frame in the video to be processed, a filter is used to filter it, and after obtaining the filtering result map of the image frame, the filtering result map of the image frame can be stored. Specifically, when the current image frame in the video to be processed is subjected to noise reduction processing, the filtering result map of the previous image frame of the current image frame in the video to be processed can be read from the cache.

[0040] It should be noted that the filtering result images of the current image frame and the previous image frame can be input into the filter for filtering processing to obtain the filtering result image of the current image frame, and the filtering result image of the previous image frame can be smoothed, wherein the filter can be an IIR filter.

[0041] In the embodiment of the present specification, the filtering result graphs of the current image frame and the previous image frame are used for smoothing, so as to obtain better smoothing and initial noise reduction effects.

[0042] Of course, in actual implementation, when determining the filtering result map of the current image frame, in addition to the above-mentioned implementation method of combining the filtering result maps of the current image frame and the previous image frame, the current image frame can also be directly input into the filter to obtain the filtering result map of the current image frame. The embodiments of this specification do not limit this.

[0043] In an optional implementation of this embodiment, filtering the filtering result images of the current image frame and the previous image frame through a filter to determine the filtering result image of the current image frame includes: determining filter coefficients of a filter configuration; By using a filter, a first pixel value of a target pixel in a current image frame, a second pixel value of the target pixel in a filter result image of a previous image frame, and a filter coefficient are calculated to determine a target pixel value of the target pixel, wherein the target pixel is a pixel at any position in the current image frame; According to the target pixel value of each pixel in the current image frame, the filtering result map of the current image frame is output.

[0044] Among them, the filter coefficient refers to a key parameter in the filter design, which is used to indicate how the filter processes the input signal. In the embodiment of this specification, the filter coefficient is configured to represent the influence weight of the original pixel value in the current image frame on the filter result, and the value range of the filter coefficient is [0,1]. That is, the filter coefficient can indicate the influence degree of the pixel value in the current image frame on the filter result map of the current image frame, or the influence degree of the pixel value in the filter result map of the previous image frame on the filter result map of the current image frame.

[0045] In specific implementation, the filter coefficient can be configured as a fixed value, and each image frame of the video to be processed and each pixel in the image frame uses the same filter coefficient; alternatively, different filter coefficients can be calculated based on different image frames, and each pixel in the image frame uses the filter coefficient corresponding to the image frame; alternatively, the corresponding filter coefficients can be calculated based on different pixels in the image frame, and the corresponding filter coefficients can be configured pixel by pixel.

[0046] In actual implementation, the filter result images of the current image frame and the previous image frame can be input into the filter, and the filter performs pixel-by-pixel weighted calculation on the two input images to obtain and output the filter result image of the current image frame. Specifically, the filter can determine the target pixel value of the target pixel based on the first pixel value of the target pixel in the current image frame, the second pixel value of the target pixel in the filter result image of the previous image frame, and the filter coefficient. The pixel at any position in the current image frame can be used as the target pixel to obtain the corresponding target pixel value. According to the target pixel value of each pixel in the current image frame, the filter result image of the current image frame can be obtained.

[0047] Specifically, the first pixel value of the target pixel in the current image frame may be multiplied by the filter coefficient to obtain a first intermediate result, and the second pixel value of the target pixel in the filter result map of the previous image frame may be multiplied by (1-filter coefficient) to obtain a second intermediate result. The first intermediate result and the second intermediate result are summed to obtain the target pixel value of the target pixel. Specifically, the filter result map of the current image frame may be obtained by the following formula (1): (1) in, Indicates the first frame in the t-th image frame The pixel value of the position pixel output by the filter, that is, the pixel value of the position pixel in the t-th frame image frame The target pixel value of the position pixel; Indicates the first frame in the t-th image frame The input pixel value of the position pixel in the filter, that is, the first pixel in the t-th frame image frame A first pixel value of a position pixel; Indicates the first frame in the t-1th frame The pixel at position t is the output pixel value after the filter, that is, the filtering result image of the t-1th frame. The second pixel value of the pixel at that location. is the filter coefficient, and its value range is [0, 1].

[0048] In the embodiments of the present specification, the filtering result images of the current image frame and the previous image frame are used to perform pixel weighting at each position, and there is no need to align the image spatial position, thereby preventing pixel misalignment caused by image alignment in certain static background noise areas of the image, and ensuring better smoothing and initial noise reduction effects.

[0049] In an optional implementation of this embodiment, determining a filter coefficient of a filter configuration includes: Determine a pixel difference value of the target pixel according to a first pixel value of the target pixel in the current image frame and a second pixel value of the target pixel in the filtering result image of the previous image frame; According to the pixel difference and the set difference threshold, the filter coefficient corresponding to the target pixel in the filter is determined.

[0050] The difference threshold is a pre-configured value used to determine the corresponding filter coefficient, and the difference threshold can be pre-configured based on business needs, such as 5. The configuration rule can be that the larger the difference, the closer the filter result image of the current image frame needs to be to the original image, that is, the larger the difference, the larger the filter coefficient corresponding to the target pixel in the current image frame can be configured.

[0051] In actual implementation, the pixel difference between the first pixel value of the target pixel in the current image frame and the second pixel value of the target pixel in the filtering result image of the previous image frame can be determined, and then the relationship between the pixel difference and the set difference threshold can be determined to determine the filtering coefficient corresponding to the target pixel to calculate the target pixel value of the target pixel.

[0052] In one implementation, a set difference threshold can be pre-configured. If the pixel difference is less than or equal to the set difference threshold, the filter coefficient corresponding to the target pixel is set to 0; if the pixel difference is greater than the set difference threshold, the filter coefficient corresponding to the target pixel is set to 1.

[0053] For example, the filter coefficient corresponding to the target pixel can be calculated by the following formula (2) and formula (3): (2) (3) in, Represents the filter coefficient corresponding to the pixel at position i, j) in the t-th image frame; Represents the pixel difference of the pixel at position (i, j) in the t-th image frame; Indicates the first frame in the t-th image frame The input pixel value of the position pixel in the filter, that is, the first pixel in the t-th frame image frame A first pixel value of a position pixel; Indicates the first frame in the t-1th frame The pixel at position t is the output pixel value after the filter, that is, the filtering result image of the t-1th frame. The second pixel value of the pixel at that location.

[0054] In another implementation, multiple difference thresholds may be set, and different difference thresholds may form different difference ranges, and different difference ranges may correspond to filter coefficients of different sizes. The target difference threshold corresponding to the pixel difference is determined, and the filter coefficient corresponding to the target difference threshold is used as the filter coefficient of the target pixel. Specifically, the filter coefficient corresponding to the target pixel may be calculated by the following formula (4): (4) in, The pixel difference is located at Range (greater than or equal to 0 and less than ) corresponding filter coefficient; The pixel difference is located at Range (greater than or equal to and less than )Corresponding filter coefficients;……; The pixel difference is located at Range (greater than or equal to ) corresponding to the filter coefficient. Among them, Less than Less than...less than , corresponding to Less than Less than...less than , that is, the larger the pixel difference, the closer the filter coefficient is to 1, so that the pixel value in the current image frame has a greater impact on the filter result image of the current image frame, and the filter result image of the current image frame is closer to the original image.

[0055] For example, assuming that the pre-configured difference thresholds are 5 and 10, and the pixel difference is greater than or equal to 0 and less than 5, the corresponding filter coefficient is 0; the pixel difference is greater than or equal to 5 and less than 8, the corresponding filter coefficient is 0.4; the pixel difference is greater than or equal to 8 and less than 10, the corresponding filter coefficient is 0.6; the pixel difference is greater than or equal to 10, the corresponding filter coefficient is 1.

[0056] In the embodiments of the present specification, due to the use of a weighted smoothing method for each pixel position, more serious motion ghosting may occur in the moving areas of the previous and next image frames. Therefore, the filter can determine the difference image of the filtering result image of the current image frame and the previous image frame, and determine the corresponding filter coefficient pixel by pixel. This can effectively reduce the motion ghosting and avoid introducing excessive motion ghosting in the subsequent noise reduction model part, which affects the noise reduction effect, thereby obtaining better noise reduction image quality.

[0057] It should be noted that the pixel difference between the filtering result images of the current image frame and the previous image frame can be obtained, and the pixel-by-pixel filtering coefficient can be obtained based on the pixel difference, and the filtering coefficient can be used to calculate the filtering result image of the current image frame; or, the filtering coefficients of other adjacent pixels around the target pixel can be combined to determine the uniform filtering pixel of the target pixel, and then the filtering result image of the current image frame can be calculated.

[0058] In an optional implementation of this embodiment, determining a filter coefficient of a filter configuration includes: Determine the adjacent pixels around the target pixel according to the set coordinate range; The filter coefficient corresponding to the target pixel in the filter is determined according to the filter coefficients of the adjacent pixels, wherein the filter coefficients of the adjacent pixels are determined based on the pixel difference between the filter result images of the current image frame and the previous image frame.

[0059] In actual implementation, the adjacent pixels around the target pixel refer to the adjacent pixels within the set coordinate range. The filter coefficients of the adjacent pixels can be determined by the pixel difference between the filter result images of the current image frame and the previous image frame, and then the filter coefficients of the adjacent pixels are summed to determine the filter coefficient corresponding to the target pixel.

[0060] Specifically, the filter coefficient of the target pixel can be calculated by the following formula (5): (5) in, is a preconfigured coordinate range for the target pixel , superimposed successively from - to The filter coefficient of the pixel corresponding to the coordinates is used to obtain the target pixel The filter coefficient . , from- to The values ​​are integers.

[0061] For example, assuming If it is 2, then determine , , , , , , , , , , , , , , , , , , , , , , , , . Sum the filter coefficients of the pixels at the above positions in turn, and then add Multiplying together, we can get the filter coefficient of the target pixel. .

[0062] The filter coefficient of the pixel at any of the above positions can be obtained by calculating the pixel difference using the above formula (2), formula (3) and formula (4).

[0063] In the embodiments of the present specification, uniform filtering may be performed on the filter coefficients of the adjacent pixels around the target pixel to obtain the filter coefficient of the target pixel, thereby ensuring better smoothing and initial noise reduction effects.

[0064] Step 106: Obtain image coding features according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and decode the image coding features to obtain the noise reduction result map of the current image frame.

[0065] It should be noted that image encoding can be performed based on the current image frame, the filtering result map of the current image frame, and the noise reduction result map of the previous image frame to obtain image encoding features, and then the image encoding features are decoded to obtain the noise reduction result map of the current image frame. By using the noise reduction results before the current image frame for noise reduction, the temporal features can be learned to improve the image noise reduction effect. Specifically, the original image of the current frame, the result of the current frame after preliminary filtering, and the result of the previous frame that has completed the noise reduction process are combined to jointly extract image encoding features containing different levels of detail and time continuity (temporal features). By decoding the correlation and change pattern between the image encoding features, more accurate and detailed noise characteristics can be learned, and a higher quality noise reduction result map of the current frame can be generated accordingly.

[0066] In actual implementation, a denoising model can be used to obtain image coding features according to the current image frame, the filtering result map of the current image frame, and the denoising result map of the previous image frame, and the image coding features are decoded to obtain the denoising result map of the current image frame. The denoising model can be a recursive learning model with temporal feature denoising capability, that is, the denoising model has the ability to model temporal correlation, and can effectively utilize the inter-frame information of the previous and next image frames in the video to be processed, accurately distinguish noise and useful information, and thus achieve denoising. The denoising model can be an architecture that can capture spatiotemporal features, such as 3D CNNs or recurrent neural networks (RNN). In the embodiments of this specification, the denoising model is an RNN network structure as an example. Among them, 3D CNNs refers to a three-dimensional convolutional neural network, which is a deep learning model specially designed to process three-dimensional data. It is similar in structure to a two-dimensional convolutional neural network (2D CNNs), but can operate on the spatial dimension and time dimension of the data at the same time, which means that 3DCNNs are suitable for processing time-series image data such as video, because they can capture information between frames and spatial information within a single frame.

[0067] In actual implementation, the denoising model adopts the RNN network structure as an example. The RNN network structure uses a recursive neural network based on U-Net (RNN-U-Net), where U-Net is a classic convolutional neural network architecture. The characteristic of U-Net is that it has an encoder-decoder structure, and in the decoder part, the feature map in the encoder is connected to the corresponding layer in the decoder through skip connections, thereby helping to retain more spatial information. RNN-U-Net is a deep learning model that combines recursive neural networks (RNN) and U-Net architectures. It is mainly used to process image processing tasks in time series data. This model is suitable for application scenarios such as video segmentation or continuous image information that need to consider temporal context information. RNN-U-Net combines the temporal modeling ability of RNN with the spatial modeling ability of U-Net to better process time series image data. Specifically, RNN-U-Net usually introduces RNN units (such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit)) in certain layers of U-Net to add information of the time dimension to the network. That is to say, in this structure, each convolutional layer is followed by an activation function (such as ReLU), and LSTM units are added to the key layers to process temporal information. In this way, RNN-U-Net can effectively combine spatial and temporal information to provide more accurate image processing results.

[0068] In the embodiments of this specification, the denoising model may include an encoder and a decoder, wherein the encoder is used to reduce the size of the input image and enrich the number of channels of the input image, that is, the size of the input image is gradually reduced, but the implicit information included is increasingly rich, so that the decoder can restore the current frame image as noise-free as possible based on the rich implicit information. That is, the encoder can encode the input image data and extract key features that help distinguish noise from useful signals, that is, image coding features; then, the decoder can reconstruct the current frame image as noise-free as possible based on the image coding features, which not only considers the spatial relationship within a single image, but also fully exploits the temporal correlation between adjacent frames, so that it can effectively remove noise while maintaining image clarity.

[0069] Specifically, the encoder can use convolution and pooling layers to extract image coding features, and the decoder uses upsampling layers to gradually restore the denoising results, wherein the upsampling layer is used to increase the spatial size of the feature map, which is usually used in tasks such as image segmentation and super-resolution reconstruction. It can be implemented based on transposed convolution (deconvolution or fractional step convolution), or can also be implemented based on nearest neighbor interpolation, attention mechanism in deep learning, etc. Information is transmitted between the encoder and the decoder through jump connections. In addition, in the embodiments of this specification, a recursive neural network structure is added on top of the basic U-Net framework, and the denoising output of the previous image frame is used as the input for denoising processing of the current image frame, and the denoising result is gradually optimized.

[0070] It should be noted that the input of the denoising model can be obtained based on three images, namely the current image frame, the filtering result map of the current image frame and the denoising result map of the previous image frame. The current image frame saves the most original image information, including image noise information and effective information, which is used to guide the denoising model to select effective information; the filtering result map of the current image frame is the initial smoothed image, which has a strong denoising effect in the static background area and can retain more image details. Although there may be ghosting in the moving area, as the input of the denoising model, it can effectively eliminate motion ghosting after being analyzed and processed by the denoising model; the denoising result map of the previous image frame is also used as the input of the denoising model, which can effectively maintain the continuity of the denoising effect between image frames and prevent the problem of fluctuation in the effect between frames.

[0071] In one implementation method, the current image frame, the filtering result map of the current image frame and the denoising result map of the previous image frame can be directly spliced ​​and input into the denoising model, and the denoising model is used to realize the spatial position alignment in time sequence, and then the denoising process is performed to obtain the denoising result map of the current image frame. In another implementation method, the denoising result map of the previous image frame and the current image frame can be spatially aligned first, and then input into the denoising model for denoising process to obtain the denoising result map of the current image frame.

[0072] In an optional implementation of this embodiment, obtaining image coding features according to the current image frame, the filtering result map of the current image frame, and the noise reduction result map of the previous image frame, and decoding the image coding features to obtain the noise reduction result map of the current image frame includes: Performing spatial position alignment on the denoising result map of the previous image frame and the current image frame to obtain an alignment map of the denoising result of the previous image frame; splicing the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame, and inputting the spliced ​​images into an encoder of a denoising model for encoding to obtain the image coding feature; The image coding features are input into the decoder of the denoising model, and the denoising result image of the current image frame is obtained by decoding.

[0073] Among them, spatial position alignment is also called image registration or image alignment, which refers to aligning two or more images in space so that they are consistent in geometric position. It is widely used in medical imaging, remote sensing, computer vision and other fields. The goal of spatial position alignment is to find a transformation that aligns the denoising result of the previous image frame with the current image frame in space.

[0074] In actual implementation, the denoising result map of the previous image frame and the current image frame can be spatially aligned to obtain the denoising result alignment map of the previous image frame, and then the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame can be denoised through the denoising model to obtain the denoising result map of the current image frame.

[0075] It should be noted that spatial alignment of two images (also known as image registration) is a common computer vision task, the purpose of which is to find a transformation that allows one image to be spatially aligned with another image. In a possible implementation, optical flow calculation can be performed first, and then spatial alignment can be performed. In another possible implementation, the denoising result map of the previous image frame and the current image frame can be directly aligned in spatial position. Specifically, the denoising result map of the previous image frame and the current image frame can be read and converted into grayscale images; then, a SIFT (Scale-Invariant Feature Transform) feature detector is created, and the SIFT algorithm is used to detect key points and generate descriptors; then, a FLANN (Fast Library for Approximate Nearest Neighbors) matcher is used to find matching key point pairs, and good matches are screened out through a ratio test; then, the RANSAC (Random Sample Consensus, an iterative algorithm for estimating model parameters from a data set containing a large amount of noise and outliers) algorithm can be used to estimate the perspective transformation matrix, and the warpPerspective function can be used to apply the transformation matrix to the denoising result map of the previous image frame to obtain the aligned denoising result alignment map of the previous image frame. In actual implementation, the feature detector, matching strategy, and transformation model can be adjusted according to the specific application scenario. Among them, the warpPerspective function is a function in the OpenCV library, which is used to perform perspective transformation on an image. Perspective transformation is a two-dimensional to two-dimensional projection transformation, which can simulate the change from one viewpoint to another, such as converting a photo taken from one angle to a photo taken from another angle.

[0076] in addition, Figure 2 It is a structural diagram of a denoising model provided by an embodiment of the present specification. The denoising model adopts an RNN network architecture and may include an encoder and a decoder.

[0077] After obtaining the denoising result alignment map of the previous image frame, the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame can be spliced ​​to obtain a spliced ​​image, and the spliced ​​image is input into the encoder of the denoising model. The image features are obtained through the feature extraction layer in the encoder. The feature extraction layer includes a set number of convolution layers, and each convolution layer is connected to an activation function, such as Figure 2As shown, the value is set to 2. The stitched image is input into the first convolution layer, and a sliding window (such as a 3x3 matrix) is used to perform operations on the input stitched image. The image is then entered into the subsequent ReLU activation function to maintain positive values ​​and eliminate negative values. The output of the activation function is then processed into the second convolution layer and the activation function to output image features.

[0078] The image features output by the feature extraction layer are input to the pooling layer, and the maximum pooling operation is performed to obtain the initial coding features. Then, the initial coding features are further input to the next feature extraction layer in the encoder for the above processing, until the image features output by the last feature extraction layer are used as the image coding features finally output by the encoder.

[0079] In the above image encoding process, the size of the input spliced ​​image gradually decreases, but the number of channels gradually increases. For example, if the input spliced ​​image is 256*256, the image coding feature obtained by encoding is 32*32, but the implicit information (number of channels) included in the image coding feature is becoming more and more abundant, which can be learned by the subsequent decoder.

[0080] The image coding features are input to the decoder, and upsampled through the upsampling layer in the decoder to obtain the restored image features. An optional method is that the upsampling layer can be upsampled based on the transposed convolution operation, and the upsampling result is input to the activation function to obtain the restored image features, that is, the upsampling layer may include a transposed convolution layer and an activation function connected thereto. The transposed convolution layer is used to restore the size of the input image coding features. The working principle of the transposed convolution layer is similar to that of the standard convolution layer, but its weights are specially arranged so that the output is larger than the input. The transposed convolution layer and the convolution layer of the feature extraction layer in the encoder use the same sliding window size (such as a 3x3 matrix).

[0081] The restored image features output by the upsampling layer can be input to the feature decoding layer to obtain the initial decoding result. The feature decoding layer can also include a set number of convolutional layers, and each convolutional layer is connected to an activation function, such as Figure 2 As shown, the value is set to 2, and the restored image features are input into the first convolution layer. A sliding window (such as a 3x3 matrix) is used to perform operations on the input restored image features, and then the ReLU activation function is used to maintain positive values ​​and eliminate negative values. The output of the activation function is then processed into the second convolution layer and the activation function to output the initial decoding result.

[0082] The initial decoding result output by the feature decoding layer is input to the next upsampling layer for the above processing until the last feature decoding layer, and the initial decoding result output by the last feature decoding layer is input to the dimension mapping layer, which can be a convolution layer, followed by an activation function, and the dimension mapping layer can reduce the number of channels of the input initial decoding result or change the dimension of the feature mapping, and output the denoising result map of the current image frame. The convolution size of the dimension mapping layer is different from the convolution size of the convolution layer in the feature extraction layer and the feature recovery layer, such as the dimension mapping layer is a 1x1 convolution layer, followed by an activation function.

[0083] In addition, each feature extraction layer in the encoder can crop and copy the image features of the layer to the feature recovery layer at the corresponding position through a jump connection, so that the feature recovery layer at the corresponding position can perform shape matching and adapt to the subsequent feature recovery layer.

[0084] The denoising result map of the current image frame output by the dimension mapping layer can be used as the input of the next image frame in the denoising model, and image denoising is achieved through a recursive mechanism. That is, the next image frame of the target video is obtained, the next image frame is used as the current image frame, and the above operation is continued until the last image frame.

[0085] In the embodiments of this specification, on the basis of recursive iteration, an inter-frame image alignment strategy is added to spatially align the denoising result image of the previous frame with the original image of the current frame, and use them together as input of the denoising model, further improving the denoising effect of the recursive iteration.

[0086] Of course, it is also possible not to spatially align the denoising result map of the previous image frame and the current image frame, but to directly splice the current image frame, the filtering result map of the current image frame and the denoising result map of the previous image frame and input them into the denoising model to obtain the denoising result map of the current image frame. The structure of the denoising model and the specific data processing process are similar to the above process and will not be repeated here.

[0087] In an optional implementation of this embodiment, spatially aligning the denoising result map of the previous image frame and the current image frame to obtain the denoising result alignment map of the previous image frame includes: Obtain the image frame before the current image frame in the video to be processed; Perform optical flow calculation on the previous image frame and the current image frame to obtain an optical flow coordinate mapping diagram; The denoising result map of the previous image frame and the optical flow coordinate mapping map are spatially aligned to obtain the denoising result alignment map of the previous image frame.

[0088] The previous image frame of the current image frame is the previous original image frame.

[0089] Optical flow refers to the velocity vector field of a pixel moving from one position to another between consecutive image frames. Simply put, optical flow describes the temporal motion of each pixel in the image. Optical flow calculation (OpticalFlow) is a computer vision technology used to estimate the motion of each pixel in an image sequence. Optical flow provides a way to track the motion of objects in videos or continuous images. Optical flow calculation is based on the following basic assumptions: the constant brightness assumption, the brightness of the same object surface is constant in a short time; the small motion assumption, the motion between adjacent frames is small; the spatial consistency assumption, the motion of adjacent pixels is similar.

[0090] The main methods for optical flow calculation can include the Lucas-Kanade method, a local optical flow estimation method, which assumes that all pixels in a small window have the same motion. This method estimates the optical flow by minimizing the brightness change; the Horn-Schunck method, a global optical flow estimation method, which introduces a smoothing term to ensure the spatial consistency of the optical flow field; the Farnebäck method, a multi-scale optical flow estimation method, which combines local and global methods to improve the accuracy and robustness of optical flow estimation through a multi-scale structure.

[0091] In one possible implementation, only the motion vectors of the sparse feature points in the previous image frame and the current image frame can be calculated to obtain the corresponding optical flow coordinate mapping. In another possible implementation, the dense optical flow coordinate mapping of the previous image frame and the current image frame can be calculated, that is, the optical flow coordinate mapping can refer to the dense optical flow coordinate mapping of the corresponding pixel points of the current image frame (the original image of the current frame) relative to the previous image frame (the original image of the previous frame). Dense optical flow refers to calculating the motion vector of each pixel in the image, rather than only calculating the motion vector of the sparse feature points. Dense optical flow provides more comprehensive motion information and is suitable for application scenarios that require a detailed understanding of the motion of all pixels in the image. The dense optical flow coordinate mapping is a visualization tool for displaying the motion of each pixel in the previous image frame and the current image frame. This mapping usually uses color coding to indicate the motion direction and speed of each pixel.

[0092] In actual implementation, any of the aforementioned optical flow calculation methods can be used to calculate the optical flow of the previous image frame and the current image frame to obtain the corresponding optical flow coordinate mapping map. Then, the denoising result map of the previous image frame and the optical flow coordinate mapping map are spatially aligned to obtain the denoising result alignment map of the previous image frame. The specific spatial alignment process can refer to the detailed description of directly aligning the denoising result map of the previous image frame and the current image frame in space, and the embodiments of this specification will not be repeated here.

[0093] In the embodiments of this specification, spatial alignment is performed before entering the denoising model, that is, the denoising result map of the previous image frame and the spatial alignment process of the current image frame are placed in front, which reduces the learning error of the denoising model for the pixel misalignment of the previous and next image frames and provides a denoising effect. In addition, starting from each pixel of the current image frame, its corresponding position in the previous image frame is calculated to achieve spatial alignment, that is, spatial alignment is achieved by using backward mapping, avoiding the common hole problem in forward mapping and ensuring that each pixel has a reasonable value, and is generally easier to implement and process.

[0094] In other words, by calculating the optical flow vectors of the previous and next image frames and the backward mapping method, the denoising result map of the previous image frame is spatially aligned with the current image frame, and they are used together as the input of the denoising model. The denoising result map of the previous image frame and the current image frame are spatially aligned and placed in front, which further improves the denoising effect of the recursive iteration.

[0095] In an optional implementation of this embodiment, the current image frame, the filtering result map of the current image frame, and the denoising result alignment map of the previous image frame are spliced ​​and input into an encoder of a denoising model for encoding to obtain the image coding feature, including: The current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame are spliced ​​along the channel dimension to obtain a multi-channel spliced ​​image; The multi-channel spliced ​​image is input into the encoder of the denoising model to obtain the image encoding features.

[0096] Among them, stitching along the channel dimension refers to stitching multiple different images along the channel dimension (depth direction) to form a new image with multiple channels.

[0097] In actual implementation, an image frame may include multiple channels. For the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame, the channels of each image frame are stitched together in sequence along the channel dimension to obtain a new multi-channel image, that is, a multi-channel stitched image.

[0098] As an example, assuming that the current image frame, the filtering result map of the current image frame, and the denoising result alignment map of the previous image frame are all standard RGB color images, that is, each image has three color channels (red, green, and blue). The current image frame, the filtering result map of the current image frame, and the denoising result alignment map of the previous image frame are spliced ​​together according to the channel dimension. The current image frame, the filtering result map of the current image frame, and the denoising result alignment map of the previous image frame all contribute three channels, and a new 9-channel spliced ​​image can be obtained.

[0099] In actual implementation, image stitching can be achieved through simple matrix operations. For example, in Python's NumPy library, the numpy.concatenate() function can be used; if it is in the deep learning framework of TensorFlow or PyTorch, the tf.concat() and torch.cat() functions can be used to achieve splicing.

[0100] In the embodiments of the present specification, the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame are spliced ​​along the channel dimension to obtain a multi-channel spliced ​​image, which can effectively integrate image frame data from multiple sources and then input them into the encoder of the denoising model for image encoding, so that the denoising model can make full use of the abundant available information to achieve denoising processing. Through multi-channel series connection, the denoising result output by the denoising model can take into account the advantages of multiple algorithms at the same time, achieve efficient noise suppression, retain more detail information, and improve the quality of image denoising.

[0101] In the embodiment of this specification, the denoising model can perform denoising on the input multi-channel stitched image to obtain the denoising result image of the current image frame. Specifically, the denoising result image of the current image frame can be obtained by the following formula (6): (6) in, The denoising result image of the t-th image frame; An alignment map representing the denoising result of the previous image frame; Indicates the current image frame; Represents the filtering result map of the current image frame. It represents the current image frame, the filtering result map of the current image frame, the denoising result alignment map of the previous image frame, and the multi-channel stitching image obtained by channel stitching, which is the input of the denoising model. and represents the weight matrix and bias vector in the denoising model, is the activation function, such as or For the first image frame of the video to be processed, the output denoising result image is equal to the input original image, that is, .

[0102] In actual implementation, the denoising model can be trained by the following method: Collect or generate noisy sample images and corresponding clean image labels, the sample images may include the current sample image frame, the sample filtering result map of the current sample image frame, and the sample denoising result alignment map of the previous sample image frame; perform preprocessing operations such as standardization and normalization on the sample image; increase data diversity by rotation, flipping, scaling, etc. to obtain an enhanced sample image; input the enhanced sample image into the initial denoising model to obtain its output predicted denoising result map. Calculate the loss value based on the clean image label and the predicted denoising result map, the loss value is used to indicate the pixel difference between the predicted denoising result map and the clean image label; reversely adjust the model parameters of the initial denoising model based on the loss value, and continue to return to the step of collecting or generating noisy sample images and corresponding clean image labels until the training stop condition is met, and gradually optimize the parameters through multiple iterations of model training to obtain a trained denoising model. The above training process can select a suitable optimizer, such as Adam, RMSprop, etc.; set a suitable batch size, usually 16, 32 or 64; set a suitable learning rate and adjust it according to the training situation.

[0103] In an optional implementation of this embodiment, after obtaining image coding features according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and decoding the image coding features to obtain the noise reduction result map of the current image frame, the method further includes: Cache the current image frame, the filtering result map of the current image frame, and the noise reduction result map of the current image frame; The next image frame in the video to be processed is obtained as the current image frame, and the operation steps of obtaining the current image frame of the video to be processed and the noise reduction result image of the previous image frame are returned until the last image frame in the video to be processed, and the target video after noise reduction is obtained.

[0104] In actual implementation, the current image frame, the filtering result map of the current image frame, and the noise reduction result map of the current image frame can be stored, and then the next image frame in the video to be processed is obtained as the current image frame. For the updated current image frame, its previous image frame, the filtering result map of the previous image frame, and the noise reduction result map of the previous image frame can be read from the cache, and the noise reduction processing is continued on the updated current image frame until the last image frame in the video to be processed is completed, and the denoised target video can be obtained.

[0105] As an example, Figure 3 FIG. 1 is an algorithm diagram of an image denoising method provided by an embodiment of the present specification, and is described by taking an IIR filter as a filter and an RNN network as a denoising model as an example. Figure 3 As shown, it is determined that the current image frame in the video to be processed is , get the filtering result of the previous image frame, that is, the output of the IIR filter for the previous image frame , the filtering result map of the current image frame and the previous image frame Input IIR filter and output the filtering result image of the current image frame .

[0106] Get the denoising result of the previous image frame, that is, the denoising result output by the RNN network . The filtering result map of the current image frame , current image frame And the denoising result of the previous image frame Input the RNN network and output the denoising result of the current image frame .

[0107] Then, As the current image frame, obtain the filtering result map of the previous image frame, that is, the output of the IIR filter for the previous image frame , the current image frame And the filtering result of the previous image frame Input IIR filter and output the filtering result image of the current image frame .

[0108] Get the denoising result of the previous image frame, that is, the denoising result output by the RNN network . The filtering result map of the current image frame , current image frame And the denoising result of the previous image frame Input the RNN network and output the denoising result of the current image frame .

[0109] Afterwards, As the current image frame, obtain the filtering result map of the previous image frame, that is, the output of the IIR filter for the previous image frame , the current image frame And the filtering result of the previous image frame Input IIR filter and output the filtering result image of the current image frame .

[0110] Get the denoising result of the previous image frame, that is, the denoising result output by the RNN network . The filtering result map of the current image frame , current image frame And the denoising result of the previous image frame Input the RNN network and output the denoising result of the current image frame .

[0111] And so on, until the last image frame in the video to be processed is processed.

[0112] As an example, Figure 4 1 is a schematic diagram of a network structure of an image denoising method provided by an embodiment of the present specification, taking an IIR filter as a filter and an RNN network as a denoising model as an example for explanation, and the RNN network is an RNN-U-Net network structure, such as Figure 4 As shown, the filtering result map output by the IIR filter for the previous image frame can be used together with the current image frame as the input of the IIR filter, and pixel-by-pixel weighting is performed to output the filtering result map of the current image frame.

[0113] The current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame are channel-joined and then input into the RNN network, which includes an encoder and a decoder, and outputs the denoising result map of the current image frame based on the processing of the convolution layer, pooling layer, upsampling layer, feature decoding layer, dimension mapping layer and activation function in the encoder and the decoder. The denoising result alignment map of the previous image frame is obtained by spatially aligning the denoising result map of the previous image frame and the current image frame.

[0114] like Figure 4As shown in the figure, in the RNN-U-Net network structure, Conv.3x3+ReLU represents a convolutional layer with a 3x3 filter, followed by a nonlinear activation function, usually ReLU (Rectified Linear Unit). This combination is usually used to extract image features. The convolutional layer uses a sliding window (3x3 matrix) to perform operations on the input image, and then applies the ReLU activation function to keep positive values ​​and eliminate negative values. Max-pooling (2x2) means performing a maximum pooling operation, in which the input image is divided into a 2x2 grid, and then the maximum value of each grid is taken as the output, which helps to reduce the size of the feature map while retaining important features. Conv.Trans.3x3+ReLU represents a 3x3 transposed convolution layer, also known as an "upsampling" layer, which is used to increase the size of the feature map. The working principle of the transposed convolution layer is similar to that of the standard convolution layer, but its weights are specially arranged so that the output is larger than the input. Similarly, this layer will also be followed by a ReLU activation function. Conv.1x1+ReLU represents a 1x1 convolution layer, which is mainly used to reduce the number of channels or change the dimension of the feature map. Although the width and height are only 1 pixel, they can perform linear transformations between channels. Similarly, this layer will be followed by a ReLU activation function. Crop© represents a cropping and copying operation. In order to match the shapes between different layers, in deep learning networks, the feature map needs to be cropped to adapt to the requirements of subsequent layers and then copied to the new location. These layers are usually stacked in a certain order to form a complete RNN network structure.

[0115] like Figure 4 As shown in the figure, in the encoding stage, after the input data undergoes two Conv.3x3+ReLU operations and one Max-pooling (2x2) operation, the output data becomes smaller in height but increases in width, indicating that the image size is reduced but the number of channels is increased in this process; in the decoding stage, after each Conv.Trans.3x3+ReLU and two Conv.3x3+ReLU operations, the output data becomes larger in height but decreases in width, indicating that the image size is increased but the number of channels is reduced in this process, which is used to characterize that the decoding stage gradually restores the image size based on the channel information.

[0116] The optical flow is calculated for the denoising result map of the current image frame output by the RNN network, and the spatial position is aligned with the next image frame (the next original frame) to obtain the denoising result alignment map of the next image frame. The recursive iteration is continued until the last image frame in the video to be processed is processed.

[0117] It should be noted that a recursive neural network structure is added to the basic U-Net framework. The denoising model uses the RNN-U-Net network structure to perform multiple iterative calculations on each image frame in the video to be processed, and uses the denoising result output of the previous image frame as the input data for denoising the current image frame, gradually optimizing the denoising results of each image frame in the video to be processed and improving the denoised image quality of the image frame.

[0118] The embodiments of the present specification provide an image denoising method, which performs filtering processing before encoding the current image frame, combines filtering capability with image feature encoding capability, and realizes efficient suppression of noise in the image frame while retaining more detail information; furthermore, the denoising result output by the previous image frame can be used as the input for the denoising processing of the current image frame, and the denoising result of the previous image frame is learned when the denoising processing is performed on the current image frame, and a recursive mechanism is used to realize timing modeling, and the denoising result of the image frame is optimized by combining filtering processing and timing modeling, thereby improving the accuracy of the denoising processing, obtaining better denoised image quality, ensuring the denoising effect, and improving the quality of the image frame obtained by denoising.

[0119] The following combination Figure 5 Taking the application of the image denoising method provided in this specification in the short video scene of the content application platform as an example, the image denoising method is further described. Figure 5 A processing flow chart of an image noise reduction method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0120] Step 502: Obtain the short video to be processed from the content application platform.

[0121] Step 504: for the first frame original image PreSrcIn in the short video, determine that the first frame filtering result image PreIIROut and the first frame denoising result image PreRNNOut are both the first frame original image PreSrcIn, and cache the first frame original image PreSrcIn, the first frame filtering result image PreIIROut, and the first frame denoising result image PreRNNOut.

[0122] Step 506: collect the original image CurSrcIn of the t-th frame in the short video, combine it with the filtering result image PreIIROut of the t-1th frame, and obtain the filtering result image CurIIROut of the t-th frame through the IIR filter position-by-position pixel weighting and ghosting elimination calculation method.

[0123] Here, t is initially 2.

[0124] Step 508: using the t-1th frame original image PreSrcIn and the tth frame original image CurSrcIn, calculate the dense optical flow coordinate mapping map FlowMap of the tth frame original image relative to the corresponding pixel points of the t-1th frame original image.

[0125] Step 510: Use the dense optical flow coordinate mapping map FlowMap to perform an image space position alignment operation on the t-1th frame denoising result map PreRNNOut to obtain the t-1th frame denoising alignment result map PreAlignRNNOut aligned with the tth frame original image.

[0126] Step 512: Channel superposition is performed on the filtering result image CurIIROut of the t-th frame, the noise reduction alignment result image PreAlignRNNOut of the t-1th frame, and the original image CurSrcIn of the t-th frame to obtain a 9-channel spliced ​​image as the input of the RNN network.

[0127] Step 514: The RNN network calculates and outputs the t-th frame noise reduction result graph CurRNNOut as the final noise reduction result.

[0128] Step 516: Cache the filtering result image CurIIROut of the t-th frame, the noise reduction result image CurRNNOut of the t-th frame, and the original image CurSrcIn of the t-th frame.

[0129] Let t be incremented by 1, and return to execute the above step 506 to form a recursive reasoning closed loop, until t is equal to the number of image frames included in the short video, the short video denoising process is completed, and the target video is obtained.

[0130] Figure 6 FIG. 1 shows a schematic diagram of an algorithm implementation of an image denoising method provided by an embodiment of the present specification, taking an IIR filter as an example and an RNN network as a denoising model as an example. Figure 6 As shown, the filtering result images of the current image frame and the previous image frame are input into the IIR filter to obtain the filtering result image of the current image frame. The optical flow is calculated for the previous image frame and the current image frame to obtain an optical flow coordinate mapping image, and then the optical flow coordinate mapping image and the denoising result image of the previous image frame are spatially aligned to obtain the denoising result alignment image of the previous image frame.

[0131] The filtering result map of the current image frame, the denoising result alignment map of the previous image frame, and the current image frame are input into the RNN network to obtain the denoising result map of the current image frame.

[0132] The filtering result map of the current image frame, the noise reduction result map of the current image frame, and the iterative calculation process of the current image frame entering the next image frame determine the noise reduction result map of the next image frame.

[0133] The embodiments of the present specification provide an image denoising method, which performs filtering processing before encoding the current image frame, combines filtering capability with image feature encoding capability, and realizes efficient suppression of noise in the image frame while retaining more detail information; furthermore, the denoising result output by the previous image frame can be used as the input for the denoising processing of the current image frame, and the denoising result of the previous image frame is learned when the denoising processing is performed on the current image frame, and a recursive mechanism is used to realize timing modeling, and the denoising result of the image frame is optimized by combining filtering processing and timing modeling, thereby improving the accuracy of the denoising processing, obtaining better denoised image quality, ensuring the denoising effect, and improving the quality of the image frame obtained by denoising.

[0134] Corresponding to the above method embodiment, this specification also provides an image noise reduction device embodiment, Figure 7 FIG. 1 is a schematic diagram showing the structure of an image noise reduction device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises: The acquisition module 702 is configured to acquire a current image frame of the video to be processed and a noise reduction result image of a previous image frame; The determination module 704 is configured to perform filtering processing on the current image frame to obtain a filtering result image of the current image frame; The noise reduction processing module 706 is configured to obtain image coding features according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and decode the image coding features to obtain the noise reduction result map of the current image frame.

[0135] Optionally, the determination module 704 is further configured to: Obtain a filtering result image of a previous image frame of the video to be processed; The filtering result graphs of the current image frame and the previous image frame are filtered by the filter to determine the filtering result graph of the current image frame.

[0136] Optionally, the determination module 704 is further configured to: determining filter coefficients of a filter configuration; By using a filter, a first pixel value of a target pixel in a current image frame, a second pixel value of the target pixel in a filter result image of a previous image frame, and a filter coefficient are calculated to determine a target pixel value of the target pixel, wherein the target pixel is a pixel at any position in the current image frame; According to the target pixel value of each pixel in the current image frame, the filtering result map of the current image frame is output.

[0137] Optionally, the determination module 704 is further configured to: Determine a pixel difference value of the target pixel according to a first pixel value of the target pixel in the current image frame and a second pixel value of the target pixel in the filtering result image of the previous image frame; According to the pixel difference and the set difference threshold, the filter coefficient corresponding to the target pixel in the filter is determined.

[0138] Optionally, the determination module 704 is further configured to: Determine the adjacent pixels around the target pixel according to the set coordinate range; The filter coefficient corresponding to the target pixel in the filter is determined according to the filter coefficients of the adjacent pixels, wherein the filter coefficients of the adjacent pixels are determined based on the pixel difference between the filter result images of the current image frame and the previous image frame.

[0139] Optionally, the noise reduction processing module 706 is further configured to: Performing spatial position alignment on the denoising result map of the previous image frame and the current image frame to obtain an alignment map of the denoising result of the previous image frame; splicing the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame, and inputting the spliced ​​images into an encoder of a denoising model for encoding to obtain the image coding feature; The image coding features are input into the decoder of the denoising model, and the denoising result image of the current image frame is obtained by decoding.

[0140] Optionally, the noise reduction processing module 706 is further configured to: Obtain the image frame before the current image frame in the video to be processed; Perform optical flow calculation on the previous image frame and the current image frame to obtain an optical flow coordinate mapping diagram; The denoising result map of the previous image frame and the optical flow coordinate mapping map are spatially aligned to obtain the denoising result alignment map of the previous image frame.

[0141] Optionally, the noise reduction processing module 706 is further configured to: The current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame are spliced ​​along the channel dimension to obtain a multi-channel spliced ​​image; The multi-channel spliced ​​image is input into the encoder of the denoising model to obtain the image encoding features.

[0142] Optionally, the device further includes a cache module configured to: Cache the current image frame, the filtering result map of the current image frame, and the noise reduction result map of the current image frame; The next image frame in the video to be processed is obtained as the current image frame, and the operation steps of obtaining the current image frame of the video to be processed and the noise reduction result image of the previous image frame are returned until the last image frame in the video to be processed, and the target video after noise reduction is obtained.

[0143] The embodiment of the present specification provides an image denoising device, including an acquisition module, a determination module and a denoising processing module. Before performing image encoding on the current image frame, filtering processing is first performed, and filtering capability and image feature encoding capability are combined to achieve efficient suppression of noise in the image frame while retaining more detail information; and the denoising result output by the previous image frame can be used as the input for the denoising processing of the current image frame, and the denoising result of the previous image frame is learned when performing denoising processing on the current image frame, and a recursive mechanism is used to implement timing modeling, and the denoising result of the image frame is optimized by combining filtering processing and timing modeling, thereby improving the accuracy of the denoising processing, obtaining better denoised image quality, ensuring the denoising effect, and improving the quality of the image frame obtained by denoising.

[0144] The above is a schematic scheme of an image denoising device of this embodiment. It should be noted that the technical scheme of the image denoising device and the technical scheme of the above-mentioned image denoising method belong to the same concept, and the details not described in detail in the technical scheme of the image denoising device can be referred to the description of the technical scheme of the above-mentioned image denoising method.

[0145] Figure 8 The block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and the database 850 is used to store data.

[0146] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0147] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0148] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0149] The processor 820 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned image noise reduction method.

[0150] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned image denoising method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned image denoising method.

[0151] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned image noise reduction method when executed by a processor.

[0152] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned image denoising method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned image denoising method.

[0153] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image noise reduction method.

[0154] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned image denoising method belong to the same concept, and the details not described in detail in the technical solution of the computer program can be referred to the description of the technical solution of the above-mentioned image denoising method.

[0155] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0156] Computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of computer readable media may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0157] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image denoising method, characterized in that: include: Get the current image frame of the video to be processed and the noise reduction result image of the previous image frame; Performing filtering processing on the current image frame to obtain a filtering result image of the current image frame; An image coding feature is obtained according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and the image coding feature is decoded to obtain the noise reduction result map of the current image frame.

2. The image denoising method according to claim 1, characterized in that: The filtering process is performed on the current image frame to obtain a filtering result image of the current image frame, including: Obtaining a filtering result image of a previous image frame of the video to be processed; The filtering result images of the current image frame and the previous image frame are filtered by a filter to determine the filtering result image of the current image frame.

3. The image denoising method according to claim 2, characterized in that: The filtering process is performed on the filtering result graphs of the current image frame and the previous image frame by the filter to determine the filtering result graph of the current image frame, including: determining filter coefficients of the filter configuration; By using the filter, a first pixel value of a target pixel in the current image frame, a second pixel value of the target pixel in the filter result image of the previous image frame, and the filter coefficient are calculated to determine a target pixel value of the target pixel, wherein the target pixel is a pixel at any position in the current image frame; According to the target pixel value of each pixel in the current image frame, a filtering result image of the current image frame is output.

4. The image denoising method according to claim 3, characterized in that: Determining the filter coefficients of the filter configuration includes: Determine a pixel difference value of the target pixel according to a first pixel value of the target pixel in the current image frame and a second pixel value of the target pixel in the filtering result image of the previous image frame; A filter coefficient corresponding to the target pixel in the filter is determined according to the pixel difference and a set difference threshold.

5. The image denoising method according to claim 3, characterized in that: The determining of the filter coefficients of the filter configuration comprises: Determine adjacent pixels around the target pixel according to a set coordinate range; The filter coefficient corresponding to the target pixel in the filter is determined according to the filter coefficients of the adjacent pixels, wherein the filter coefficients of the adjacent pixels are determined based on the pixel difference between the filter result images of the current image frame and the previous image frame.

6. The image denoising method according to any one of claims 1 to 5, characterized in that: The step of obtaining an image coding feature according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and decoding the image coding feature to obtain the noise reduction result map of the current image frame comprises: Performing spatial position alignment on the noise reduction result map of the previous image frame and the current image frame to obtain an aligned noise reduction result map of the previous image frame; splicing the current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame, and inputting the spliced ​​images into an encoder of a denoising model for encoding to obtain the image coding feature; The image coding features are input into the decoder of the denoising model, and the denoising result image of the current image frame is obtained by decoding.

7. The image denoising method according to claim 6, characterized in that: The step of spatially aligning the denoising result map of the previous image frame and the current image frame to obtain an aligned denoising result map of the previous image frame includes: Obtaining an image frame previous to a current image frame in the video to be processed; Performing optical flow calculation on the previous image frame and the current image frame to obtain an optical flow coordinate mapping diagram; The denoising result map of the previous image frame and the optical flow coordinate mapping map are spatially aligned to obtain an aligned denoising result map of the previous image frame.

8. The image denoising method according to claim 6, characterized in that: The step of splicing the current image frame, the filtering result map of the current image frame, and the denoising result alignment map of the previous image frame and inputting them into the encoder of the denoising model for encoding to obtain the image coding feature includes: The current image frame, the filtering result map of the current image frame and the denoising result alignment map of the previous image frame are spliced ​​along the channel dimension to obtain a multi-channel spliced ​​image; The multi-channel spliced ​​image is input into the encoder of the denoising model to obtain the image coding features.

9. The image denoising method according to any one of claims 1 to 5, characterized in that: After obtaining the image coding feature according to the current image frame, the filtering result map of the current image frame and the noise reduction result map of the previous image frame, and decoding the image coding feature to obtain the noise reduction result map of the current image frame, the method further includes: caching the current image frame, a filtering result map of the current image frame, and a noise reduction result map of the current image frame; The next image frame in the video to be processed is obtained as the current image frame, and the operation steps of obtaining the current image frame of the video to be processed and the noise reduction result image of the previous image frame are returned until the last image frame in the video to be processed, so as to obtain the target video after noise reduction.

10. An image noise reduction device, characterized in that: include: An acquisition module is configured to acquire a current image frame of the video to be processed and a noise reduction result image of a previous image frame; A determination module is configured to perform filtering processing on the current image frame to obtain a filtering result image of the current image frame; The denoising processing module is configured to obtain image coding features according to the current image frame, the filtering result map of the current image frame and the denoising result map of the previous image frame, and decode the image coding features to obtain the denoising result map of the current image frame.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the image denoising method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the image denoising method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the image denoising method according to any one of claims 1 to 9.

Images