Video noise reduction method and device, equipment and storage medium

By combining an infinite impulse response filter and an AI noise reduction network in a multi-scale Encoder-Decoder structure, the problems of hardware resource consumption and flicker in video surveillance under extremely low illumination are solved, achieving low latency and efficient video noise reduction.

CN118967499APending Publication Date: 2024-11-15SENSLAB INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411007219.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Under extremely low illumination, existing video noise reduction methods suffer from excessive hardware resource consumption, high latency, and noticeable flickering. This is especially true in video surveillance, where the computing power of terminal devices is limited, making it difficult to achieve effective noise reduction.

Method used

A method combining temporal filtering based on infinite impulse response with an AI noise reduction network is adopted. The method calculates two frames of images, performs temporal fusion using temporal filtering based on infinite impulse response, and combines it with a multi-scale Encoder-Decoder structure based on convolutional neural network for noise reduction.

Benefits of technology

It achieves low memory usage and low latency video noise reduction, effectively eliminating flickering and improving the continuity and naturalness of video viewing, while also achieving real-time noise reduction on resource-limited devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967499A_ABST
    Figure CN118967499A_ABST
Patent Text Reader

Abstract

The invention provides a video noise reduction method and device, equipment and a storage medium. The method comprises the following steps: acquiring an input image and a first fusion image of a video; performing time domain fusion on the input image and the first fusion image through time domain filtering of infinite pulse response to obtain a second fusion image; performing noise reduction processing on the second fusion image through an AI noise reduction network model to obtain a noise reduction image; and outputting the noise-reduced image. Compared with the prior art in which only the input image of the current frame and the first fusion image of the previous frame are adopted for calculation during noise reduction of the video frame, calculation is carried out only through two frames without increasing the frame number, and compared with a five-frame calculation mode in the prior art, the method has the advantages that lower memory occupancy and lower delay can be realized, and the noise reduction efficiency is improved. Through combination of time domain filtering of infinite pulse response and an AI noise reduction network model, the noise reduction effect in an extremely low illumination environment is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a video noise reduction method, device, equipment and storage medium. Background Art

[0002] The video surveillance field has very high requirements for low illumination of images. Objects need to be seen clearly in a dark environment. However, the surveillance images captured by the camera under extremely low illumination contain a lot of noise and the signal-to-noise ratio is extremely poor. Although infrared fill light can be used, infrared images will lose color and a lot of details, and have a limited range. Therefore, for low-illuminance visible light images, noise reduction processing must be performed to restore the real scene within the monitoring range.

[0003] At present, although the noise reduction effect of deep learning methods is good, in scenes with extremely low illumination (for example, less than 0.1lux), the video after AI noise reduction is prone to produce incoherent front and back frames, and obvious flickering, which is abnormal and uncomfortable to watch. On this basis, DVDnet and FastDVDNet have made improvements for extremely low illumination and flickering, but the cost is that 5 consecutive frames of video need to be input. In order to cache these frames, too many hardware resources will be occupied. In addition, the middle frame of the 5 consecutive frames is used for noise reduction, so it will generate at least 2 frames of delay. If the number of frames is increased, the hardware resources will be further increased, and the flexibility is poor. Summary of the invention

[0004] The purpose of the present invention is to provide a video noise reduction method, device, equipment and storage medium that can reduce memory usage by calculating only two frames and using smaller parameters.

[0005] In a first aspect, a video denoising method comprises: obtaining an input image and a first fused image of a video, wherein the input image is an original image obtained by an image sensor; performing time-domain fusion of the input image and the first fused image through time-domain filtering of an infinite impulse response to obtain a second fused image, comprising: performing motion estimation on the input image and a reference image to dynamically obtain a gain coefficient and a feedback coefficient of the time-domain filtering of the infinite impulse response according to the input image and the reference image; performing denoising processing on the second fused image through an AI denoising network model to obtain a denoised image; and outputting the denoised image.

[0006] Further, performing time-domain fusion on the input image and the first fused image through infinite impulse response time-domain filtering to obtain the second fused image includes: performing weighted addition on the input image and the first fused image to obtain the second fused image.

[0007] Further, performing weighted addition on the input image and the first fused image to obtain the second fused image includes: cascading and decomposing the differential equation of the time domain filter of the Q-order infinite impulse response into a plurality of first-order filter equations as follows:

[0008] y j [t] = g j ·x[t]+p j y[t-1]

[0009] Where j represents the index value of different filter equations, ranging from 1 to Q, x[t] represents the input image of the jth first-order filter equation at time t, and y j [t] represents the fused image output of the jth first-order filter equation at time t, y[t-1] represents the fused image output of the jth first-order filter equation at time t-1, g j It means that x[t] depends on y under the jth first-order filter equation j [t] Gain coefficient of influence degree, p j It means that y[t-1] is proportional to y under the jth first-order filter equation. j [t] Feedback coefficient of the impact degree.

[0010] Further, performing motion estimation on the input image and the reference image to dynamically obtain the gain coefficient and feedback coefficient of the time domain filtering of the infinite impulse response according to the input image and the reference image includes: performing motion estimation according to the input image and the first fused image to determine the motion area and intensity of each pixel.

[0011] Further, g is determined according to the motion area and intensity of each pixel of the input image and the first fused image. j and p j Including: Determine g by motion estimation method j and p j , the formula is as follows:

[0012] {g j ,p j =MotionEst(x[t],y[t-1])

[0013] Among them, g j It means that x[t] depends on y under the jth first-order filter equation j [t] Gain coefficient of influence degree, p j It means that y[t-1] is proportional to y under the jth first-order filter equation. j[t] is the feedback coefficient of the influence degree, x[t] represents the input image of the j-th first-order form of the filter equation at time t, y[t-1] represents the fused image output of the j-th first-order form of the filter equation at time t-1, and MotionEst represents the motion estimation algorithm.

[0014] Furthermore, performing motion estimation based on the input image and the first fused image to determine the motion area and intensity of each pixel includes: determining the motion area and intensity of each pixel based on at least one of an inter-frame difference method, a background subtraction method, an optical flow method, a connected region analysis, a spectral analysis, and a multimodal fusion method.

[0015] Furthermore, performing denoising on the second fused image through an AI denoising network model to obtain a denoised image includes: performing denoising on the second fused image using a multi-scale Encoder-Decoder structure based on a convolutional neural network.

[0016] Furthermore, adopting a multi-scale Encoder-Decoder structure based on a convolutional neural network includes: extracting multi-scale features through multiple convolutional layers and pooling layers of the Encoder structure.

[0017] Furthermore, features are transferred between different levels through skip connections or residual structures. Through the Decoder structure, upsampling or deconvolution is used in combination with ordinary convolutional layers, or the image size is restored layer by layer by pixel rearrangement, and the features of skip connections or residual structures are combined for fusion.

[0018] Furthermore, combining the features of the skip layer connection or the residual structure for fusion includes: adding a skip layer connection between the Encoder and the Decoder, and passing the features extracted in the Encoder stage to the corresponding Decoder stage.

[0019] Furthermore, combining the features of skip-layer connection or residual structure for fusion includes: introducing residual blocks and skipping one or more convolutional layers through short connections.

[0020] Unit 2: This application also proposes a video noise reduction device, including:

[0021] An acquisition unit, configured to acquire an input image and a first fused image of a video, wherein the input image is an original image acquired by an image sensor;

[0022] The time domain filtering unit is used to perform time domain fusion of the input image and the first fused image through the time domain filtering of infinite impulse response to obtain the second fused image, including: performing motion estimation on the input image and the reference image to dynamically obtain the gain coefficient and feedback coefficient of the time domain filtering of infinite impulse response according to the input image and the reference image;

[0023] An algorithm filtering unit, used for performing noise reduction processing on the second fused image through an AI noise reduction network model to obtain a noise reduction image;

[0024] The output unit is used to output the denoised image.

[0025] In a third aspect, the present application further proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the at least one processor can execute the method described in the above embodiment.

[0026] In a fourth aspect, the present application further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0027] The beneficial effects of the present invention are:

[0028] (1) Compared with the solution in the prior art, when denoising the video frame, only the input image of the current frame and the first fused image of the previous frame are used for calculation. The calculation is performed on two frames without increasing the number of frames. Compared with the prior art method of calculating five frames, lower memory usage and lower latency can be achieved;

[0029] (2) By combining the infinite impulse response time domain filtering and AI noise reduction network model, the noise reduction effect in extremely low illumination environments is effectively improved. This solves the problems in the existing technology that the terminal equipment of video surveillance has very limited computing power and cannot effectively reduce video noise, the deep learning method produces incoherent front and back frames with obvious flickering, and the DVDnet and FastDVDNet methods produce large delays and memory usage.

[0030] (3) By utilizing the time-domain fusion characteristics of the infinite impulse response time-domain filter, the previous and next frames of the video are weighted, and each frame is associated with the previous frame to achieve smooth processing of the image. In this way, the temporal correlation can be used to eliminate instantaneous noise, thereby reducing the flicker phenomenon, effectively eliminating the flicker phenomenon in AI noise reduction, and improving the continuity and naturalness of the video perception;

[0031] (4) Effectively utilize the temporal correlation between the previous and next frames of the video, and the denoised image retains structural details, and the denoised image retains more structural details;

[0032] (5) The infinite impulse response (IIR) time domain filtering removes part of the time domain noise, reduces the overall noise level, and makes the fused frame contain more information, providing a cleaner input for the AI ​​denoising network model and significantly reducing the parameters and computational complexity of the model. Fewer parameters means a lighter model, which can run efficiently on resource-limited devices while ensuring the denoising effect, achieving real-time, low memory, and low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural block diagram of the method of the present invention;

[0034] Figure 2 It is a flowchart of the present invention;

[0035] Figure 3 is a schematic structural diagram of a receiving device provided by an embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0038] Currently, the video surveillance field has very high requirements for low-light images, and objects need to be seen clearly in dark environments. However, the surveillance images taken by cameras under extremely low illumination contain a lot of noise and the signal-to-noise ratio is extremely poor. Although infrared fill light can be used, infrared images will lose color and a lot of details, and have range limitations. Therefore, for low-light visible light images, noise reduction processing must be performed to restore the real scene within the monitoring range.

[0039] The current video denoising methods are divided into two categories; one is the traditional method, such as BM3D / V-BM4D / VNLB, etc.; the other is based on deep learning methods. Currently, the more advanced technologies include DVDnet, VNLnet and FastDVDNet.

[0040] The computing power of video surveillance terminal devices is very limited. Traditional methods are too computationally intensive and consume a lot of resources while often failing to meet real-time requirements. In addition, traditional algorithms are prone to motion ghosting and blurring, and obvious low-frequency noise still remains in the denoised image. They are unable to target the noise generated by a specific image sensor.

[0041] Deep learning methods have better noise reduction effects, but in scenes with extremely low illumination (e.g., below 0.1 lux), the AI ​​noise reduction performance of a single frame image is weakened. For example, VNLnet only performs best for smaller noises. When the illumination decreases and the noise increases, the peak signal-to-noise ratio (PSNR) of the noise-reduced image decreases sharply. In addition, surveillance cameras shoot video data, and AI noise-reduced videos are prone to produce incoherent front and back frames, with obvious flickering, which is abnormal and uncomfortable to watch.

[0042] DVDnet and FastDVDNet have made improvements for extremely low illumination and flickering, but at the cost of requiring the input of 5 continuous video frames. In order to cache these frames, too many hardware resources will be occupied. In addition, the noise reduction is performed on the middle frame of the 5 continuous frames, so at least 2 frames of delay will be generated.

[0043] In the second stage, DVDNet also aligns adjacent frames to the middle frame through optical flow deformation. The optical flow method has a huge amount of calculation and will introduce artifacts.

[0044] DVDnet and FastDVDNet only use 5 frames for historical images. Increasing the frame number will further increase hardware resources and have poor flexibility.

[0045] In order to solve the above technical defects, the following solution is proposed. Figure 1 and Figure 2 , a video noise reduction method, comprising:

[0046] S101, obtaining an input image and a first fused image of a video, wherein the input image is an original image obtained by an image sensor;

[0047] S102, performing time domain fusion on the input image and the first fused image through infinite impulse response time domain filtering to obtain a second fused image, including:

[0048] Performing motion estimation on the input image and the reference image to dynamically obtain the gain coefficient and feedback coefficient of the time domain filtering of the infinite impulse response according to the input image and the reference image;

[0049] S103, performing denoising processing on the second fused image through an AI denoising network model to obtain a denoised image;

[0050] S104: output the denoised image.

[0051] Specifically, an input image and a first fused image of a video are obtained, wherein the input image is a current frame read from the video as the input image, and the first fused image is an image of the previous frame processed by an IIR filter. The input image and the first fused image are time-domain fused using an IIR filter to obtain a second fused image, and the second fused image is subjected to AI denoising. Here, we can use a pre-trained convolutional neural network (CNN) model to achieve denoising, and we can use an existing model or train a model ourselves. This solution is a prior art and will not be described in detail here. Then, the processed image is output;

[0052] Through this method, effective noise reduction can be performed in the time domain of the video while retaining image details, thereby improving the quality of the video in a dark light environment. Compared with the prior art in which only the input image of the current frame and the first fused image of the previous frame are used for calculation when reducing the noise of the video frame, only two frames are used for calculation, and there is no need to increase the number of frames for calculation. Compared with the prior art in which five frames are used for calculation, lower memory usage and lower latency can be achieved. By combining the infinite impulse response time domain filtering and the AI ​​noise reduction network model, the noise reduction effect in an extremely low illumination environment is effectively improved, and the problems in the prior art in which the terminal equipment for video surveillance has very limited computing power and cannot effectively reduce video noise, the deep learning method produces incoherent front and back frames with obvious flickering, and the DVDnet and FastDVDNet methods produce large latency and memory usage.

[0053] In addition, the gain coefficient and the feedback coefficient can be dynamically obtained for the input image and the first fused image each time to perform IIR calculation without introducing other artifacts.

[0054] As a further specific embodiment, Figure 2 , performing time domain fusion of the input image and the first fused image through infinite impulse response time domain filtering to obtain a second fused image includes: performing weighted addition of the input image and the first fused image to obtain the second fused image.

[0055] As a further specific embodiment, Figure 2 , performing weighted addition of the input image and the first fused image to obtain the second fused image includes: cascading and decomposing the differential equation of the time domain filter of the Q-order infinite impulse response into multiple first-order filter equations as follows:

[0056] y j [t] = g j ·x[t]+p j y[t-1]

[0057] Where j represents the index value of different filter equations, ranging from 1 to Q, x[t] represents the input image of the jth first-order filter equation at time t, and y j [t] represents the fused image output of the jth first-order filter equation at time t, y[t-1] represents the fused image output of the jth first-order filter equation at time t-1, g j It means that x[t] depends on y under the jth first-order filter equation j [t] Gain coefficient of influence degree, p j It means that y[t-1] is proportional to y under the jth first-order filter equation. j [t] Feedback coefficient of the impact degree.

[0058] Specifically, Figure 2 The current noisy video frame x[t] represents the input image under the j-th first-order form of the filtering equation, and the previous frame I IR fused frame y[t-1] represents the first fused image obtained after the input image of the previous frame is calculated. It is calculated by the input image of the previous frame and the first fused image of the previous frame, and the time domain fusion characteristics of the time domain filtering of the infinite impulse response are used. By weighting the previous and next frames of the video, each frame is associated with the previous frame to achieve smoothing of the image. In this way, time correlation can be used to eliminate instantaneous noise, thereby reducing flickering.

[0059] As a further specific embodiment, performing motion estimation on the input image and the reference image to dynamically obtain the gain coefficient and feedback coefficient of the time domain filtering of the infinite impulse response according to the input image and the reference image includes: performing motion estimation according to the input image and the first fused image to determine the motion area and intensity of each pixel.

[0060] As a further specific embodiment, g is determined according to the motion area and intensity of each pixel of the input image and the first fused image. j and p j Including: Determine g by motion estimation method j and p j , the formula is as follows:

[0061] {g j ,p j =MotionEst(x[t],y[t-1])

[0062] Among them, g j It means that x[t] depends on y under the jth first-order filter equation j [t] Gain coefficient of influence degree, p j It means that y[t-1] is proportional to y under the jth first-order filter equation. j[t] is the feedback coefficient of the influence degree, x[t] represents the input image of the j-th first-order form of the filter equation at time t, y[t-1] represents the fused image output of the j-th first-order form of the filter equation at time t-1, and MotionEst represents the motion estimation algorithm.

[0063] As a further specific embodiment, performing motion estimation based on the input image and the first fused image to determine the motion area and intensity of each pixel includes: determining the motion area and intensity of each pixel based on at least one of an inter-frame difference method, a background subtraction method, an optical flow method, a connected region analysis, a spectral analysis, and a multimodal fusion method.

[0064] Specifically, the inter-frame difference method calculates the pixel differences between the current frame and the previous frame to detect areas with large changes, which are usually motion areas;

[0065] Background subtraction method, using background modeling (such as Gaussian mixture model) to build a background model, and then subtract the background from the current frame to obtain the foreground image, the foreground area is the motion area;

[0066] Optical flow method: Use an optical flow algorithm (such as the Farneback method or the Lucas-Kanade method) to calculate the optical flow field between adjacent frames, obtain the motion vector of each pixel, and calculate the motion intensity by the size of the motion vector;

[0067] Connected region analysis: Conduct connected region analysis on the motion detection results to determine the shape and connectivity of the motion region and further accurately divide the motion region;

[0068] Spectral analysis, converting the image to the frequency domain and detecting motion by analyzing the frequency domain features;

[0069] Multimodal fusion,combines the results of the above methods and fuses the information of multiple motion,detection methods to improve the robustness and accuracy of motion detection.

[0070] As a further specific embodiment, performing denoising processing on the second fused image through an AI denoising network model to obtain a denoised image includes: performing denoising processing on the second fused image using a multi-scale Encoder-Decoder structure based on a convolutional neural network.

[0071] As a further specific embodiment, adopting a multi-scale Encoder-Decoder structure based on a convolutional neural network includes: extracting multi-scale features through multiple convolutional layers and pooling layers of the Encoder structure.

[0072] As a further specific embodiment, features are transferred between different levels through skip-layer connections or residual structures; the image size is restored layer by layer through upsampling of the Decoder structure, and the features of the skip-layer connections or residual structures are combined for fusion.

[0073] As a further specific embodiment, fusing features in combination with skip-layer connections or residual structures includes: adding a skip-layer connection between the Encoder and the Decoder, and passing the features extracted in the Encoder stage to the corresponding Decoder stage.

[0074] As a further specific embodiment, combining the features of skip-layer connection or residual structure for fusion includes: introducing residual blocks and skipping one or more convolutional layers through short connections.

[0075] Specifically, a multi-scale Encoder-Decoder structure based on an end-to-end deep convolutional neural network (CNN) is used to perform noise reduction on the second fused image. Figure 2 :

[0076] Input layer: The input image is the second fused image after IIR stage processing. This image has removed some temporal noise, reduced the overall noise level, and provided a cleaner input for CNN.

[0077] Convolution layer and pooling layer (Conv+ReLU+Pooling):

[0078] Convolution layer (Conv): extracts image features, performs local perception and feature extraction on the image through a series of convolution kernels (filters), and extracts local features of the image through a series of convolution operations. The size of the convolution kernel is usually 3x3 or 5x5;

[0079] Activation function (ReLU): introduces nonlinearity and increases the expressiveness of the model;

[0080] Pooling layer: Use the maximum pooling or average pooling layer for downsampling, reduce the size of the feature map through downsampling operations, retain the main features, and reduce the amount of calculation; obtain feature maps of different scales through multi-layer convolution and pooling operations so that they can be fused in the Decoder stage;

[0081] Multi-scale Encoder-Decoder structure:

[0082] Encoder part: contains multiple convolution and pooling operations, obtains features of different scales by downsampling layer by layer, and gradually extracts multi-scale features of the image through multiple convolution layers and pooling layers;

[0083] Decoder: Use upsampling or deconvolution combined with ordinary convolution layers, or restore the image size layer by layer by pixel rearrangement, and combine the features of skip-layer connection or residual structure for fusion, retain high-frequency details, avoid information loss and over-smoothing; this multi-scale structure can better handle noise at different scales and improve the noise reduction effect;

[0084] Upsample layer: Upsamples layer by layer to restore the image size, and combines the features extracted by the Encoder part for feature fusion;

[0085] Output layer (Output Frame[t]): The last layer of convolution operation maps the feature map back to the image space to generate the final denoised image;

[0086] Skip Connections or Residual Connections:

[0087] Skip Connections: Adding skip connections between the Encoder and the Decoder allows you to directly pass the features extracted in the Encoder stage to the corresponding Decoder stage, and stack the two feature maps in the channel direction.

[0088] Residual Blocks: Add two feature maps with the same number of channels.

[0089] Combining multiple motion estimation methods, the system's adaptability in different scenarios is enhanced. The residual structure alleviates the gradient vanishing problem in deep networks and improves the network training efficiency. The time domain filter coefficient is adaptively adjusted according to the motion intensity to ensure the noise reduction effect in moving and static areas. In a dark environment, the application of this video denoising method can significantly improve the clarity and quality of the video.

[0090] As a further specific embodiment, acquiring an input image of a video includes: extracting an input image containing noise from a video sequence received by a sensor.

[0091] As a further specific embodiment, obtaining the first fused image of the video includes: determining whether the input image is the first frame image, if so, setting the input image as the first fused image, if not, the first fused image is a fused image calculated by IIR using the previous frame image obtained by the above method, and the second fused image is a fused image calculated by IIR using the current frame image (input image) obtained by the above method.

[0092] As a further specific embodiment, it also includes: compressing the input image before acquiring the input image of the video and the first fused image; and performing convolution restoration on the denoised image after outputting the denoised image.

[0093] As a further specific embodiment, the input image is compressed to 1 / 4 or 1 / n of the original resolution, where n is an integer power of 2, to further reduce the running time and memory requirements.

[0094] The weights used by the network are obtained through pre-training, and there are no special restrictions on the data sets, hyperparameters, loss functions, etc. used in the training process.

[0095] As a further specific embodiment, while the second fused image is transmitted to the AI ​​denoising stage, the second fused image is temporarily retained until the next frame of image is inserted into the input, and the fused image output of the next frame is calculated, and the image correction of this stage is achieved through continuous iteration in the time domain.

[0096] Combination Figure 3 In some embodiments, a video noise reduction device is provided, comprising:

[0097] An acquisition unit 201 is used to acquire an input image and a first fused image of a video, wherein the input image is an original image acquired by an image sensor;

[0098] The time domain filtering unit 202 is used to perform time domain fusion of the input image and the first fused image through infinite impulse response time domain filtering to obtain a second fused image, including: performing motion estimation on the input image and the reference image to dynamically obtain a gain coefficient and a feedback coefficient of the infinite impulse response time domain filtering according to the input image and the reference image;

[0099] The algorithm filtering unit 203 is used to perform noise reduction processing on the second fused image through the AI ​​noise reduction network model to obtain a noise reduction image;

[0100] The output unit 204 is used to output the noise-reduced image.

[0101] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0102] In some other embodiments of the present invention, an electronic device 300 is disclosed. Figure 4As shown, the electronic device 300 may include: one or more integrated circuits 301; a memory 302; a display 303; one or more applications (not shown); and one or more computer programs 304. The above components may be connected via one or more communication buses 305. The one or more computer programs 304 are stored in the memory 302 and configured to be executed by the one or more processors 301. The one or more computer programs 304 include instructions. The instructions may be used to execute the following steps: Figure 1-Figure 4 And each step in the corresponding embodiment.

[0103] Specifically, the program containing the present method is processed by the integrated circuit 301, and the denoised original video output by the algorithm is sent to the ISP on the integrated circuit 301 for processing. The final video output by the ISP is transmitted to a terminal, which may be displayed on the display 303 or other display device, or not displayed.

[0104] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0105] Each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program codes.

[0107] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A video noise reduction method, characterized in that: include: Acquire an input image and a first fused image of a video, wherein the input image is an original image acquired by an image sensor; Performing time-domain fusion on the input image and the first fused image through infinite impulse response time-domain filtering to obtain a second fused image; Performing denoising processing on the second fused image through an AI denoising network model to obtain a denoised image; Output denoised image.

2. The video noise reduction method according to claim 1, characterized in that: Performing time-domain fusion on the input image and the first fused image through infinite impulse response time-domain filtering to obtain a second fused image includes: Motion estimation is performed on the input image and the reference image to dynamically obtain gain coefficients and feedback coefficients of time-domain filtering of an infinite impulse response according to the input image and the reference image.

3. The video noise reduction method according to claim 1, characterized in that: Performing time-domain fusion on the input image and the first fused image through infinite impulse response time-domain filtering to obtain a second fused image also includes: The input image and the first fused image are weighted added to obtain a second fused image.

4. The video noise reduction method according to claim 3, characterized in that: Performing weighted addition on the input image and the first fused image to obtain the second fused image includes: The differential equation of the time domain filter of the Q-order infinite impulse response is cascaded and decomposed into multiple first-order filter equations as follows: y j [t]=g j ·x[t]+p j ·y[t-1] Where j represents the index value of different filter equations, ranging from 1 to Q, x[t] represents the input image of the jth first-order filter equation at time t, and y j [t] represents the fused image output of the jth first-order filter equation at time t, y[t-1] represents the fused image output of the jth first-order filter equation at time t-1, g j It means that x[t] depends on y under the jth first-order filter equation j [t] Gain coefficient of influence degree, p j It means that y[t-1] is proportional to y under the jth first-order filter equation. j [t] Feedback coefficient of the impact degree.

5. The video noise reduction method according to claim 4, characterized in that: Motion estimation is performed on the input image and the reference image to dynamically obtain the infinite impulse response of the time domain filter according to the input image and the reference image. The gain coefficient and feedback coefficient include: Motion estimation is performed based on the input image and the first fused image to determine the motion area and intensity of each pixel.

6. The video noise reduction method according to claim 5, characterized in that: The coefficient g is determined based on the motion area and intensity of each pixel of the input image and the first fused image. j and coefficient p j include: Determine g by motion estimation j and p j , the formula is as follows: {g j ,p j }=MotionEst(x[t],y[t-1]) Among them, g j It means that x[t] depends on y under the jth first-order filter equation j [t] Gain coefficient of influence degree, p j It means that y[t-1] is proportional to y under the jth first-order filter equation. j [t] is the feedback coefficient of the influence degree, x[t] represents the input image of the j-th first-order form of the filter equation at time t, y[t-1] represents the fused image output of the j-th first-order form of the filter equation at time t-1, and MotionEst represents the motion estimation algorithm.

7. The video noise reduction method according to claim 5, characterized in that: Performing motion estimation based on the input image and the first fused image to determine the motion area and intensity of each pixel includes: The motion region and intensity of each pixel are determined according to at least one of the following methods: inter-frame difference method, background subtraction method, optical flow method, connected region analysis, spectral analysis, and multimodal fusion.

8. The video noise reduction method according to claim 1, characterized in that: The second fused image is subjected to denoising processing through the AI ​​denoising network model to obtain a denoised image including: The second fused image is denoised using a multi-scale Encoder-Decoder structure based on a convolutional neural network.

9. The video noise reduction method according to claim 8, characterized in that: The multi-scale Encoder-Decoder structure based on convolutional neural network includes: Through the Encoder structure, multiple convolutional layers and pooling layers, multi-scale features are extracted; Transfer features between different levels through skip connections or residual structures; Through the Decoder structure, upsampling or deconvolution is used in combination with ordinary convolutional layers, or the image size is restored layer by layer by pixel rearrangement, and the features of skip-layer connection or residual structure are combined for fusion.

10. The video noise reduction method according to claim 9, characterized in that: Fusion combining features of skip layer connections or residual structures includes: Add a skip-layer connection between the Encoder and the Decoder to pass the features extracted in the Encoder stage to the corresponding Decoder stage.

11. The video noise reduction method according to claim 9, characterized in that: Fusion combining features of skip layer connections or residual structures includes: The residual block is introduced, and the residual structure performs feature fusion by adding two feature maps with the same number of channels.

12. The video noise reduction method according to claim 1, characterized in that: Also includes: The input image is compressed before obtaining the input image of the video and the first fused image; After outputting the denoised image, the denoised image is restored by convolution.

13. A video noise reduction device, characterized in that: include: An acquisition unit, configured to acquire an input image of a video and a first fused image, wherein the input image is an original image acquired by an image sensor; A time domain filtering unit, used for performing time domain fusion on the input image and the first fused image through infinite impulse response time domain filtering to obtain a second fused image; An algorithm filtering unit, used for performing noise reduction processing on the second fused image through an AI noise reduction network model to obtain a noise reduction image; The output unit is used to output the denoised image.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the computer program is executed by the processor, the at least one processor is able to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.