Lightweight video denoising method, model training method and system based on frequency domain learning

Abstract

The invention discloses a lightweight video denoising method based on frequency domain learning and a model training method and system, and belongs to the technical field of video processing. The method comprises the steps that a clear video set and a corresponding noise video set are obtained, the noise video set serves as input of a lightweight video denoising model, the clear video set serves as a label of the clear video set, and the noise video set serves as the label of the lightweight video denoising model; the lightweight video denoising model is trained; acquiring features of each video frame in the noise video set in the feature domain, calculating fusion weights of amplitude components and phase components corresponding to the features of each video frame in the frequency domain, then fusing the amplitudes and phases of all the video frames, and converting a fusion result to the feature domain to obtain a fused video frame; and cascading the fused video frame and the corresponding variance map, inputting the cascaded fused video frame and the corresponding variance map into an additive Transformer structure to obtain a denoised video frame feature, and converting the denoised video frame feature to a pixel domain to obtain a denoised video frame, the method significantly improves the calculation efficiency, improves the video denoising quality, can be applied to a computing power limited device, and has a wide application prospect. And the performance of subsequent tasks such as video segmentation, target detection and target identification is ensured.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to video denoising, and specifically to a lightweight video denoising method, a model training method and a system based on frequency domain learning. Background Art

[0002] In the era of mobile Internet, high-quality video is an important medium for information transmission and is widely used in various fields such as national defense security, medical imaging, industrial monitoring, audio and video entertainment, etc. However, due to the hardware limitations of camera sensors and the influence of external environmental factors (such as temperature, light, etc.), the imaged video usually contains a certain noise component, which destroys the texture details of the image or video frame and may lead to the performance degradation of subsequent high-level image and video processing tasks (such as segmentation, detection, classification, etc.). Therefore, high-performance and high-efficiency video denoising technology has broad application prospects.

[0003] Denoising is one of the fundamental ill-posed inverse problems in the field of image and video processing. Existing denoising methods can be divided into model-based methods and deep learning-based methods. In the early stage of denoising technology, model-based denoising methods were mainly used. Model-based image denoising methods construct data items and prior items (such as non-local means, sparse characteristics, Gaussian scale mixtures, etc.) according to the Bayesian inference framework to obtain the objective function, and obtain the restored result by iteratively solving the objective function. Compared with image denoising, the key to video denoising is to make full use of the time domain information in adjacent frames. Therefore, model-based video denoising methods are generally based on image denoising technology and integrated with time domain correlation. However, since the calculation process requires multiple iterations to converge, and a single manual prior cannot fully describe the prior knowledge, model-based video denoising methods have the problems of long inference time and inaccurate manual prior modeling.

[0004] With the vigorous development of hardware resources, in recent years, deep learning-based video denoising methods have achieved significant enhancement of denoising performance in a data-driven manner by learning the mapping relationship between degraded data and annotated data, and have gradually become the mainstream denoising algorithm. The core of the neural network-based video denoising method is to build a network structure that can fully learn the temporal and spatial correlation of the video. It can be divided into the following two categories: deep convolutional network denoising methods based on residuals and dense connections and video denoising methods based on recurrent convolutional networks (RNNs). The deep convolutional network denoising method based on residuals and dense connections is an extension of the image denoising method. It mainly includes two modules: time domain alignment and spatial domain denoising. Video denoising is achieved by designing effective time domain alignment and spatial domain denoising sub-networks. The RNN-based video denoising method utilizes the strong time series modeling ability of RNN to effectively capture the temporal correlation of adjacent frames of the video and finally obtain the restored video frame. The existing technology regards the neural network as a black box, and improves the denoising performance by continuously deepening and widening the network structure, ignoring the exponential increase in the amount of calculation. Therefore, the existing deep learning-based video denoising methods usually have the limitation of large amount of calculation and difficulty in practical application. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a lightweight video denoising method, a model training method and a system based on frequency domain learning, which can reduce the computational complexity of the denoising method and enhance the restoration quality of each video frame in the video sequence.

[0006] The present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present application provides a lightweight video denoising model training method based on frequency domain learning, comprising:

[0008] Obtain a clear video set and a corresponding noisy video set, wherein the noisy video set is obtained by adding noise to the clear video set;

[0009] The noisy video set is used as the input of the lightweight video denoising model, and the clear video set is used as the label of the clear video set to train the lightweight video denoising model;

[0010] During training, the features of each video frame in the noise video set are obtained in the feature domain, and the fusion weights of its amplitude component and phase component are calculated in the frequency domain. Then, the amplitude and phase of all video frames are fused, and the fusion results are converted to the feature domain to obtain the fused video frame.

[0011] The fused video frame and the corresponding variance map are cascaded and input into the additive Transformer structure to obtain the denoised video frame features, and the denoised video frame features are converted to the pixel domain to obtain the denoised video frame.

[0012] Preferably, the clear video set is added with noise, comprising:

[0013] Gaussian distribution is used as the noise model, and Gaussian noise is added to the clear video set to obtain the noisy video set.

[0014] Preferably, the fusion weights of the amplitude component and the phase component corresponding to the video frame feature include:

[0015] The amplitude and phase of the video frame are obtained according to its features, the amplitude components and phase components of all video frames are cascaded and input into the convolutional network respectively, and the fusion weight of the amplitude component and phase component of each video frame is calculated.

[0016] Preferably, obtaining the fused video frame includes:

[0017] The point multiplication operation is used to aggregate the fusion weights of the amplitude and phase of all video frames respectively. According to the fused amplitude and phase, the fusion result is estimated. Finally, the fusion result is converted to the feature domain through inverse Fourier transform to obtain the fused video frame.

[0018] Preferably, Fourier transform is used to transform the video frame features into the frequency domain; and inverse Fourier transform is used to transform the fusion result into the feature domain.

[0019] Preferably, the additive Transformer module includes a normalization layer, an attention layer, a residual, a normalization layer, a feedforward layer and a residual.

[0020] In a second aspect, the present application provides a lightweight video denoising model training system based on frequency domain learning, characterized in that it includes:

[0021] A first acquisition module is used to obtain a clear video set and a corresponding noisy video set, wherein the noisy video set is obtained by adding noise to the clear video set;

[0022] A training module, used for training the lightweight video denoising model by using the noisy video set as an input of the lightweight video denoising model and the clear video set as a label of the clear video set;

[0023] During training, the features of each video frame in the noise video set are obtained in the feature domain, and the fusion weights of the amplitude component and phase component corresponding to the features of each video frame are calculated in the frequency domain. Then, the amplitude and phase of all video frames are fused, and the fusion results are converted to the feature domain to obtain the fused video frame.

[0024] The fused video frame and the corresponding variance map are cascaded and input into the additive Transformer structure to obtain the denoised video frame features, and the denoised video frame features are converted to the pixel domain to obtain the denoised video frame.

[0025] In a third aspect, the present application provides a lightweight video denoising method based on frequency domain learning, characterized in that it includes:

[0026] Step 100: Obtain a noisy video set;

[0027] Step 200: input the acquired noisy video set into the lightweight video denoising model trained according to any one of claims 1 to 6 to obtain a denoised video set.

[0028] In a fourth aspect, the present application provides a lightweight video denoising system based on frequency domain learning, characterized in that it includes:

[0029] The second acquisition module obtains a noise video set;

[0030] A denoising module is used to input the acquired noisy video set into the lightweight video denoising model trained by any one of claims 1-6 to obtain a denoised video set.

[0031] In a fifth aspect, the present application provides an electronic device, characterized in that it includes:

[0032] Memory for storing computer programs;

[0033] A processor, used to implement the steps of the lightweight video denoising model training method based on frequency domain learning as described in any one of claims 1-6 when executing the computer program.

[0034] Compared with the prior art, the present invention has the following beneficial technical effects:

[0035] The lightweight video denoising method based on frequency domain learning applied by the present invention makes full use of time domain information, which is the key to improving the video denoising performance. The existing methods usually use complex motion estimation and motion compensation methods or group fusion methods, resulting in large computational complexity and limited denoising ability of the video denoising algorithm. According to the characteristics of the global receptive field in the frequency domain, the present invention realizes the time domain fusion of adjacent video frames in the frequency domain, which effectively utilizes the time domain information while greatly reducing the computational complexity and improving the efficiency of the video denoising algorithm; secondly, the existing spatial domain denoising method usually does not process the edge texture features sufficiently, resulting in the edges of the denoised video frames being too smooth and lacking texture details. The improved Transformer structure proposed in the present invention has excellent global context modeling capabilities and can properly restore the texture details of the video frames. In addition, the additive attention mechanism proposed in the present invention greatly reduces the memory consumption of the algorithm and further enhances the denoising efficiency; different from the existing video denoising methods, the time domain fusion module based on frequency domain transformation and the spatial domain denoising structure based on the additive Transformer structure proposed in the present invention greatly save the computational amount, effectively improve the speed and performance of the video denoising method, and can be applied to scenes with limited computing power to efficiently remove video noise.

[0036] The present application also proposes a lightweight video denoising system based on frequency domain learning, an electronic device and a computer storage medium, which have all the advantages of the above-mentioned lightweight video denoising method based on frequency domain learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a flow chart of a lightweight video denoising method based on frequency domain learning of the present invention;

[0039] Figure 2 It is a structural schematic diagram of the time domain fusion module of the present invention;

[0040] Figure 3 It is a structural schematic diagram of the spatial domain denoising module of the present invention;

[0041] Figure 4 This is a comparison diagram of the video frame before and after denoising of the present invention;

[0042] Among them, Figure a is the original video frame, and Figure b is the clear video frame after denoising;

[0043] Figure 5It is a comparison diagram of video frames processed by the denoising method of the present invention and the FastDvDnet recovery method.

[0044] Among them, Figure a is a video frame denoised by FastDvDnet, and Figure b is a video frame denoised by the denoising method of this application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0046] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0047] Compared with model-based video denoising methods, deep learning-based video denoising methods have achieved significant performance improvements, but most deep learning-based video denoising methods still rely on constructing complex network structures to achieve significant performance improvements, ignoring the exponential increase in computational complexity, making existing algorithms difficult to apply to scenarios with limited computing power, limiting their application in practical scenarios. In order to break through the bottleneck of large computational complexity of existing methods.

[0048] The principle of this lightweight video denoising method based on frequency domain learning is to improve the time domain fusion efficiency by using the global view of the frequency domain. Specifically, by respectively predicting the fusion weights of the amplitude and phase of all pixels of each video frame in the frequency domain, full use of the time domain information is achieved; then, the present invention uses the additive Transformer structure as a spatial denoiser to remove noise while restoring the texture details of the video frame, thereby reducing the amount of calculation; finally, the present invention uses the noisy video frame and the clear video frame as the input samples and label samples of the deep network model, respectively, trains the neural network model, and achieves high-quality restoration of the video frame, effectively improving the efficiency of video denoising.

[0049] A lightweight video denoising model training method based on frequency domain learning includes the following steps:

[0050] Step 1: Obtain a clear video set and a corresponding noisy video set, wherein the noisy video set is obtained by adding Gaussian noise to the clear video set;

[0051] It should be noted that when training the lightweight video denoising model, a clear video set and a corresponding noisy video set are required. When actually using the lightweight video denoising model for prediction, the input is a video set containing noise, and then the corresponding clear video set is obtained, that is, the prediction result of the lightweight video denoising model. During training, the prediction result is known.

[0052] Step 2: Using the noisy video set as the input of the lightweight video denoising model and the clear video set as the label of the clear video set, the lightweight video denoising model is trained;

[0053] During training, the features of each video frame in the noise video set are obtained in the feature domain, and the fusion weights of the amplitude component and phase component corresponding to the features of each video frame are calculated in the frequency domain. Then, the amplitude and phase of all video frames are fused, and the fusion results are converted to the feature domain to obtain the fused video frame.

[0054] The fused video frame and the corresponding variance map are cascaded and input into the additive Transformer structure to obtain the denoised video frame features, and the denoised video frame features are converted to the pixel domain to obtain the denoised video frame.

[0055] During the training process, this method fully utilizes the time domain information by predicting the fusion weights of the amplitude and phase of all pixels in each video frame in the frequency domain. This fusion method can more accurately capture the time domain correlation between video frames, which helps to improve the denoising effect and the quality of video frame restoration. Secondly, the additive Transformer structure, as a spatial denoiser, can not only effectively remove noise, but also restore the texture details of the video frame. This makes the denoised video frame clearer and more natural, improving the visual experience.

[0056] This method cascades the fused video frames and the corresponding variance maps and inputs them into the additive Transformer structure. The variance map provides information about the spatial variability of the video frames, which helps the additive Transformer structure to better understand and process the noise and details in the input data, further improving the denoising effect and detail recovery ability. This method improves the efficiency of time domain fusion by utilizing the global view of the frequency domain, and uses the additive Transformer structure as a spatial denoiser, effectively reducing the amount of computation. This enables the method to be more suitable for practical scenarios with limited computing power while maintaining high performance. It solves the problem that traditional deep learning-based video denoising methods rely on constructing complex network structures to achieve performance improvement, resulting in an exponential increase in the amount of computation; the lightweight video denoising method based on frequency domain learning has many advantages, such as high computational efficiency, full use of time domain information, strong texture detail recovery ability, clear and effective training process, and variance maps providing additional information. These advantages make this method have broad application prospects and practical value in the field of video denoising.

[0057] Example 1

[0058] See also Figure 1 , a lightweight video denoising method based on frequency domain learning, comprising the following steps:

[0059] Step 10. Get a clear video set and noise video collection

[0060] For a given N clear video sequences, store their backup as the clear label set Each sequence contains (2T+1) video frames, so any clear video sequence is represented by X i ={x -T ,...,x0,...,x T}; Add Gaussian noise to the clean video dataset to get the noisy video dataset Any noisy video sequence is represented by Y i ={y -T ,...,y0,...,y T}.

[0061] For example, let 2T+1=5, that is, take 5 consecutive video frames as a sequence, select the DAVIS training data set containing 86 complete videos, each video can be decomposed into video sequences of different lengths, which are limited to contain 85 video frames at most, and every 5 video frames are taken as a video sequence, and these clear video sequences are used as the clear video training set in this embodiment. The resolution of each video frame is 854×480.

[0062] In addition, Gaussian distribution is selected as the noise model, and noise is added to the clear video dataset to obtain the noisy video dataset. The range of the comparison noise standard deviation is [5,55]. The Gaussian noise video set is constructed as follows:

[0063] y=x+N(0,σ 2 )

[0064] Step 20: Build a deep neural network model, which mainly includes a time domain fusion module and a spatial domain denoising module. and noise video collection ) Train a lightweight video denoising model. Given a clear data set of N video sequences, store the backup data as a clear label set. Add simulated Gaussian noise to the clear video data set to obtain the noisy video set The deep neural network model constructed by corresponding set training is used to obtain the trained lightweight video denoising. The training method is as follows:

[0065] S20.1. Since the feature domain can capture more data characteristics, we first use 3 layers of convolution with a kernel of 3x3 to transform each input noisy video frame into the feature domain.

[0066] S20.2, time domain fusion module, converts all video frame features to the frequency domain through Fourier transform, extracts the amplitude and phase of each video frame feature respectively, concatenates the amplitude components and phase components of all video frames respectively and inputs them into a small network composed of 3 layers of convolution, calculates the fusion weights of the amplitude component and phase component of each video frame, and then aggregates the amplitude and phase of all video frames respectively through dot multiplication operation, estimates the fused result, and finally converts the fusion result to the feature domain through inverse Fourier transform.

[0067] It should be noted that in the frequency domain, the characteristics of each video frame can be expressed as two parts: amplitude and phase. The amplitude represents the strength or size of the signal, while the phase represents the phase angle or time delay of the signal.

[0068] The amplitude components and phase components of all video frames are cascaded respectively. Cascading is an operation of connecting multiple data sequences together, where the amplitude or phase components of all video frames in the frequency domain are arranged in a certain order into a longer sequence.

[0069] The cascaded amplitude component and phase component are respectively input into a small network consisting of 3 layers of convolution. The role of this small network is to learn how to calculate the fusion weight of each video frame based on its amplitude and phase information in the frequency domain. The fusion weight indicates the contribution of each video frame to the final fusion result. By calculating the fusion weight, the effective utilization and integration of different video frame information can be achieved.

[0070] Using the calculated fusion weights, the amplitudes and phases of all video frames are aggregated separately through dot multiplication operations. Dot multiplication is an element-by-element multiplication operation, where the amplitude (or phase) of each video frame is multiplied by the corresponding fusion weight, and then all the results are added together to obtain the fused amplitude (or phase).

[0071] Finally, the fused result is estimated based on the fused amplitude and phase. This result is a representation in the frequency domain that incorporates the information of all video frames and takes into account the contribution of each video frame.

[0072] The fused result is converted back to the feature domain through inverse Fourier transform. Inverse Fourier transform is the inverse process of Fourier transform, which can convert the signal in the frequency domain back to the time domain or feature domain.

[0073] The time domain fusion module can effectively utilize the global view of the frequency domain and improve the fusion efficiency of the time domain information. This method fully utilizes the time domain information by predicting the fusion weights of the amplitude and phase of each video frame in the frequency domain, providing high-quality fused video frame features for the subsequent spatial denoising module.

[0074] S20.3, Spatial denoising module: The fused video frame and variance map are cascaded and input into an additive Transformer structure. The entire module consists of a downsampling encoder and an upsampling decoder to capture multi-scale feature information. Each encoder or decoder contains a Transformer module, including a normalization layer, an attention layer, a residual, a normalization layer, a feedforward layer, and a residual. In order to improve the efficiency of spatial denoising, an additive attention layer is used instead of the traditional multiplicative attention layer, which greatly improves the computational efficiency.

[0075] It should be noted that the fused video frame is obtained by fusing the features of multiple video frames in the frequency domain and then converting them back to the time domain through inverse Fourier transform. It contains information from multiple video frames and is the basis for subsequent denoising. The fused video frame is used as the input of the denoising module, providing a preliminary estimate of the video frame, on which the additive Transformer structure will perform further denoising.

[0076] The variance map represents the noise level or uncertainty of each video frame or pixel. In the video denoising task, the variance map can reflect the distribution and intensity of noise in each video frame. The variance map provides the denoising module with additional information about the noise. By combining the fused video frame and the variance map, the denoising module can identify and process the noise more accurately, thereby improving the denoising effect.

[0077] The fused video frame and variance map are cascaded and input into the additive Transformer structure, so that the denoising module can consider both video content and noise information at the same time. This cascade input method helps the additive Transformer structure better capture and utilize the relationship between the two, thereby improving denoising efficiency.

[0078] S20.4. Finally, the denoised video frame features are transformed from the feature domain to the pixel domain through two convolutional layers to obtain the restored video frame.

[0079] Step 30: Input a noisy video sequence into the trained lightweight video denoising model, and the output of the lightweight video denoising model is the denoised video frame.

[0080] The following simulation experiments are used to specifically illustrate the technical effects of the present invention:

[0081] 1. Simulation conditions:

[0082] 1) The Gaussian noise matrix in the simulation experiment is obtained by generating a matrix that obeys the Gaussian distribution;

[0083] 2) The programming platform used in the simulation experiment is JetBrains PyCharm;

[0084] 3) The structure of the time domain fusion module constructed in the simulation experiment is as follows Figure 2 As shown; the structure of the spatial denoising module is as follows Figure 3 As shown;

[0085] 4) The noisy video frames used in the simulation experiment are as follows Figure 4 As shown in (a), the original clear video frame is Figure 4 (b)

[0086] 5) In the simulation experiment, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators are used to evaluate the denoising results. The peak signal-to-noise ratio (PSNR) is defined as:

[0087]

[0088] Among them, MSE represents the mean square error of the denoised image, μ x , μ y are the means of x and y respectively, are the variances of x and y, σ xy is the covariance of x and y, c1 and c2 are two constants.

[0089] 6) In the simulation experiment, Giga Floating-point Operations Per Second (GFLOPs) is used to evaluate the computational complexity of different video denoising networks.

[0090] Simulation content: The method of the present invention and the existing fast video denoising method FastDvDnet are used to denoise the video sequence containing Gaussian noise. In order to achieve a fair comparison, the noise standard deviation of the tested Gaussian noise is set to σ=30, and the number of input video frames for both methods is set to 5 frames. The denoising result of a video frame in the video "cats-car" in the DAVIS validation set is as follows: Figure 5 As shown;

[0091] The video frames restored by the fast video denoising method FastDvDnet are as follows Figure 5 As shown in (a), the denoised video frame obtained by the present invention is as follows Figure 5 As shown in (b), it can be seen that the video frame after denoising by the method of the present invention has clearer texture details and higher denoising quality.

[0092] 2. Peak signal-to-noise ratio PSNR comparison

[0093] Using the DAVIS validation set containing 30 videos as test data, the peak signal-to-noise ratio (PSNR) of the restored video frames by the existing fast video denoising method FastDvDnet and the method of the present invention are calculated respectively, and the results are shown in Table 1.

[0094] Table 1 Comparison of the average results of peak signal-to-noise ratio (PSNR) (dB) of FastDvDnet and the method of the present invention for removing Gaussian noise (σ=10, 20, 30, 40, 50)

[0095] σ 10 20 30 40 50 average FastDvDnet 38.96 35.72 33.87 32.58 31.59 34.54 Method of the present invention 39.14 35.92 34.08 32.80 31.82 34.75

[0096] It can be seen from the experimental results in Table 1 that the average peak signal-to-noise ratio (PSNR) of the denoised image by the method of the present invention is 0.21 dB higher than that of FastDvDnet, and the method of the present invention improves the accuracy of video denoising.

[0097] 3. Comparison of GFLOPs of Video Denoising Network

[0098] The GFLOPs of the existing fast video denoising network FastDvDnet and the video denoising network proposed by the method of the present invention are tested respectively. The video used is a video sequence containing 5 video frames and a resolution of 854×480. The GFLOPs comparison is shown in Table 2.

[0099] Table 2 Comparison of GLOPs between FastDvDnet and the network of the present invention

[0100]

[0101] It can be seen from Table 2 that, compared with the fast video denoising method FastDvDnet, the method of the present invention greatly reduces the computational complexity of the video denoising network.

[0102] Based on the above-mentioned lightweight video denoising model training method based on frequency domain learning, the present application also proposes a lightweight video denoising model training system based on frequency domain learning, which may include:

[0103] An acquisition module, used for acquiring a clear video set and a corresponding noisy video set, wherein the noisy video set is obtained by adding Gaussian noise to the clear video set;

[0104] A training module, used for training the lightweight video denoising model by using the noisy video set as an input of the lightweight video denoising model and the clear video set as a label of the clear video set;

[0105] During training, the features of each video frame in the noise video set are obtained in the feature domain, and the fusion weights of the amplitude component and phase component corresponding to the features of each video frame are calculated in the frequency domain. Then, the amplitude and phase of all video frames are fused, and the fusion results are converted to the feature domain to obtain the fused video frame.

[0106] The fused video frame and the corresponding variance map are cascaded and input into the additive Transformer structure to obtain the denoised video frame features, and the denoised video frame features are converted to the pixel domain to obtain the denoised video frame.

[0107] This application also proposes a lightweight video denoising method based on frequency domain learning, which may include:

[0108] Step 100: Obtain a noisy video set;

[0109] Step 200: input the acquired noisy video set into a trained lightweight video denoising model to obtain a denoised video set.

[0110] Correspondingly, the present application also proposes a lightweight video denoising method based on frequency domain learning, which may include:

[0111] An acquisition module, used to obtain a noisy video set;

[0112] The denoising module is used to input the acquired noisy video set into the trained lightweight video denoising model to obtain the denoised video set.

[0113] This application provides a lightweight video denoising method, which mainly solves the problem that existing deep learning-based video denoising methods have large computational complexity and are difficult to apply to scenarios with limited computing power. The key technology lies in building a time domain fusion network based on frequency domain transformation and a spatial domain denoising network based on additive Transformer. By making full use of adjacent frame information and fine denoising in the spatial domain, the efficiency and quality of video denoising can be improved, including: obtaining paired training data (clear video set and noise video collection ), construct a deep neural network structure (mainly including a time domain fusion network and a spatial domain denoising network), train a deep neural network model based on paired data, and input the noisy video sequence into the trained video denoising network model to calculate the denoised video frame. The present invention significantly improves the computational efficiency and the quality of video denoising, and can be applied to devices with limited computing power to ensure the performance of subsequent tasks such as video segmentation, target detection, and target recognition.

[0114] It should be noted that in the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each module is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another device, or some features can be ignored or not executed. The module described as a separate component may or may not be physically separated. The component displayed as a module may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0115] In addition, each module in each embodiment of the present invention may be integrated into a processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] An electronic device provided in an embodiment of the present application includes a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps of the lightweight video denoising method based on frequency domain learning described in any of the above embodiments are implemented.

[0117] Another electronic device provided in the embodiment of the present application may also include: an input port connected to the processor, used to transmit multimodal data collected by an external acquisition device to the processor; and a display unit connected to the processor, used to display the processing results of the processor to the outside world; a communication module connected to the processor, used to realize communication between the electronic device and the outside world. The display unit can be a display panel, a laser scanning display, etc.; the communication mode adopted by the communication module includes but is not limited to mobile high-definition link technology (HML), universal serial bus (USB), high-definition multimedia interface (HDMI), wireless connection (including wireless fidelity technology (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s).

[0118] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the lightweight video denoising method based on frequency domain learning as described in any of the above embodiments are implemented.

[0119] For the description of the relevant parts of the lightweight video denoising system based on frequency domain learning, the electronic device and the computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in the lightweight video denoising method based on frequency domain learning provided in the embodiments of the present application, which will not be repeated here. In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0120] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A lightweight video denoising model training method based on frequency domain learning, characterized in that: include: Obtain a clear video set and a corresponding noisy video set, wherein the noisy video set is obtained by adding noise to the clear video set; The noisy video set is used as the input of the lightweight video denoising model, and the clear video set is used as the label of the clear video set to train the lightweight video denoising model; During training, the features of each video frame in the noise video set are obtained in the feature domain, and the fusion weights of its amplitude component and phase component are calculated in the frequency domain. Then, the amplitude and phase of all video frames are fused, and the fusion results are converted to the feature domain to obtain the fused video frame. The fused video frame and the corresponding variance map are cascaded and input into the additive Transformer structure to obtain the denoised video frame features, and the denoised video frame features are converted to the pixel domain to obtain the denoised video frame.

2. According to the lightweight video denoising model training method based on frequency domain learning in claim 1, it is characterized in that: The clear video set adds noise, including: Gaussian distribution is used as the noise model, and Gaussian noise is added to the clear video set to obtain the noisy video set.

3. According to the lightweight video denoising model training method based on frequency domain learning in claim 1, it is characterized in that: The fusion weights of the amplitude component and the phase component corresponding to the video frame feature include: The amplitude and phase of the video frame are obtained according to its features, the amplitude components and phase components of all video frames are cascaded and input into the convolutional network respectively, and the fusion weight of the amplitude component and phase component of each video frame is calculated.

4. According to the lightweight video denoising model training method based on frequency domain learning in claim 1, it is characterized in that: The obtaining of the fused video frame includes: The point multiplication operation is used to aggregate the fusion weights of the amplitude and phase of all video frames respectively. According to the fused amplitude and phase, the fusion result is estimated. Finally, the fusion result is converted to the feature domain through inverse Fourier transform to obtain the fused video frame.

5. According to the lightweight video denoising model training method based on frequency domain learning in claim 1, it is characterized in that: Fourier transform is used to transform the video frame features into the frequency domain; inverse Fourier transform is used to transform the fusion result into the feature domain.

6. The method for training a lightweight video denoising model based on frequency domain learning according to claim 1, characterized in that: The additive Transformer module includes a normalization layer, an attention layer, a residual, a normalization layer, a feedforward layer, and a residual.

7. A lightweight video denoising model training system based on frequency domain learning, characterized in that: include: A first acquisition module is used to obtain a clear video set and a corresponding noisy video set, wherein the noisy video set is obtained by adding noise to the clear video set; A training module, used for training the lightweight video denoising model by using the noisy video set as an input of the lightweight video denoising model and the clear video set as a label of the clear video set; During training, the features of each video frame in the noise video set are obtained in the feature domain, and the fusion weights of the amplitude component and phase component corresponding to the features of each video frame are calculated in the frequency domain. Then, the amplitude and phase of all video frames are fused, and the fusion results are converted to the feature domain to obtain the fused video frame. The fused video frame and the corresponding variance map are cascaded and input into the additive Transformer structure to obtain the denoised video frame features, and the denoised video frame features are converted to the pixel domain to obtain the denoised video frame.

8. A lightweight video denoising method based on frequency domain learning, characterized in that: include: Step 100: Obtain a noisy video set; Step 200: input the acquired noisy video set into the lightweight video denoising model trained according to any one of claims 1 to 6 to obtain a denoised video set.

9. A lightweight video denoising system based on frequency domain learning, characterized in that: include: The second acquisition module obtains a noise video set; A denoising module is used to input the acquired noisy video set into the lightweight video denoising model trained by any one of claims 1-6 to obtain a denoised video set.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the lightweight video denoising model training method based on frequency domain learning as described in any one of claims 1-6 when executing the computer program.

Images