Two-stage video denoising model training method, video denoising method and system

Abstract

The present application provides a two-stage video denoising model training method, a video denoising method and a system, the method comprising: using the noise-free video data of each noise video sample and each image frame group to train the residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frame in each image frame group in the first stage to obtain the first stage denoising result, and then extracts the static features of each main image frame based on each first stage denoising result in the second stage to obtain the second stage denoising result, thereby generating a two-stage video denoising model for video denoising. The present application can extract dynamic features and static features from two stages for different characteristics of the two dimensions of time and space, can effectively improve the performance of the trained video denoising model, and can effectively improve the robustness, significance and reliability of the video denoising result output by the two-stage video denoising model.

Description

Technical Field

[0001] The present application relates to the field of image and video information processing technology, and in particular to a dual-stage video denoising model training method, a video denoising method and a system. Background Art

[0002] As a widely used medium, video contains more useful information than text and images. It not only has the spatial information of a single image, but also the temporal information between frames. However, in specific applications in production and life, a series of video processing processes such as video acquisition, video compression and video encoding may be affected by environmental factors and equipment conditions, which may bring a lot of noise, such as low light illumination, inconsistent image sensor size and other factors, resulting in a decrease in video quality. Therefore, video denoising, as a basic work in computer vision, aims to maintain the basic visual content of the original video while removing interference noise. At the same time, the results of video denoising can greatly affect the recognition and understanding of video data in subsequent downstream tasks, such as old movie restoration, video surveillance, etc., which has attracted a large number of scholars to study it.

[0003] At present, the existing video denoising methods are mainly divided into three categories, including traditional methods based on image blocks (patch), explicit motion compensation methods and non-explicit motion compensation methods. However, the training process adopted by the existing video denoising methods, which simply superimposes the spatial and temporal dimension information, will cause the trained video denoising model to have poor denoising effect. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a two-stage video denoising model training method, a video denoising method and a system to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present application provides a two-stage video denoising model training method, comprising:

[0006] Acquire a plurality of image frame groups corresponding to each noise video sample containing noise data, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame;

[0007] The preset residual self-attention model is trained using the noise-free video data corresponding to each of the pre-acquired noisy video samples and each of the image frame groups, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames based on the first stage denoising results of each of the main image frames in the second stage to obtain the second stage denoising results corresponding to each of the main image frames, thereby generating a two-stage video denoising model for video denoising.

[0008] In some embodiments of the present application, the residual self-attention model includes:

[0009] The first-stage network is used to extract the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, and perform feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups to obtain the dynamic features of each of the main image frames; and then perform pixel reorganization on the dynamic features of each of the main image frames to obtain the first-stage denoising results corresponding to each of the main image frames;

[0010] The second stage network is connected to the output end of the first stage network, and is used to respectively extract the static features of the main image frames corresponding to each of the first stage denoising results, and then downsample the static features of each of the main image frames to obtain the second stage denoising results corresponding to each of the main image frames.

[0011] In some embodiments of the present application, the first-stage network includes:

[0012] A first encoder module, configured to extract global dynamic features of the previous image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism;

[0013] A second encoder module, configured to extract global dynamic features of the main image frames in each of the image frame groups by using a convolutional neural network and a self-attention mechanism;

[0014] A third encoder module, configured to extract global dynamic features of the latter image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism;

[0015] a feature fusion module, connected to the output ends of the first encoder module, the second encoder module and the third encoder module respectively, and used for performing feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, so as to obtain the dynamic features of each of the main image frames;

[0016] The first decoder module is connected to the output end of the feature fusion module and is also connected to the second encoder module using a residual connection. The first decoder module is used to use a convolutional neural network to perform pixel reorganization on the dynamic features of each of the main image frames to obtain a first-stage denoising result corresponding to each of the main image frames.

[0017] In some embodiments of the present application, the second-stage network includes:

[0018] a fourth encoder module, connected to the output end of the first-stage network, for respectively extracting static features of the main image frames corresponding to the first-stage denoising results by using a convolutional neural network and a self-attention mechanism;

[0019] The second decoder module adopts a residual connection with the fourth encoder module. The second decoder module is used to downsample the static features of each of the main image frames using a convolutional neural network to obtain a second-stage denoising result corresponding to each of the main image frames.

[0020] In some embodiments of the present application, the first encoder module, the second encoder module, the third encoder module, and the fourth encoder module each include: a feature extraction model and two downsampling models connected in sequence;

[0021] The feature extraction model includes a convolutional neural network for extracting initial features of the image frame;

[0022] The downsampling model includes: a downsampling layer, a convolutional neural network and a self-attention module connected in sequence, which is used to perform downsampling processing based on the initial features of the image frame and extract the global dynamic features of the image frame, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

[0023] In some embodiments of the present application, the first decoder module and the second decoder module each include: two upsampling models and a feature extraction model connected in sequence;

[0024] The upsampling model includes: a convolutional neural network and an upsampling layer connected to each other, used for upsampling the dynamic features of each of the main image frames based on a pixel reorganization strategy;

[0025] The feature extraction model includes a convolutional neural network, which is used to obtain denoising results corresponding to each of the main image frames based on the dynamic features of each of the main image frames after downsampling processing, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

[0026] Another aspect of the present application is a video denoising method, comprising:

[0027] Acquire multiple image frame groups corresponding to the target video, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame;

[0028] The multiple image frame groups corresponding to the target video are input into a two-stage video denoising model pre-trained based on the two-stage video denoising model training method, so that the two-stage video denoising model outputs the second stage denoising result corresponding to the target video.

[0029] The third aspect of the present application provides a two-stage video denoising model training system, comprising:

[0030] An image frame extraction module, used for acquiring a plurality of image frame groups corresponding to each noise video sample containing noise data, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame which are sequentially adjacent to the main image frame;

[0031] A two-stage model training module is used to use the noise-free video data corresponding to each of the pre-acquired noisy video samples and each of the image frame groups to perform model training on a preset residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first-stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames based on the first-stage denoising results of each of the main image frames in the second stage to obtain the second-stage denoising results corresponding to each of the main image frames, thereby generating a two-stage video denoising model for video denoising.

[0032] A fourth aspect of the present application provides a video denoising system, comprising:

[0033] An image frame acquisition module, used for acquiring a plurality of image frame groups corresponding to a target video, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame;

[0034] The two-stage model denoising module is used to input multiple image frame groups corresponding to the target video into a two-stage video denoising model pre-trained based on the two-stage video denoising model training method, so that the two-stage video denoising model outputs the second stage denoising result corresponding to the target video.

[0035] The fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the two-stage video denoising model training method is implemented, or the video denoising method is implemented.

[0036] The sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the two-stage video denoising model training method, or implements the video denoising method.

[0037] The dual-stage video denoising model training method provided in the present application obtains a plurality of image frame groups corresponding to each noisy video sample containing noise data, wherein each of the image frame groups contains a main image frame, a previous image frame and a next image frame that are temporally adjacent to the main image frame; uses the pre-acquired noise-free video data corresponding to each of the noisy video samples and each of the image frame groups to perform model training on a preset residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first-stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames in the second stage based on the first-stage denoising results of each of the main image frames to obtain the second-stage denoising corresponding to each of the main image frames. As a result, a two-stage video denoising model for video denoising is generated. The present application can extract dynamic features and static features from two stages respectively according to the different characteristics of the two dimensions of time and space, use the first stage to capture dynamic information to obtain a preliminary denoising result, and use the second stage to capture static information to obtain the final denoising result. Compared with the single-stage method of directly denoising the spatial and temporal information together, the different characteristics of the two dimensions of time and space are taken into account, so that the denoising result is more robust; at the same time, by adopting the residual self-attention model, more global context features and local information can be retained in the denoising process, making the denoising result more significant, which can effectively improve the performance of the trained video denoising model, and then effectively improve the robustness, significance and reliability of the video denoising result output by the two-stage video denoising model.

[0038] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0039] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0041] Figure 1 Schematic diagram of the process of a two-stage video denoising model training method in one embodiment of the present application.

[0042] Figure 2 Schematic diagram of the overall architecture of a two-stage video denoising model in one embodiment of the present application.

[0043] Figure 3 Schematic diagram of the structure of the first-stage network in the two-stage video denoising model in one embodiment of the present application.

[0044] Figure 4 Schematic diagram of the structure of the second stage network in the two-stage video denoising model in one embodiment of the present application.

[0045] Figure 5 Schematic diagram of the structure of the encoder module in the two-stage video denoising model in one embodiment of the present application.

[0046] Figure 6 Schematic diagram of the structure of a decoder module in a two-stage video denoising model in an embodiment of the present application.

[0047] Figure 7 FIG. 4 is a flow chart of a video denoising method in another embodiment of the present application.

[0048] Figure 8 This is a structural diagram of a dual-stage video denoising model training system in the third embodiment of the present application.

[0049] Fig. 9 Schematic diagram of the structure of a video denoising system in the fourth embodiment of the present application.

[0050] Fig.10 This is a flowchart of the two-stage video denoising model training method provided in the application example of this application.

[0051] Fig.11 This is a logical example diagram of the two-stage video denoising model provided in the application example of this application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0053] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0054] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0055] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0056] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0057] It can be understood that the video denoising technology VNLnet uses a single-stage network to fuse spatial information and temporal information for denoising. It first calculates the n most similar neighbor image blocks of each image block (patch) in a rectangular spatiotemporal search window, then extracts features from similar neighbor image blocks and generates a feature vector which is then input into a convolutional neural network (CNN) to predict a clean video frame.

[0058] In addition, existing video denoising methods are mainly divided into three categories, including traditional methods based on image blocks (patch), explicit motion compensation methods and non-explicit motion compensation methods. Traditional methods based on image blocks (patch) refer to finding similar image blocks (patches) from the current frame and time series for video denoising. Image block-based methods represented by V-BM4D extend the image denoising method BM3D to the video level. With the development of neural networks, video denoising methods based on deep learning have been proposed in recent years. The earliest neural network-based methods use recurrent neural networks (RNNs) to denoise grayscale images. Explicit motion compensation methods refer to video denoising by capturing temporal information using motion estimation / compensation and optical flow. For example, DVDnet follows independent spatial and temporal steps to model the motion relationship between adjacent frames. Non-explicit motion compensation methods do not use motion estimation / compensation and optical flow to process temporal information. For example, VNLnet applies convolutional neural networks (CNN) to video denoising and uses a similarity search strategy to process non-local image patches. Subsequently, FastDVDnet expanded DVDnet by adding a U-Net network architecture. UDVD combines U-Net and Blind Spot Network without any obvious motion compensation.

[0059] Existing video denoising methods all use convolutional neural networks (CNNs), but the convolution kernels of convolutional neural networks have inherent local limitations, so these methods have limitations in capturing long-range temporal relationships. Based on this, in order to solve the problem that convolutional neural networks (CNNs) in the prior art are difficult to capture global information, this application first considers using a residual self-attention model, a model architecture that includes residual and self-attention modules, in the model training stage to solve this problem.

[0060] Furthermore, since the prior art ignores the difference between spatial and temporal dimensional information, the denoising effect of the trained model will be poor by simply superimposing the two types of information. Based on this, the present application solves this problem by designing a new two-stage model architecture, in which the first stage of the model is used to capture dynamic information to obtain preliminary denoising results, and the second stage is used to capture static information to obtain the final denoising results.

[0061] In one or more embodiments of the present application, a convolutional neural network (CNN) is a traditional artificial neural network model.

[0062] In one or more embodiments of the present application, the self-attention network model (Transformer) is a new deep learning network model. The self-attention mechanism is a weighted calculation method for image features. The multi-head self-attention layer (MSA) is a weighted calculation method for image features superimposed in multiple dimensions.

[0063] The details are described in detail through the following examples.

[0064] In order to effectively improve the application effectiveness, robustness, significance and reliability of the video denoising model, the present application embodiment provides a two-stage video denoising model training method that can be implemented by a two-stage video denoising model training system, see Figure 1 The two-stage video denoising model training method specifically includes the following contents:

[0065] Step 100: Acquire a plurality of image frame groups corresponding to each noise video sample containing noise data, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are temporally adjacent to the main image frame.

[0066] It can be understood that the noisy video samples refer to video samples containing noisy data, and each noisy video sample corresponds to a clean version of a noise-free video sample.

[0067] In one or more embodiments of the present application, the image frame refers to the smallest unit constituting a video, and in the subsequent description of the present application, the image frame may also be referred to as a video frame. The main image frame refers to an intermediate frame between three consecutive image frames in an image frame group.

[0068] In step 100, a set of training videos containing noise may be obtained. in represents a video containing noise, F i The corresponding clean video, N represents the number of videos in the video set. Traverse each video in turn All frames Each time, three consecutive frames and As the input of the subsequent network.

[0069] Step 200: Use the pre-acquired noise-free video data corresponding to each of the noisy video samples and each of the image frame groups to train the preset residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames based on the first stage denoising results of each of the main image frames in the second stage to obtain the second stage denoising results corresponding to each of the main image frames, thereby generating a two-stage video denoising model for video denoising.

[0070] In one or more embodiments of the present application, the residual self-attention model refers to a model provided with a residual structure and a self-attention module, and in the present application, the residual self-attention model is a two-stage model, the first stage is used to extract the dynamic features of the main image frames in each of the image frame groups to obtain the first-stage denoising results corresponding to each of the main image frames, and the second stage is used to extract the static features of each of the main image frames based on the first-stage denoising results of each of the main image frames to obtain the second-stage denoising results corresponding to each of the main image frames. Finally, the trained residual self-attention model forms a two-stage video denoising model that can be used for video denoising, that is, the model architecture of the two-stage video denoising model is the same as the architecture of the residual self-attention model, and the difference between the two names is only to distinguish the application stage and the training stage.

[0071] It can be understood that the specific model structure adopted in the two stages of the residual self-attention model can be set according to the actual application scenario. As long as the functions mentioned in step 200 can be implemented respectively, the residual self-attention model in this application can be applied.

[0072] From the above description, it can be seen that the two-stage video denoising model training method provided in the embodiment of the present application can extract dynamic features and static features from two stages according to the different characteristics of the two dimensions of time and space, respectively, and use the first stage to capture dynamic information to obtain a preliminary denoising result, and use the second stage to capture static information to obtain the final denoising result. Compared with the single-stage method that directly denoises the spatial and temporal information together, the different characteristics of the two dimensions of time and space are taken into account, so that the denoising result is more robust; at the same time, by adopting the residual self-attention model, more global context features and local information can be retained in the denoising process, making the denoising result more significant, which can effectively improve the performance of the trained video denoising model, and thus can effectively improve the robustness, significance and reliability of the video denoising result output by the two-stage video denoising model.

[0073] In order to further improve the application reliability and effectiveness of the residual self-attention model, in a two-stage video denoising model training method provided in an embodiment of the present application, see Figure 2 , the residual self-attention model specifically includes the following contents:

[0074] The first-stage network 1 and the second-stage network 2 are interconnected.

[0075] The first-stage network 1 is used to extract the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, and perform feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups to obtain the dynamic features of each of the main image frames; and then perform pixel reorganization on the dynamic features of each of the main image frames to obtain the first-stage denoising results corresponding to each of the main image frames;

[0076] The second stage network 2 is connected to the output end of the first stage network, and is used to respectively extract the static features of the main image frames corresponding to each of the first stage denoising results, and then downsample the static features of each of the main image frames to obtain the second stage denoising results corresponding to each of the main image frames.

[0077] Based on this, this application uses the first stage to capture dynamic information to obtain preliminary denoising results, and uses the second stage to capture static information to obtain the final denoising results. Compared with the single-stage method that directly denoises both spatial and temporal information, the different characteristics of the two dimensions of time and space are taken into account, making the denoising results more robust.

[0078] In order to further improve the application reliability and effectiveness of the first-stage network, in a two-stage video denoising model training method provided in an embodiment of the present application, see Figure 3 , the first stage network 1 in the residual self-attention model specifically includes the following contents:

[0079] A first encoder module 11, configured to extract global dynamic features of the previous image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism;

[0080] A second encoder module 12, configured to extract global dynamic features of the main image frames in each of the image frame groups by using a convolutional neural network and a self-attention mechanism;

[0081] A third encoder module 13, configured to extract global dynamic features of the subsequent image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism;

[0082] A feature fusion module 14 is connected to the output ends of the first encoder module, the second encoder module and the third encoder module respectively, and is used to perform feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, so as to obtain the dynamic features of each of the main image frames;

[0083] The first decoder module 15 is connected to the output end of the feature fusion module and is also connected to the second encoder module using a residual connection. The first decoder module is used to use a convolutional neural network to perform pixel reorganization on the dynamic features of each of the main image frames to obtain a first-stage denoising result corresponding to each of the main image frames.

[0084] Based on this, the present application can utilize the advantages of convolutional neural networks (CNN) and self-attention models (Transformer) to perform video denoising from two aspects: global temporal modeling and local image feature extraction.

[0085] In order to further improve the application reliability and effectiveness of the second-stage network, in a two-stage video denoising model training method provided in an embodiment of the present application, see Figure 4 , the second stage network 2 in the residual self-attention model specifically includes the following contents:

[0086] The fourth encoder module 21 is connected to the output end of the first-stage network, and is used to extract the static features of the main image frames corresponding to each of the first-stage denoising results by using a convolutional neural network and a self-attention mechanism.

[0087] The second decoder module 22 is connected to the fourth encoder module by a residual connection. The second decoder module is used to downsample the static features of each of the main image frames by using a convolutional neural network to obtain a second-stage denoising result corresponding to each of the main image frames.

[0088] (2) Construct the second-stage network. The second-stage network model is similar to the first-stage network model, including encoder and decoder parts and residual connections. The architecture of the encoder and decoder is the same as that of the first stage. The difference is that the second-stage network has only one encoder module, which is used to receive the output of the first-stage network, and there is no feature fusion module.

[0089] In order to further improve the application reliability and effectiveness of the encoder module, in a two-stage video denoising model training method provided in an embodiment of the present application, see Figure 5 The first encoder module, the second encoder module, the third encoder module and the fourth encoder module all specifically include the following contents:

[0090] The feature extraction model 3 and the two downsampling models 4 are connected in sequence;

[0091] The feature extraction model 3 includes a convolutional neural network 5 for extracting initial features of the image frame;

[0092] The downsampling model 4 includes: a downsampling layer 6, a convolutional neural network 5 and a self-attention module 7 connected in sequence, which is used to perform downsampling processing based on the initial features of the image frame and extract the global dynamic features of the image frame, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

[0093] In order to further improve the application reliability and effectiveness of the decoder module, in a two-stage video denoising model training method provided in an embodiment of the present application, see Figure 6 The first decoder module and the second decoder module specifically include the following contents:

[0094] Two upsampling models 8 and feature extraction model 3 connected in sequence;

[0095] The upsampling model 8 comprises: a convolutional neural network 5 and an upsampling layer 9 connected to each other, for upsampling the dynamic features of each of the main image frames based on a pixel reorganization strategy;

[0096] The feature extraction model 3 includes a convolutional neural network 5, which is used to obtain the denoising results corresponding to each of the main image frames according to the dynamic features of each of the main image frames after downsampling processing, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

[0097] Specifically, before training the residual self-attention model, the architecture of the residual self-attention model can be constructed first. First, the first-stage network is constructed. The first-stage network contains three identical encoder modules (i.e., the first to third encoding modules), a feature fusion module, and a decoder module. Among them, the encoder part contains a convolutional neural network (CNN) feature extraction model and two downsampling models. The convolutional neural network (CNN) feature extraction part contains two layers of 3×3 convolutions, followed by a batch normalization layer (BN layer) and an activation function layer (Relu) operation, and followed by two layers of 3×3 convolutions and a self-attention layer after downsampling. In addition, in order to capture global information, the encoder module also contains a self-attention module; the first decoder module mainly consists of three parts, two upsampling parts and a convolutional neural network (CNN) feature extraction part. Two layers of 3×3 convolution layers are connected before upsampling. The convolutional neural network (CNN) feature extraction part also contains two layers of 3×3 convolutions, and the upsampling operation uses a pixel reorganization (Pixel shuffle) strategy. Since the purpose of this process is to repair the intermediate frame, a residual connection is used between the second encoder module and the first decoder module of the intermediate frame to retain the global context features and local information of the current main repaired frame to facilitate subsequent image restoration.

[0098] In the process of constructing the architecture of the residual self-attention model, the second-stage network can be further constructed after the first-stage network is constructed. The second-stage network model is similar to the first-stage network model, including an encoder module, a decoder module and a residual connection. The architecture of the encoder module and the decoder module is the same as that of the first stage. The difference is that the second-stage network has only one encoder module, which is used to accept the output of the first-stage network, and there is no feature fusion module.

[0099] In order to further improve the reliability and effectiveness of the two-stage video denoising model training process, in a two-stage video denoising model training method provided in an embodiment of the present application, based on the structural improvement of the above-mentioned two-stage video denoising model, step 200 in the two-stage video denoising model training method specifically includes the following contents:

[0100] The three frames in each image frame group are sequentially (in ) are input into the designed two-stage model, where C represents the number of channels, H represents the height, W represents the width, and R represents the set of real numbers. In the first stage, in the three encoder modules with the same architecture, the three frames are firstly passed through the first layer of convolutional neural network (CNN) feature extraction part to obtain the initial features of size C×H×W. Then, the downsampling layer, convolution layer and self-attention layer in the second layer of downsampling part are used to obtain the initial features of size The features of are then obtained through the third layer downsampling part to obtain a size of characteristics.

[0101] In this process, in order to better capture global information, a self-attention mechanism layer is introduced in the last two convolution modules. The self-attention layer mainly includes a self-attention model (Transformer) layer.

[0102] The self-attention model (Transformer) layer follows the convolutional neural network (CNN). For the self-attention model (Transformer) layer, given the feature x∈R C×H×W , the number of channels is C, the height is H, and the width is W. First, the image is divided into patches of size P×P, so that the size of the feature changes to in Represents the total number of windows. Then, calculate each window The self-attention matrix is ​​obtained by using the local window:

[0103]

[0104] Among them, B is the learnable relative position bias, K represents the Key matrix, Q represents the Query matrix, and V represents the Value matrix. T represents the matrix transpose, and d k is the feature vector dimension.

[0105] Subsequently, it is passed to the multi-head self-attention layer (MSA) to complete the global feature extraction. In this process, the multi-layer perceptron (MLP) is also used for further feature transformation, and both layers are preceded by a regularization (LN) layer. The process can be described as follows:

[0106]

[0107]

[0108] The output results of the three decoder parts are input into the feature fusion module, and the feature fusion module fuses the three features through convolution operation to obtain the fused output.

[0109]

[0110] in They represent the outputs of the three codes respectively, and CONV() represents the convolution operation. fuse The input is sent to the decoder module, which first passes through two convolutional layers and an upsampling layer to get the output Then two convolutional layers and one upsampling layer are used to obtain the features. Finally, the features are output to two convolutional layers to get the final output

[0111] Will As the input of the second stage. That is, input into the encoder module to obtain the output feature Then input to the decoder part to get the output It is the final output of the denoising model

[0112] During the training process, the stochastic gradient optimization algorithm is used to optimize the model, and the total loss function L(θ) is:

[0113]

[0114] in and They represent the denoising result and true value of the model output respectively. N is the number of training data, and θ represents all the parameters that the network needs to learn.

[0115] Based on the aforementioned two-stage video denoising model training method, the accuracy and reliability of video denoising are further improved. The present application also provides a video denoising method, see Figure 7 The video denoising method specifically includes the following contents:

[0116] Step 300: Acquire multiple image frame groups corresponding to the target video, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame.

[0117] Step 400: Input multiple image frame groups corresponding to the target video into a two-stage video denoising model pre-trained based on a two-stage video denoising model training method, so that the two-stage video denoising model outputs a second-stage denoising result corresponding to the target video.

[0118] In step 400, the two-stage video denoising model training method refers to all or part of the steps in the two-stage video denoising model training method mentioned in the above embodiment, and its functions are not repeated here. Please refer to the detailed description of the above two-stage video denoising model training method embodiment.

[0119] From the above description, it can be seen that the video denoising method provided in the embodiment of the present application can extract dynamic features and static features from two stages according to the different characteristics of the two dimensions of time and space, and can effectively improve the performance of the trained video denoising model, thereby effectively improving the robustness, significance and reliability of the video denoising results output by the two-stage video denoising model.

[0120] From the software level, the present application also provides a two-stage video denoising model training system for executing all or part of the two-stage video denoising model training method, see Figure 8 The two-stage video denoising model training system specifically includes the following contents:

[0121] The image frame extraction module 10 is used to obtain a plurality of image frame groups corresponding to each noise video sample containing noise data, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are adjacent to the main image frame in time sequence;

[0122] The two-stage model training module 20 is used to use the noise-free video data corresponding to each of the pre-acquired noisy video samples and each of the image frame groups to perform model training on the preset residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first-stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames based on the first-stage denoising results of each of the main image frames in the second stage to obtain the second-stage denoising results corresponding to each of the main image frames, thereby generating a two-stage video denoising model for video denoising.

[0123] The embodiment of the two-stage video denoising model training system provided in the present application can be specifically used to execute the processing flow of the embodiment of the two-stage video denoising model training method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned two-stage video denoising model training method embodiment.

[0124] The part of the dual-stage video denoising model training system that performs dual-stage video denoising model training can be executed in the server, and in another practical application scenario, all operations can also be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the dual-stage video denoising model training.

[0125] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0126] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0127] From the above description, it can be seen that the two-stage video denoising model training system provided in the embodiment of the present application can extract dynamic features and static features from two stages according to the different characteristics of the two dimensions of time and space, and can effectively improve the performance of the trained video denoising model, thereby effectively improving the robustness, significance and reliability of the video denoising results output by the two-stage video denoising model.

[0128] From the software level, the present application also provides a video denoising system for executing all or part of the video denoising method, see Fig. 9 , the video denoising system specifically includes the following contents:

[0129] The image frame acquisition module 30 is used to acquire a plurality of image frame groups corresponding to the target video, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame;

[0130] The two-stage model denoising module 40 is used to input multiple image frame groups corresponding to the target video into a two-stage video denoising model pre-trained based on the two-stage video denoising model training method, so that the two-stage video denoising model outputs the second stage denoising result corresponding to the target video.

[0131] In the two-stage model denoising module 40, the two-stage video denoising model training method refers to all or part of the steps in the two-stage video denoising model training method mentioned in the aforementioned embodiment. Its functions are not repeated here, and reference may be made to the detailed description of the above-mentioned two-stage video denoising model training method embodiment.

[0132] The embodiment of the video denoising system provided in the present application can be specifically used to execute the processing flow of the embodiment of the video denoising method in the above embodiment. Its functions are not described in detail here, and reference can be made to the detailed description of the above video denoising method embodiment.

[0133] The part of the video denoising system that performs video denoising can be executed in the server, and in another practical application scenario, all operations can also be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of video denoising.

[0134] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0135] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0136] From the above description, it can be seen that the video denoising system provided in the embodiment of the present application can extract dynamic features and static features from two stages according to the different characteristics of the two dimensions of time and space, and can effectively improve the performance of the trained video denoising model, thereby effectively improving the robustness, significance and reliability of the video denoising results output by the two-stage video denoising model.

[0137] In order to further illustrate the present solution, the present application also provides a specific application example of a two-stage video denoising model training method and system based on a residual self-attention mechanism. The application example of the present application improves the performance of video denoising from the following two aspects:

[0138] (1) Taking advantage of both convolutional neural networks (CNN) and self-attention models (Transformer), features are processed in both global and local dimensions;

[0139] (2) A new two-stage model architecture is proposed, in which the first stage of the model is used to capture dynamic information to obtain preliminary denoising results, and the second stage is used to capture static information to obtain the final denoising results.

[0140] See also Fig.10 The two-stage video denoising model training method proposed in the application example of this application specifically includes the following contents:

[0141] 1. Obtain the training set data, traverse each video in the training set, and use the three adjacent frames in the video as the input of the network model.

[0142] The specific method is:

[0143] Get a set of training videos containing noise in represents a video containing noise, F i The corresponding clean video. Traverse each video in turn All frames Each time, three consecutive frames As the input of the subsequent network.

[0144] 2. Construct a two-stage denoising network.

[0145] The specific method is:

[0146] (1) Construct the network model of the first stage. The network contains three identical encoder modules, a feature fusion module and a decoder module. The encoder part contains a convolutional neural network (CNN) feature extraction model and two downsampling models. The convolutional neural network (CNN) feature extraction part contains two layers of 3×3 convolutions. The convolutional layer is followed by a batch normalization layer (BN layer) and an activation function layer (Relu) operation. The downsampling is followed by two layers of 3×3 convolutions and a self-attention layer. In addition, in order to capture global information, the encoder module also contains a self-attention module; the decoder module mainly consists of three parts, two upsampling parts and a convolutional neural network (CNN) feature extraction part. Two layers of 3×3 convolution layers are connected before upsampling. The convolutional neural network (CNN) feature extraction part also contains two layers of 3×3 convolutions. The upsampling operation uses a pixel reorganization (Pixel shuffle) strategy. Since the purpose of this method is to repair the intermediate frame, a residual connection is used between the encoder and decoder modules of the intermediate frame to retain the global context features and local information of the current main repair frame, which is conducive to subsequent image restoration.

[0147] (2) Construct the second-stage network. The second-stage network model is similar to the first-stage network model, including encoder and decoder parts and residual connections. The architecture of the encoder and decoder is the same as that of the first stage. The difference is that the second-stage network has only one encoder module, which is used to receive the output of the first-stage network, and there is no feature fusion module.

[0148] 3. Input the features of three adjacent frames of the video into the one-stage network and output the one-stage denoising result of the middle frame.

[0149] The specific method is:

[0150] The three frames are (in ) are respectively input into the designed two-stage model (2), where C represents the number of channels, H represents the height, W represents the width, and R represents the set of real numbers. In the first stage, in the three encoder modules with the same architecture, the three frames are firstly passed through the first layer of convolutional neural network (CNN) feature extraction part to obtain the initial features of size C×H×W. Then, the downsampling layer, convolution layer and self-attention layer in the second layer of downsampling part are used to obtain the initial features of size The features of are then obtained through the third layer downsampling part to obtain a size of characteristics.

[0151] In this process, in order to better capture global information, a self-attention mechanism layer is introduced in the last two convolution modules. The self-attention layer mainly includes a self-attention model (Transformer) layer.

[0152] The self-attention model (Transformer) layer follows the convolutional neural network (CNN). For the self-attention model (Transformer) layer, given the feature x∈R C×H×W , the number of channels is C, the height is H, and the width is W. First, the image is divided into patches of size P×P, so that the size of the feature changes to in Represents the total number of windows. Then, calculate each window The self-attention matrix is ​​obtained by using the local window:

[0153]

[0154] Among them, B is the learnable relative position bias, K represents the Key matrix, Q represents the Query matrix, and V represents the Value matrix. T represents the matrix transpose, and d k is the feature vector dimension. Subsequently, it is passed to the multi-head self-attention layer (MSA) to complete the global feature extraction. In this process, the multi-layer perceptron (MLP) is also used for further feature transformation, and both layers are preceded by a regularization (LN) layer. The process can be described as follows:

[0155]

[0156]

[0157] The output results of the three decoder parts are input into the feature fusion module, and the feature fusion module fuses the three features through convolution operation to obtain the fused output.

[0158]

[0159] in They represent the outputs of the three codes respectively, and CONV() represents the convolution operation.

[0160] The feature f fuse The input is sent to the decoder module, which first passes through two convolutional layers and an upsampling layer to get the output Then two convolutional layers and one upsampling layer are used to obtain the features. Finally, the features are output to two convolutional layers to get the final output

[0161] 4. Input the denoising result of the first stage into the second stage network and output the final denoising result of the intermediate frame.

[0162] The specific method is:

[0163] The output of step 3 As the input of the second stage, it is input into the encoder module as in step 3 to obtain the output feature Then input to the decoder part to get the output It is the final output of the denoising model

[0164] 5. Perform end-to-end training on the two-stage network to obtain a trained model.

[0165] The specific method is:

[0166] During the training process, the stochastic gradient optimization algorithm is used to optimize the model, and the total loss function L(θ) is:

[0167]

[0168] in and They represent the denoising result and true value of the model output respectively. N is the number of training data, and θ represents all the parameters that the network needs to learn.

[0169] 6. Input the test set into the trained model to obtain the denoised test video.

[0170] The specific method is:

[0171] Traverse the test video frames according to step 1, and then input the continuous video frames into the trained model to obtain the denoised test video frames, thereby obtaining the final clean video.

[0172] See also Fig.11 The two-stage video denoising model proposed in this application example specifically includes the following contents:

[0173] 1. One-stage and two-stage encoder modules: extract features of video frames and understand the content of video frames.

[0174] 2. One-stage feature fusion module: used to fuse the features of temporal video frames.

[0175] 3. One-stage and two-stage decoder modules: restore the abstract features of video frames to their original shapes.

[0176] 4. One-stage and two-stage self-attention modules: extracting global information of video frames.

[0177] The video frame j-1 represents the previous image frame in the three consecutive frames in the image frame group that is adjacent to the main image frame in time sequence. Video frame j represents the main image frame among three consecutive frames in the image frame group. Video frame j+1 represents the next image frame in the three consecutive frames in the image frame group that is adjacent to the main image frame in time sequence.

[0178] In summary, the application examples of this application respectively utilize the advantages of convolutional neural networks (CNN) and self-attention models (Transformer) to perform video denoising from two aspects: global time series modeling and local image feature extraction. Taking into account the different characteristics of temporal information and spatial information, a two-stage approach is used for progressive denoising. The first stage is used to capture dynamic information to obtain preliminary denoising results, and the second stage is used to capture static information to obtain the final denoising results. It is proposed to introduce a residual connection operation after the self-attention layer to retain the global context features and local information of the current main repair frame, thereby improving the denoising effect.

[0179] Based on this, compared with the prior art, the beneficial effects of the application example of this application are:

[0180] (1) The present invention fully utilizes the advantages of both convolutional neural network (CNN) and self-attention model (Transformer), and can process features from both global and local aspects. Compared with the method of using only convolutional neural network (CNN), it can perform better denoising.

[0181] (2) The present invention uses a first stage to capture dynamic information to obtain a preliminary denoising result, and uses a second stage to capture static information to obtain a final denoising result. Compared with a single-stage method that directly denoises both spatial and temporal information, the present invention takes into account the different characteristics of the two dimensions of time and space, thereby making the denoising result more robust.

[0182] (3) The present invention introduces a residual operation after the self-attention layer to retain more global context features and local information during the denoising process, making the denoising result more significant.

[0183] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the two-stage video denoising model training method or the video denoising method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via wired or wireless means.

[0184] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0185] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the two-stage video denoising model training method or the video denoising method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the two-stage video denoising model training method or the video denoising method in the above method embodiment is implemented.

[0186] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The one or more modules are stored in the memory, and when executed by the processor, perform the dual-stage video denoising model training method or video denoising method in the embodiment.

[0188] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0189] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0190] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0191] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned two-stage video denoising model training method or video denoising method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0192] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0193] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0194] In this application, the features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with or replace the features of other embodiments.

[0195] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A two-stage video denoising model training method, characterized in that: include: Acquire a plurality of image frame groups corresponding to each noise video sample containing noise data, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame; The pre-acquired noise-free video data corresponding to each of the noise video samples and each of the image frame groups are used to train the preset residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames based on the first stage denoising results of each of the main image frames in the second stage to obtain the second stage denoising results corresponding to each of the main image frames, thereby generating a dual-stage video denoising model for video denoising; The residual self-attention model includes: The first-stage network is used to extract the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, and perform feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups to obtain the dynamic features of each of the main image frames; and then perform pixel reorganization on the dynamic features of each of the main image frames to obtain the first-stage denoising results corresponding to each of the main image frames; A second-stage network is connected to the output end of the first-stage network, and is used to extract static features of the main image frames corresponding to the first-stage denoising results, and then downsample the static features of the main image frames to obtain second-stage denoising results corresponding to the main image frames; The first phase network includes: A first encoder module, configured to extract global dynamic features of the previous image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism; A second encoder module, configured to extract global dynamic features of the main image frames in each of the image frame groups by using a convolutional neural network and a self-attention mechanism; A third encoder module, configured to extract global dynamic features of the latter image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism; a feature fusion module, connected to the output ends of the first encoder module, the second encoder module and the third encoder module respectively, and used for performing feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, so as to obtain the dynamic features of each of the main image frames; a first decoder module, connected to the output end of the feature fusion module, and also connected to the second encoder module by a residual connection, the first decoder module being used to perform pixel reorganization on the dynamic features of each of the main image frames by using a convolutional neural network to obtain a first-stage denoising result corresponding to each of the main image frames; The second phase network includes: a fourth encoder module, connected to the output end of the first-stage network, for respectively extracting static features of the main image frames corresponding to the first-stage denoising results by using a convolutional neural network and a self-attention mechanism; A second decoder module is connected to the fourth encoder module by a residual connection, and the second decoder module is used to downsample the static features of each of the main image frames by using a convolutional neural network to obtain a second-stage denoising result corresponding to each of the main image frames; The first encoder module, the second encoder module, the third encoder module and the fourth encoder module each include: a feature extraction model and two downsampling models connected in sequence; The feature extraction model includes a convolutional neural network for extracting initial features of the image frame; The downsampling model includes: a downsampling layer, a convolutional neural network and a self-attention module connected in sequence, which is used to perform downsampling processing based on the initial features of the image frame and extract the global dynamic features of the image frame, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

2. The two-stage video denoising model training method according to claim 1, characterized in that: The first decoder module and the second decoder module each include: two upsampling models and a feature extraction model connected in sequence; The upsampling model includes: a convolutional neural network and an upsampling layer connected to each other, used for upsampling the dynamic features of each of the main image frames based on a pixel reorganization strategy; The feature extraction model includes a convolutional neural network, which is used to obtain denoising results corresponding to each of the main image frames based on the dynamic features of each of the main image frames after downsampling processing, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

3. A video denoising method, characterized in that: include: Acquire multiple image frame groups corresponding to the target video, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame that are sequentially adjacent to the main image frame; The multiple image frame groups corresponding to the target video are input into a two-stage video denoising model pre-trained based on the two-stage video denoising model training method described in claim 1 or 2, so that the two-stage video denoising model outputs the second stage denoising result corresponding to the target video.

4. A two-stage video denoising model training system, characterized in that: include: An image frame extraction module, used for acquiring a plurality of image frame groups corresponding to each noise video sample containing noise data, wherein each of the image frame groups includes a main image frame, a previous image frame and a next image frame which are sequentially adjacent to the main image frame; A two-stage model training module, used to use the noise-free video data corresponding to each of the pre-acquired noisy video samples and each of the image frame groups to perform model training on a preset residual self-attention model, so that the residual self-attention model first extracts the dynamic features of the main image frames in each of the image frame groups in the first stage to obtain the first-stage denoising results corresponding to each of the main image frames, and then extracts the static features of each of the main image frames based on the first-stage denoising results of each of the main image frames in the second stage to obtain the second-stage denoising results corresponding to each of the main image frames, thereby generating a two-stage video denoising model for video denoising; The residual self-attention model includes: The first-stage network is used to extract the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, and perform feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups to obtain the dynamic features of each of the main image frames; and then perform pixel reorganization on the dynamic features of each of the main image frames to obtain the first-stage denoising results corresponding to each of the main image frames; A second-stage network is connected to the output end of the first-stage network, and is used to extract static features of the main image frames corresponding to the first-stage denoising results, and then downsample the static features of the main image frames to obtain second-stage denoising results corresponding to the main image frames; The first phase network includes: A first encoder module, configured to extract global dynamic features of the previous image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism; A second encoder module, configured to extract global dynamic features of the main image frames in each of the image frame groups by using a convolutional neural network and a self-attention mechanism; A third encoder module, configured to extract global dynamic features of the latter image frame in each of the image frame groups by using a convolutional neural network and a self-attention mechanism; a feature fusion module, connected to the output ends of the first encoder module, the second encoder module and the third encoder module respectively, and used for performing feature fusion processing on the global dynamic features corresponding to the main image frame, the previous image frame and the next image frame in each of the image frame groups, so as to obtain the dynamic features of each of the main image frames; a first decoder module, connected to the output end of the feature fusion module, and also connected to the second encoder module by a residual connection, the first decoder module being used to perform pixel reorganization on the dynamic features of each of the main image frames by using a convolutional neural network to obtain a first-stage denoising result corresponding to each of the main image frames; The second phase network includes: a fourth encoder module, connected to the output end of the first-stage network, for respectively extracting static features of the main image frames corresponding to the first-stage denoising results by using a convolutional neural network and a self-attention mechanism; A second decoder module is connected to the fourth encoder module by a residual connection, and the second decoder module is used to downsample the static features of each of the main image frames by using a convolutional neural network to obtain a second-stage denoising result corresponding to each of the main image frames; The first encoder module, the second encoder module, the third encoder module and the fourth encoder module each include: a feature extraction model and two downsampling models connected in sequence; The feature extraction model includes a convolutional neural network for extracting initial features of the image frame; The downsampling model includes: a downsampling layer, a convolutional neural network and a self-attention module connected in sequence, which is used to perform downsampling processing based on the initial features of the image frame and extract the global dynamic features of the image frame, wherein the convolutional neural network includes: a plurality of convolutional layers connected in sequence.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the dual-stage video denoising model training method as described in claim 1 or 2, or implements the video denoising method as described in claim 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dual-stage video denoising model training method as described in claim 1 or 2 is implemented, or the video denoising method as described in claim 3 is implemented.

Images