Video super-resolution reconstruction method based on D3D convolutional intra-group fusion network

Abstract

The present invention discloses a video super-resolution reconstruction method based on a D3D convolutional intra-group fusion network, comprising the following steps: obtaining a low-resolution video sequence to be reconstructed; inputting the low-resolution video sequence to be reconstructed into a trained video super-resolution reconstruction network model; the model outputting a video super-resolution reconstruction result, namely a high-resolution video sequence; wherein the video super-resolution reconstruction network model comprises a temporal grouping module, a C3D shallow feature extraction module, a D3D convolutional intra-group fusion module, an inter-group attention mechanism module, and a reconstruction module. This method improves the utilization rate of video frames, can learn to obtain offset vectors from the current input feature map, integrates temporal and spatial information, and achieves superior reconstruction performance while maintaining temporal consistency of video frames.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a video super-resolution reconstruction method based on a D3D convolutional group fusion network. Background Art

[0002] Super-resolution refers to the use of software algorithms to restore corresponding high-resolution images from low-resolution images. It can be specifically divided into single-image super-resolution and video super-resolution. With the continuous updating of mobile Internet and communication technologies, video super-resolution has been widely used in remote sensing imaging, panoramic video, high-definition television and other fields. At the same time, people's pursuit of higher-definition videos has made video super-resolution receive more and more attention.

[0003] Video super-resolution (VSR) can, to some extent, be viewed as an extension of single-image super-resolution. By dividing a video into multiple frames and then applying the principles of single-image super-resolution to these frames, high-resolution (HR) video frames can be obtained. However, super-resolution reconstruction performed in this manner fails to utilize inter-frame motion information and preserves intra-frame temporal correlations, resulting in artifacts and interference within the video frames and poor reconstruction results. Therefore, recent deep learning-based VSR reconstruction methods have focused on exploring efficient frame alignment methods that exploit the spatiotemporal information between video frames. Existing VSR reconstruction methods typically consist of an alignment module, a feature extraction and characterization module, and a reconstruction module.

[0004] In the alignment module, the optical flow method is the most widely used. It uses the temporal changes of pixels in a video sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames. This method based on display motion compensation can cause large distortion and errors when dealing with complex motion or in the presence of occlusion, affecting super-resolution performance. The subsequent method of using dynamic upsampling filters and implicit motion compensation to estimate the motion relationship between video frames to reconstruct HR video frames has improved the accuracy of estimating the motion information between video frames. However, the size of the upsampling filter directly determines the reconstruction performance, so the calculation speed is not ideal. Later, the deformable convolution method was used. Although it further utilized the spatiotemporal information between adjacent frames, its modeling process first performed spatial feature extraction and then motion estimation. It was unable to fully utilize the spatiotemporal correlation between adjacent frames, affecting the reconstruction performance and visual effects. Summary of the Invention

[0005] In response to the current problem of insufficient utilization of spatiotemporal information in video super-resolution, the present invention provides a video super-resolution reconstruction method based on a D3D convolutional group fusion network. The input video sequence is rearranged into several groups of subsequences with different frame rates, and the inter-frame information of video sequences with different frame rates is integrated by time grouping and layering to improve the utilization rate of video frames. The method can learn to obtain offset vectors from the current input feature map, integrate temporal and spatial information, and obtain better reconstruction performance while maintaining the temporal consistency of video frames.

[0006] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, a video super-resolution reconstruction method based on a D3D convolutional group fusion network is provided, comprising:

[0008] Obtaining a low-resolution video sequence to be reconstructed;

[0009] Inputting the low-resolution video sequence to be reconstructed into the video super-resolution reconstruction network model trained for testing;

[0010] Obtaining a video super-resolution reconstruction result, i.e., a high-resolution video sequence, according to the output of the video super-resolution reconstruction network model;

[0011] The video super-resolution reconstruction network model includes a time grouping module, a C3D shallow feature extraction module, a D3D convolution group fusion module, an inter-group attention mechanism module and a reconstruction module;

[0012] The temporal grouping module is used to group the input low-resolution video sequence according to different frame rates to obtain N groups of video subsequences;

[0013] The C3D shallow feature extraction module is used to perform preliminary feature extraction and time alignment on the video subsequences grouped by the time grouping module to obtain video features F, and send them to the D3D convolution group fusion module;

[0014] The D3D convolution group fusion module is used to extract spatial features from video features using a spatial feature extractor to obtain spatial features, and then use five D3D convolution residual blocks with a convolution kernel of 3×3×3 to fuse the spatial features with spatiotemporal features to obtain fused features, and use two-dimensional dense blocks to extract group features from the fused features to generate group features within the group.

[0015] The inter-group attention mechanism module is used to analyze the characteristics of multiple groups within the group. Perform temporal attention fusion to generate attention feature map M n (x,y) j, and sent to the reconstruction module;

[0016] The reconstruction module is used to perform bicubic interpolation upsampling on the original low-resolution video sequence to be reconstructed to generate the original video residual image The attention feature map is fed into six cascaded residual blocks and a sub-pixel convolutional layer for reconstruction, and the corresponding residual map R is generated after processing. t ; The residual graph R t and the original video residual map Add together to generate the final high-resolution video sequence

[0017] In some embodiments, the training method of the video super-resolution reconstruction network model includes:

[0018] Obtain a low-resolution video sequence dataset;

[0019] The video super-resolution reconstruction network model is trained and tested using the data set to obtain a tested and trained video super-resolution reconstruction network model.

[0020] In some embodiments, the loss function L1(x) during the video super-resolution reconstruction network model training process is:

[0021]

[0022] Where x represents the set of weighted and bias parameters, i represents the number of iterations during training, and m represents the number of training video frames. High-resolution video sequences output by the model, Represents the low-resolution video sequence input to the model, F(.) represents the predicted value of generating high-resolution video, and ||*|| represents the norm.

[0023] In some embodiments, obtaining a low-resolution video sequence dataset includes:

[0024] The dataset uses a standard dataset or is collected and constructed by yourself;

[0025] If the standard vid4 and dataset are used for training and testing; the dataset contains video frame sequences of four scenes: calendar, city, foliage, and walk. Each scene contains 41, 34, 49, and 47 frames of video images. The high-resolution video frames are downsampled four times using Gaussian blur with a standard deviation of σ=1.6 to generate the corresponding low-resolution video frames.

[0026] In some embodiments, the processing of the time grouping module includes:

[0027] The adjacent 2N frames are divided into N groups according to the temporal distance to the reference frame, and the original video sequence is reordered as {G1,...G n}, n∈[i:N], where From the previous frame Reference Frame and the next frame where L represents the symbol of the low-resolution video sequence.

[0028] In some embodiments, the method for constructing the D3D convolution group fusion module includes:

[0029] The D3D convolution group fusion module includes a spatial feature extractor, a D3D convolution layer and a two-dimensional dense block;

[0030] Each unit of the spatial feature extractor consists of a 3×3 convolution layer, a batch normalization BN layer and a Relu activation function. It takes the video feature F extracted by the C3D shallow feature extraction module as input and outputs the spatial feature after processing.

[0031] The spatial features Use 5 D3D convolution residual blocks with 3×3×3 convolution kernels to fuse spatiotemporal features to obtain fused features.

[0032] The fusion features The two-dimensional dense block is fed into the two-dimensional dense block, and the intra-group features are extracted by applying 18 two-dimensional units in the two-dimensional dense block to generate the intra-group features.

[0033] In some embodiments, the processing of the C3D shallow feature extraction module includes:

[0034] 1) Perform three-dimensional convolution kernel sampling on the input feature x to obtain the sampling value;

[0035] 2) Perform weighted summation of the sampled values ​​using the function w;

[0036] The features passed through the 3×3×3 convolution kernel with a dilation factor of 1 are expressed as follows:

[0037]

[0038] Among them, y(p0) represents the generated output feature, a position in the output feature can be represented by p0, p n Represents the nth value in the 3×3×3 convolutional sampling network, where N represents the number of iterations;

[0039] The processing process of the D3D convolution residual block of the fusion module in the D3D convolution group includes:

[0040]

[0041] Where Δp n Indicates the offset corresponding to the nth value in the 3×3×3 convolutional sampling network; the offset is usually a decimal, so a more precise value needs to be generated by bilinear interpolation.

[0042] In some embodiments, the inter-group attention mechanism module includes,

[0043] For group characteristics within the group After applying a 3×3 convolutional layer, the feature map of a channel is calculated. Generated feature map F1 a 、F2 a 、F3 a are further connected, and the softmax function along the time axis is applied to each position, across channels, to calculate the temporal attention feature map M(x,y);

[0044] Connect the intermediate graphs of each group and calculate the attention feature map M(x,y) through the softmax function along the time axis;

[0045]

[0046] For each group, the attention weighted features It can be calculated by the following formula:

[0047]

[0048] Among them, M n (x,y) j Represents the temporal attention mask at (x,y) j The weight of the position, Represents the group characteristics within the group, and ⊙ represents the multiplication of corresponding elements one by one.

[0049] In a second aspect, the present invention provides a video super-resolution reconstruction device based on a D3D convolutional group fusion network, comprising a processor and a storage medium;

[0050] The storage medium is used to store instructions;

[0051] The processor is configured to operate according to the instructions to execute the steps of the method according to the first aspect.

[0052] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0053] Beneficial effects: The video super-resolution reconstruction method based on the D3D convolutional group fusion network provided by the present invention has the following advantages:

[0054] (1) An implicit layered approach is used to process various motions between video frames, thereby restoring the missing details of the reference frame by utilizing the complementary information between different frame rates, and adaptively borrowing information from groups with different frame rates to integrate adjacent frames of different temporal distances. The reference frame of each group can guide the network model to extract more useful information from adjacent frames, making the information extraction and fusion of subsequent intra-group fusion modules more efficient.

[0055] (2) Deformable convolution is combined with C3D (Convolution 3D, C3D) to realize deformable three-dimensional convolution, which is added to the inter-group fusion stage, so that motion compensation can be performed adaptively and spatiotemporal information can be efficiently developed and utilized.

[0056] (3) Temporal attention is applied to the deep inter-group fusion module to integrate the information of different time groups after feature extraction, retain more high-frequency information for the subsequent reconstruction module, and obtain better peak signal-to-noise ratio and structural similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 4 is an overall flow chart of the method in an embodiment of the present invention.

[0058] Figure 2 This is a diagram of the video super-resolution reconstruction network model based on D3D convolution group fusion in an embodiment of the present invention.

[0059] Figure 3 Schematic diagram of the D3D convolution group fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0061] In the description of the present invention, "several" means more than one, "plurality" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0062] In the description of the present invention, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the exemplary expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0063] Example 1

[0064] A video super-resolution reconstruction method based on a D3D convolutional group fusion network, comprising:

[0065] Obtaining a low-resolution video sequence to be reconstructed;

[0066] Inputting the low-resolution video sequence to be reconstructed into the video super-resolution reconstruction network model trained for testing;

[0067] Obtaining a video super-resolution reconstruction result, i.e., a high-resolution video sequence, according to the output of the video super-resolution reconstruction network model;

[0068] The video super-resolution reconstruction network model includes a time grouping module, a C3D shallow feature extraction module, a D3D convolution group fusion module, an inter-group attention mechanism module and a reconstruction module;

[0069] The temporal grouping module is used to group the input low-resolution video sequence according to different frame rates to obtain N groups of video subsequences;

[0070] The C3D shallow feature extraction module is used to perform preliminary feature extraction and time alignment on the video subsequences grouped by the time grouping module to obtain video features F, and send them to the D3D convolution group fusion module;

[0071] The D3D convolution group fusion module is used to extract spatial features from video features using a spatial feature extractor to obtain spatial features, and then use five D3D convolution residual blocks with a convolution kernel of 3×3×3 to fuse the spatial features with spatiotemporal features to obtain fused features, and use two-dimensional dense blocks to extract group features from the fused features to generate group features within the group.

[0072] The inter-group attention mechanism module is used to analyze the group characteristics within the group. After applying a 3×3 convolutional layer, the feature map of a channel can be calculated Generated feature map F1 a 、F2 a、F3 a is further connected, and the softmax function along the time axis is applied to each position, across channels, to calculate the temporal attention feature map M n (x,y) j ;

[0073] Multiple group characteristics Perform temporal attention fusion to generate attention feature map M n (x,y) j , and sent to the reconstruction module;

[0074] The reconstruction module is used to perform bicubic interpolation upsampling on the original low-resolution video sequence to be reconstructed to generate the original video residual image The attention feature map is fed into six cascaded residual blocks and a sub-pixel convolutional layer for reconstruction, and the corresponding residual map R is generated after processing. t ; The residual graph R t and the original video residual map Add together to generate the final high-resolution video sequence

[0075] In some embodiments, the training method of the video super-resolution reconstruction network model includes:

[0076] Obtain a low-resolution video sequence dataset;

[0077] The video super-resolution reconstruction network model is trained and tested using the data set to obtain a tested and trained video super-resolution reconstruction network model.

[0078] In some embodiments, the loss function L1(x) during the video super-resolution reconstruction network model training process is:

[0079]

[0080] Where x represents the set of weighted and biased parameters, i represents the number of iterations during training, and m represents the number of training video frames. High-resolution video sequences output by the model, Represents the low-resolution video sequence input to the model, F(.) represents the predicted value of generating high-resolution video, and ||*|| represents the norm.

[0081] In some embodiments, as Figure 1 As shown, an image super-resolution reconstruction method based on a D3D convolutional group fusion network includes the following steps:

[0082] Step 1: Create a video dataset:

[0083] The dataset can be a standard one or self-constructed. For example, the standard vid4 dataset can be used for training and testing. This dataset contains video frame sequences for four scenes: calendar, city, foliage, and walk. Each scene contains 41, 34, 49, and 47 video frames. The high-resolution video frames are downsampled fourfold using a Gaussian blur with a standard deviation of σ = 1.6 to generate the corresponding low-resolution video frames.

[0084] Step 2: Build a video super-resolution network model, such as Figure 2 As shown, the network structure includes a time grouping module, a C3D shallow feature extraction module, a D3D convolution group fusion module, an inter-group attention mechanism module and a reconstruction module, including the following steps:

[0085] Step 2.1, build the time grouping module:

[0086] The adjacent 2N frames are divided into N groups according to the time distance to the reference frame, and the original sequence is reordered into {G1,...G n}, n∈[i:N], where From the previous frame Reference Frame and the next frame The subsequence is composed of 7 frames of input video sequence. For example, Represents the reference frame, and the other frames are adjacent frames. These 7 frames are divided into three groups, which can be divided into different groups according to the different frame rates. These three groups.

[0087] Step 2.2: Construct C3D shallow feature extraction and D3D convolution layer;

[0088] First, C3D convolution is used to perform preliminary feature extraction and time alignment on the video frames grouped by time. The extracted feature maps are then sent to the D3D group fusion module, where deformable three-dimensional convolution (D3D) is used for feature fusion. Further time alignment is performed and the images are fed into the subsequent deep fusion module. The construction method of D3D convolution is as follows:

[0089] First, we construct a C3D convolution. The specific implementation can be divided into the following two steps: 1) perform a 3D convolution kernel sampling on the input feature x and 2) perform a weighted summation of the sampled values ​​using the function w. The features transferred by the 3×3×3 convolution kernel with a dilation factor of 1 can be expressed as follows:

[0090]

[0091] Among them, a position in the output feature can be represented by p0, pn Represents the nth value in the 3×3×3 convolutional sampling network.

[0092] Then, D3D convolution is constructed based on C3D convolution, and the size of the sampling network is set to N = 27. The input features of size C×T×W×H are first input to C3D to generate offset features of size 2N×T×W×H. For two-dimensional spatial deformation, the number of channels of these offset features is generally set to 2N. The learned offset is then used to guide the ordinary C3D sampling network to perform spatial deformation, thereby generating a D3D sampling network, and finally using the D3D sampling network to generate output features. The above process can be expressed as follows:

[0093]

[0094] Where Δp n Indicates the offset corresponding to the nth value in the 3×3×3 convolutional sampling network. The offset is usually a decimal, so a more accurate value needs to be generated through bilinear interpolation.

[0095] Step 2.3, construct the D3D convolution group fusion module;

[0096] In this module, each group will deploy an intra-group fusion module. The D3D convolution intra-group fusion module includes a spatial feature extractor, a D3D convolution layer, and a two-dimensional dense block;

[0097] Each unit of the spatial feature extractor consists of a 3×3 convolution layer, a batch normalization BN layer and a Relu activation function. It takes the video feature F extracted by the C3D shallow feature extraction module as input and outputs the spatial feature after processing. All convolutional layers have appropriate expansion rates to simulate the unique inter-frame motion of each group, and the frame rate of each group determines the expansion rate of the convolutional layer. The larger the time difference between frames, the larger the motion level, and the smaller the time difference, the smaller the motion level.

[0098] The spatial features Use 5 D3D convolution residual blocks with 3×3×3 convolution kernels to fuse spatiotemporal features to obtain fused features.

[0099] The fusion features The two-dimensional dense block is fed into the two-dimensional dense block, and the intra-group features are extracted by applying 18 two-dimensional units in the two-dimensional dense block to generate the intra-group features. The information between frames within the group is deeply integrated, and the spatiotemporal information is efficiently utilized.

[0100] Step 2.4, construct the inter-group attention mechanism module;

[0101] For each group, in the corresponding feature map After applying a 3×3 convolutional layer, the feature map of a channel can be calculated Afterwards, these generated feature maps F1 a 、F2 a 、F3 a are further connected, and the softmax function along the time axis is applied to each position, across channels, to calculate the temporal attention feature map.

[0102] Connect the intermediate graphs of each group and calculate the attention feature map M(x,y) through the softmax function along the time axis:

[0103]

[0104] For each group, the attention weighted features It can be calculated by the following formula:

[0105]

[0106] Among them, M n (x,y) j Represents the temporal attention mask at (x,y) j The weight of the position, Represents the intra-group features generated in the intra-group fusion module, and ⊙ represents the multiplication of the corresponding elements one by one.

[0107] After calculating the attention feature maps, these feature maps are concatenated along the time axis and input into a 3D dense block. At the same time, a convolution layer containing a 1×3×3 convolution kernel is inserted at the end of the 3D dense block to reduce the number of channels. Then a 2D dense block is placed below for further fusion.

[0108] Step 2.5, build the reconstruction module:

[0109] The attention feature map is fed into six cascaded residual blocks and a sub-pixel convolutional layer for reconstruction, and the corresponding residual map R is generated after processing. t , and the original video residual map generated by bicubic interpolation upsampling Add together to generate the final high-resolution video frame The formula is as follows:

[0110]

[0111] Step 3: training a video super-resolution network model based on the D3D convolution group fusion module;

[0112] During training, the training data was augmented by flipping and rotating with a probability of 0.5. The network took seven adjacent low-resolution frames as input. The model was supervised by a pixel-wise L1 loss and optimized using the Adam optimizer with β1 = 0.9 and β2 = 0.997. During training, weight decay was set to 5×10⁻¹⁴. The learning rate was initially set to 10⁻¹³ and then multiplied by 0.9 every 10 iterations until 300 iterations. The mini-batch size was set to 64.

[0113] Step 4: Test the training data and reconstruct the video frame sequence.

[0114] The processed test vid4 dataset was fed into the trained network. Common image evaluation metrics—peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM)—were used to quantitatively analyze the reconstructed images and measure the VSR reconstruction performance. Furthermore, we used parameters (Params), network complexity (Flops), and runtime (Times) to measure network performance.

[0115] To verify the effectiveness of this method, we compared it with seven existing advanced video super-resolution methods at the x4 method multiple. Experimental data show that on the vid4 dataset, the PSNR and SSIM values ​​of this method are better than those of other models. In the visualization results, the video images restored by this network model can show more texture information, the edge information of the image can be seen more clearly, and the image resolution is significantly improved.

[0116] Example 2

[0117] In a second aspect, this embodiment provides a video super-resolution reconstruction device based on a D3D convolutional group fusion network, including a processor and a storage medium;

[0118] The storage medium is used to store instructions;

[0119] The processor is configured to operate according to the instructions to execute the steps of the method according to embodiment 1.

[0120] Example 3

[0121] In a third aspect, this embodiment provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Example 1 are implemented.

[0122] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

Claims

1. A video super-resolution reconstruction method based on a D3D convolutional group fusion network, characterized in that: The method comprises: Obtaining a low-resolution video sequence to be reconstructed; Inputting the low-resolution video sequence to be reconstructed into the video super-resolution reconstruction network model trained for testing; Obtaining a video super-resolution reconstruction result, i.e., a high-resolution video sequence, according to the output of the video super-resolution reconstruction network model; The video super-resolution reconstruction network model includes a time grouping module, a C3D shallow feature extraction module, a D3D convolution group fusion module, an inter-group attention mechanism module and a reconstruction module; The temporal grouping module is used to group the input low-resolution video sequence according to different frame rates to obtain N groups of video subsequences; The C3D shallow feature extraction module is used to perform preliminary feature extraction and time alignment on the video subsequences grouped by the time grouping module to obtain video features F, and send them to the D3D convolution group fusion module; The D3D convolution group fusion module is used to extract spatial features from video features using a spatial feature extractor to obtain spatial features, and then use five D3D convolution residual blocks with a convolution kernel of 3×3×3 to fuse the spatial features with spatiotemporal features to obtain fused features, and use two-dimensional dense blocks to extract group features from the fused features to generate group features within the group. The inter-group attention mechanism module is used to analyze the characteristics of multiple groups within the group. Perform temporal attention fusion to generate attention feature map M n (x,y) j , and sent to the reconstruction module; The reconstruction module is used to perform bicubic interpolation upsampling on the original low-resolution video sequence to be reconstructed to generate the original video residual image The attention feature map is fed into six cascaded residual blocks and a sub-pixel convolutional layer for reconstruction, and the corresponding residual map R is generated after processing. t ; The residual graph R t and the original video residual map Add together to generate the final high-resolution video sequence 2. The video super-resolution reconstruction method based on the D3D convolutional group fusion network according to claim 1 is characterized in that: The training method of the video super-resolution reconstruction network model includes: Obtain a low-resolution video sequence dataset; The video super-resolution reconstruction network model is trained and tested using the data set to obtain a tested and trained video super-resolution reconstruction network model.

3. The video super-resolution reconstruction method based on the D3D convolutional group fusion network according to claim 2 is characterized in that: The loss function L1(x) during the training of the video super-resolution reconstruction network model is: Where x represents the set of weighted and bias parameters, i represents the number of iterations during training, and m represents the number of training video frames. High-resolution video sequences output by the model, Represents the low-resolution video sequence input to the model, F(.) represents the predicted value of generating high-resolution video, and ||*|| represents the norm.

4. The video super-resolution reconstruction method based on D3D convolutional group fusion network according to claim 2 is characterized in that: The obtaining of a low-resolution video sequence dataset comprises: The dataset uses a standard dataset or is collected and constructed by yourself; If the standard vid4 and dataset are used for training and testing; the dataset contains video frame sequences of four scenes: calendar, city, foliage, and walk. Each scene contains 41, 34, 49, and 47 frames of video images. The high-resolution video frames are downsampled four times using Gaussian blur with a standard deviation of σ=1.6 to generate the corresponding low-resolution video frames.

5. The video super-resolution reconstruction method based on D3D convolutional intra-group fusion network according to claim 1, characterized in that: The processing process of the time grouping module includes: The adjacent 2N frames are divided into N groups according to the temporal distance to the reference frame, and the original video sequence is reordered as {G1,...G n }, n∈[i:N], where From the previous frame Reference Frame and the next frame where L represents the symbol of the low-resolution video sequence.

6. The video super-resolution reconstruction method based on D3D convolutional intra-group fusion network according to claim 1, characterized in that: The method for constructing the D3D convolution group fusion module includes: The D3D convolution group fusion module includes a spatial feature extractor, a D3D convolution layer and a two-dimensional dense block; Each unit of the spatial feature extractor consists of a 3×3 convolution layer, a batch normalization BN layer and a Relu activation function. It takes the video feature F extracted by the C3D shallow feature extraction module as input and outputs the spatial feature after processing. The spatial features Use 5 D3D convolution residual blocks with 3×3×3 convolution kernels to fuse spatiotemporal features to obtain fused features. The fusion features The two-dimensional dense block is fed into the two-dimensional dense block, and the intra-group features are extracted by applying 18 two-dimensional units in the two-dimensional dense block to generate the intra-group features.

7. The video super-resolution reconstruction method based on D3D convolutional intra-group fusion network according to claim 1, characterized in that: The processing process of the C3D shallow feature extraction module includes: 1) Perform three-dimensional convolution kernel sampling on the input feature x to obtain the sampling value; 2) Perform weighted summation of the sampled values ​​using the function w; The features passed through the 3×3×3 convolution kernel with a dilation factor of 1 are expressed as follows: Among them, y(p0) represents the generated output feature, a position in the output feature can be represented by p0, p n Represents the nth value in the 3×3×3 convolutional sampling network, where N represents the number of iterations; The processing process of the D3D convolution residual block of the fusion module in the D3D convolution group includes: Where Δp n Indicates the offset corresponding to the nth value in the 3×3×3 convolutional sampling network; the offset is usually a decimal, so a more precise value needs to be generated by bilinear interpolation.

8. The video super-resolution reconstruction method based on D3D convolutional intra-group fusion network according to claim 1, characterized in that: The inter-group attention mechanism module includes: For group characteristics within the group After applying a 3×3 convolutional layer, the feature map of a channel is calculated. Generated feature map F1 a 、 are further connected, and the softmax function along the time axis is applied to each position, across channels, to calculate the temporal attention feature map M(x,y); Connect the intermediate graphs of each group and calculate the attention feature map M(x,y) through the softmax function along the time axis; For each group, the attention weighted features It can be calculated by the following formula: Among them, M n (x,y) j Represents the temporal attention mask at (x,y) j The weight of the position, Represents the group characteristics within the group, and ⊙ represents the multiplication of corresponding elements one by one.

9. A video super-resolution reconstruction device based on a D3D convolutional group fusion network, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images