A multi-image super-resolution reconstruction method, device, equipment, medium and product

Abstract

This application discloses a multi-image super-resolution reconstruction method, apparatus, device, medium, and product, relating to the field of image super-resolution reconstruction. The method includes: constructing a multi-image super-resolution reconstruction model based on an inter-frame interleaved attention mechanism, an inter-window interleaved attention mechanism with overlapping regions, and a VMamba model. The inter-frame interleaved attention mechanism includes N+1 inter-frame attention mechanisms, with the outputs of the first N inter-frame attention mechanisms connected to the input of the N+1th inter-frame attention mechanism. The inter-window interleaved attention mechanism with overlapping regions replaces the Swing Transformer Block with an ST Block, where the ST Block includes first and second convolutions. The first input of the second convolution is connected to the output of the first convolution, and the second input of the second convolution is connected to the second input of the first convolution. This application can improve the quality of the reconstructed image.

Description

Technical Field

[0001] This application relates to the field of image super-resolution reconstruction, and in particular to a method, apparatus, device, medium, and product for multi-image super-resolution reconstruction. Background Technology

[0002] Image super-resolution reconstruction is a computer vision technique that upscales low-resolution images to high-resolution images. It has wide applications in medical imaging, satellite remote sensing, and image enhancement. Compared to single-image super-resolution reconstruction methods, multi-image super-resolution reconstruction methods acquire multiple frames of the same scene and utilize sub-pixel displacement information within these frames, resulting in better super-resolution reconstruction performance.

[0003] Traditional image super-resolution reconstruction methods utilize convolutional neural networks as the basic structure to build multi-image super-resolution reconstruction models. However, the reconstructed images often suffer from detail loss, resulting in unsatisfactory image quality. While traditional multi-image super-resolution models based on Transformers can make fuller use of global information, their computational complexity increases significantly with the number and resolution of images, leading to training difficulties and ultimately, unsatisfactory reconstructed image quality.

[0004] In summary, there is a need for a multi-image super-resolution reconstruction method that can improve the quality of reconstructed images. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, medium, and product for multi-image super-resolution reconstruction, which can improve the quality of reconstructed images.

[0006] To achieve the above objectives, this application provides the following solution.

[0007] In a first aspect, this application provides a multi-image super-resolution reconstruction method, comprising: constructing a multi-image super-resolution reconstruction model; the multi-image super-resolution reconstruction model comprising: a feature extraction module, a feature alignment module, and a feature fusion and reconstruction module connected in sequence; the feature extraction module is constructed based on an inter-frame interleaved attention mechanism and an inter-window interleaved attention mechanism with overlapping regions; the inter-frame interleaved attention mechanism comprises N+1 inter-frame attention mechanisms, wherein the input ends of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the output ends of the first N inter-frame attention mechanisms are all connected to the input end of the N+1 inter-frame attention mechanism; the inter-window interleaved attention mechanism with overlapping regions is to replace the SwinTransformer Block in the Swin Transformer with an ST Block, the ST Block comprising a first convolution and a second convolution; the first input end of the second convolution is connected to the output end of the first convolution; the second input end of the second convolution is connected to the second input end of the first convolution, serving as the first input end of the ST Block; the first input end of the first convolution serves as the second input end of the ST Block; the feature fusion and reconstruction module is constructed based on a VMamba model.

[0008] The reconstructed image is obtained by inputting multiple frames of images to be reconstructed into the multi-image super-resolution reconstruction model.

[0009] Optionally, the multi-image super-resolution reconstruction model further includes: a first upsampling layer, the input of which is used to input multiple frames of images to be reconstructed, and the output of which is connected to the second input of the feature fusion and reconstruction module.

[0010] Optionally, the feature extraction module specifically includes: an optical flow prediction network and a composite interleaved attention subsystem; the input terminals of the optical flow prediction network and the composite interleaved attention subsystem are both used to input multiple frames of images to be reconstructed, and the output terminals of the optical flow prediction network and the composite interleaved attention subsystem are both connected to the input terminal of the feature alignment module.

[0011] The composite interleaved attention subsystem includes: multiple sequentially connected composite interleaved attention modules; each composite interleaved attention module includes: a shallow neural network, a first inter-frame interleaved attention mechanism, an inter-window interleaved attention mechanism with overlapping regions, a first merging operation, a first residual network, and a second inter-frame interleaved attention mechanism.

[0012] The input terminal of the shallow neural network is the input terminal of the composite interleaved attention module; the output terminal of the shallow neural network is connected to the input terminal of the first inter-frame interleaved attention mechanism and the input terminal of the inter-window interleaved attention mechanism with overlapping regions, respectively. The output terminals of the shallow neural network, the first inter-frame interleaved attention mechanism, and the inter-window interleaved attention mechanism with overlapping regions are all connected to the input terminal of the first merging operation. The output terminal of the first merging operation is connected to the input terminal of the first residual network. The output terminal of the first residual network is connected to the input terminal of the second inter-frame interleaved attention mechanism. The output terminal of the second inter-frame interleaved attention mechanism is the output terminal of the composite interleaved attention module.

[0013] Optionally, the first residual network includes: a second merging operation and a first-layer normalization unit and a multilayer perceptron connected in sequence.

[0014] The input of the first normalization unit is connected to the output of the first merging operation. The input of the second merging operation is connected to the output of the multilayer perceptron and the output of the first merging operation, respectively. The output of the second merging operation is connected to the input of the second inter-frame interleaved attention mechanism.

[0015] Optionally, the feature fusion and reconstruction module specifically includes: a residual VMamba subsystem and a third merging operation; the input of the residual VMamba subsystem is connected to the output of the feature alignment module; the output of the residual VMamba subsystem and the output of the first upsampling layer are both connected to the input of the third merging operation.

[0016] The residual VMamba subsystem includes: multiple residual VMamba modules connected in sequence; each residual VMamba module includes: a second residual network, a third residual network, and a second upsampling layer connected in sequence; the second residual network is constructed based on the VMamba model; the input of the second residual network is the input of the residual VMamba module, and the output of the second upsampling layer is the output of the residual VMamba module.

[0017] Optionally, the third residual network includes: a first skip connection operation, a fourth merge operation, and a second normalization unit, a first convolutional layer, and a channel attention network connected in sequence; the channel attention network has a compression mechanism.

[0018] The input of the second-layer normalization unit and the input of the first jump connection operation are both connected to the output of the second residual network; the output of the first jump connection operation and the output of the channel attention network are both connected to the input of the fourth merging operation, and the output of the fourth merging operation is connected to the input of the second upsampling layer.

[0019] Secondly, this application provides a multi-image super-resolution reconstruction apparatus, comprising: a model building module for constructing a multi-image super-resolution reconstruction model; the multi-image super-resolution reconstruction model comprising: a feature extraction module, a feature alignment module, and a feature fusion and reconstruction module connected in sequence; the feature extraction module is constructed based on an inter-frame interleaved attention mechanism and an inter-window interleaved attention mechanism with overlapping regions; the inter-frame interleaved attention mechanism comprises N+1 inter-frame attention mechanisms, wherein the input ends of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the output ends of the first N inter-frame attention mechanisms are all connected to the input end of the N+1 inter-frame attention mechanism; the inter-window interleaved attention mechanism with overlapping regions is to replace the Swin Transformer Block in SwinTransformer with an ST Block, the ST Block comprising a first convolution and a second convolution; the first input end of the second convolution is connected to the output end of the first convolution; the second input end of the second convolution is connected to the second input end of the first convolution, serving as the first input end of the ST Block; the first input end of the first convolution is the second input end of the ST Block; the feature fusion and reconstruction module is constructed based on a VMamba model.

[0020] The reconstruction module is used to input multiple frames of images to be reconstructed into the multi-image super-resolution reconstruction model to obtain reconstructed images.

[0021] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-image super-resolution reconstruction method described in any of the preceding claims.

[0022] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-image super-resolution reconstruction method described in any of the preceding claims.

[0023] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the multi-image super-resolution reconstruction method described in any of the preceding claims.

[0024] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a multi-image super-resolution reconstruction method, apparatus, device, medium, and product. Traditional image super-resolution reconstruction methods construct multi-image super-resolution reconstruction models based on convolutional neural networks. Because the traditional convolutional kernels of convolutional neural networks are difficult to capture the global features of an image and difficult to model long-range dependencies between multiple frames, cross-scale information interaction is insufficient, resulting in the loss of reconstruction details. In the multi-image super-resolution reconstruction model of this application, the feature extraction module is based on inter-frame interleaved attention mechanisms and inter-window interleaved attention mechanisms with overlapping regions. The inter-frame interleaved attention mechanism includes N+1 inter-frame attention mechanisms. The input terminals of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the output terminals of the first N inter-frame attention mechanisms are all connected to the input terminal of the N+1 inter-frame attention mechanism. Because this residual-like network structure can pass more information to the next module, it can achieve a better effect of preserving useful information. The Swing Transformer Block in the Swing Transformer is replaced with ST. The Block mechanism employs an inter-window staggered attention mechanism with overlapping regions, allowing it to focus on features outside the window of interest. This achieves the effect of focusing on offset features. Combining the inter-frame attention mechanism with the inter-window staggered attention mechanism with overlapping regions can fully extract features from each frame that are beneficial for multi-image super-resolution reconstruction. The VMamba model can effectively capture long-distance dependencies of features. The feature fusion and reconstruction module is built based on the VMamba model, which can fully utilize the multi-frame image features extracted by the feature extraction module. Therefore, the multi-image super-resolution reconstruction model provided in this application can solve the problem of lost reconstruction details. Moreover, the VMamba model has a linear structure, which can maintain linear complexity. Compared with the Transformer-based model, the computational complexity is reduced, thus solving the problem of difficult training. Therefore, this application can improve the quality of reconstructed images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a multi-image super-resolution reconstruction method provided in an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of the inter-frame interleaved attention mechanism.

[0028] Figure 3 This is a schematic diagram illustrating the principle of inter-window attention mechanism with overlapping regions.

[0029] Figure 4 This is a schematic diagram of the structure of a multi-image super-resolution reconstruction model.

[0030] Figure 5 This is a schematic diagram of the workflow of the composite interleaved attention module.

[0031] Figure 6 This is a schematic diagram of the feature alignment module.

[0032] Figure 7 This is a schematic diagram illustrating the workflow of the residual VMamba module.

[0033] Figure 8 The super-resolution effect of the multi-image super-resolution reconstruction model provided in this application on an artificially synthesized dataset is shown in the figure.

[0034] Figure 9 The super-resolution reconstruction method for multiple images provided in this application is shown in super-resolution images on a real-world dataset.

[0035] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] In recent years, deep learning-based multi-image super-resolution reconstruction methods have achieved more accurate and detailed super-resolution results by learning the mapping relationship between high- and low-resolution images and learning the implicit reconstruction process through an end-to-end model. Based on this, this application provides a multi-image super-resolution reconstruction method. In an exemplary embodiment, as shown... Figure 1 As shown, the multi-image super-resolution reconstruction method includes steps 201 to 202.

[0039] Step 201: Construct a multi-image super-resolution reconstruction model; such as Figure 4As shown, the multi-image super-resolution reconstruction model includes: a feature extraction module, a feature alignment module, and a feature fusion and reconstruction module connected in sequence; the feature extraction module is constructed based on an inter-frame interleaved attention mechanism and an inter-window interleaved attention mechanism with overlapping regions; as shown... Figure 2 As shown, the inter-frame interleaved attention mechanism includes N+1 inter-frame attention mechanisms. The inputs of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the outputs of the first N inter-frame attention mechanisms are all connected to the input of the N+1th inter-frame attention mechanism. The inter-window interleaved attention mechanism with overlapping regions replaces the Swing Transformer Block in the Swing Transformer with the ST Block, as shown below. Figure 3 As shown, the ST Block includes a first convolution and a second convolution; the first input of the second convolution is connected to the output of the first convolution; the second input of the second convolution is connected to the second input of the first convolution, serving as the first input of the ST Block for inputting the K / V matrix with overlapping regions; the first input of the first convolution is the second input of the ST Block for inputting the Q-sampling region; the feature fusion and reconstruction module is built based on the VMamba model. The principle of the inter-window staggered attention mechanism with overlapping regions is as follows: Figure 3 As shown, by sampling with overlapping K / V matrices, the Q-sampled region is allowed to establish a connection with the regions outside it, providing attention to misaligned features between multiple frames.

[0040] Step 202: Input multiple frames of images to be reconstructed into the multi-image super-resolution reconstruction model to obtain reconstructed images.

[0041] In another exemplary embodiment of this application, the ST Block further includes: a layer normalization layer and a fully connected layer connected in sequence, wherein the layer normalization layer is connected to the output of the second convolution.

[0042] In another exemplary embodiment of this application, the feature extraction module specifically includes: an optical flow prediction network and a composite interleaved attention subsystem; the input terminals of the optical flow prediction network and the composite interleaved attention subsystem are both used to input multiple frames of images to be reconstructed, and the output terminals of the optical flow prediction network and the composite interleaved attention subsystem are both connected to the input terminal of the feature alignment module. The composite interleaved attention subsystem includes: multiple sequentially connected composite interleaved attention modules; each composite interleaved attention module includes: a shallow neural network, a first inter-frame interleaved attention mechanism, an inter-window interleaved attention mechanism with overlapping regions, a first merging operation, a first residual network, and a second inter-frame interleaved attention mechanism. The input terminal of the shallow neural network is the input terminal of the composite interleaved attention module; the output terminal of the shallow neural network is connected to the input terminal of the first inter-frame interleaved attention mechanism and the input terminal of the inter-window interleaved attention mechanism with overlapping regions, respectively. The output terminals of the shallow neural network, the first inter-frame interleaved attention mechanism, and the inter-window interleaved attention mechanism with overlapping regions are all connected to the input terminal of the first merging operation. The output terminal of the first merging operation is connected to the input terminal of the first residual network. The output terminal of the first residual network is connected to the input terminal of the second inter-frame interleaved attention mechanism. The output terminal of the second inter-frame interleaved attention mechanism is the output terminal of the composite interleaved attention module.

[0043] In another exemplary embodiment of this application, a pre-trained SpyNet network is used as the optical flow prediction network to predict optical flow and obtain the optical flow in three dimensions of the i-th frame image. .

[0044] In another exemplary embodiment of this application, the feature extraction module further includes two convolutional layers. Multiple frames of images to be reconstructed are input into the optical flow prediction network through one of the convolutional layers, and multiple frames of images to be reconstructed are input into the composite interleaved attention subsystem through the other convolutional layer.

[0045] In another exemplary embodiment of this application, the composite interleaved attention module further includes: a third-layer normalization unit, wherein the output of the shallow neural network is connected to the input of the first inter-frame interleaved attention mechanism, the input of the inter-window interleaved attention mechanism with overlapping regions, and the input of the first merging operation through the third-layer normalization unit.

[0046] In another exemplary embodiment of this application, the shallow neural network is a second convolutional layer.

[0047] In another exemplary embodiment of this application, the first residual network includes: a second merging operation and a first-layer normalization unit and a multilayer perceptron connected in sequence. The input of the first-layer normalization unit is connected to the output of the first merging operation, the input of the second merging operation is connected to the output of the multilayer perceptron and the output of the first merging operation, respectively, and the output of the second merging operation is connected to the input of the second inter-frame interleaved attention mechanism.

[0048] The composite interleaved attention module first uses a shallow neural network to perform preliminary feature extraction on each input frame. Then, it applies a first inter-frame interleaved attention mechanism and a window-to-window interleaved attention mechanism with overlapping regions in parallel to the obtained shallow features, and then merges the resulting features. Next, it uses a first-layer normalization unit and a multilayer perceptron, employing a residual network to obtain new features. Finally, it uses a second inter-frame interleaved attention mechanism to obtain the final features of this module, such as... Figure 5 As shown, the specific working process of the composite interleaved attention module is as follows: for the input... Frame size is The low-resolution image is first processed by a convolutional neural network (second convolutional layer) to extract shallow features from each frame of the low-resolution image, resulting in a feature vector of dimension 1. Features Layer normalization is performed on shallow features. Then, the large window Swing Transdormer operation LWT() (i.e., the inter-window interleaved attention mechanism with overlapping regions) and the inter-frame interleaved attention operation CFA() are performed in parallel to obtain the features. and The formula is expressed as follows: and .

[0049] Features , and Merging, the formula is as follows , This represents the control weights for the output features of the CFA operation, followed by another layer normalization. The features are then input into a multilayer perceptron, expressed as follows: .

[0050] The obtained features Features obtained from parallel operations After merging, perform another inter-frame attention operation to obtain features. The formula is expressed as follows: .

[0051] In another exemplary embodiment of this application, the feature alignment module is an optical flow-guided alignment module used to align multiple frames of images in the feature domain. For example... Figure 6 As shown, this process mainly uses a Flow-Guided Deformable Convolutional Network (FGDCN). Specifically, the optical flow obtained from the optical flow prediction network is used to correct the features of each frame of the image. The formula is as follows: ,in, , , and Let represent the image features of the i-th frame after correction, the features of the i-th frame output by the optical flow prediction network, the composite interleaved attention subsystem, and the optical flow of the i-th frame obtained by the optical flow prediction network, respectively.

[0052] Then, the corrected features Features of a reference image (which is one frame out of a set of multiple images, and is manually selected). Optical flow of the corresponding frame Merging, using convolutional neural networks Obtain offset features The formula is expressed as follows: .

[0053] Finally, the features of each frame are aligned using a deformable convolutional neural network (DCN()), expressed as follows: , This represents the features of the i-th frame image after alignment.

[0054] like Figure 4 As shown in another exemplary embodiment of this application, the multi-image super-resolution reconstruction model further includes:

[0055] The first upsampling layer has an input terminal for inputting multiple frames of images to be reconstructed, and its output terminal is connected to the second input terminal of the feature fusion and reconstruction module.

[0056] In another exemplary embodiment of this application, the multi-image super-resolution reconstruction model further includes: a third convolutional layer, through which multiple frames of images to be reconstructed are input into the first upsampling layer.

[0057] In another exemplary embodiment of this application, the feature fusion and reconstruction module specifically includes: a residual VMamba subsystem and a third merging operation; the input of the residual VMamba subsystem is connected to the output of the feature alignment module; the output of the residual VMamba subsystem and the output of the first upsampling layer are both connected to the input of the third merging operation.

[0058] The residual VMamba subsystem includes multiple sequentially connected residual VMamba modules. Each residual VMamba module includes a second residual network, a third residual network, and a second upsampling layer, all sequentially connected. The second residual network is constructed based on the VMamba model. The input of the second residual network is the input of the residual VMamba module, and the output of the second upsampling layer is the output of the residual VMamba module. The VMamba model is derived by applying the VisionState-Space Module (VSSM) to deep learning. The VMamba model processes features using both linear layers combined with SiLU activation layers and linear layers, depthwise separable convolutions, SiLU activation layers, 2D-SSM, and linear layers. The features obtained from both methods are then merged and output as features through a linear layer. The second upsampling layer is a pixel shuffle.

[0059] In another exemplary embodiment of this application, the third residual network includes: a first skip connection operation, a fourth merging operation, and a second normalization unit, a first convolutional layer, and a channel attention network connected in sequence; the channel attention network has a compression mechanism. The input of the second normalization unit and the input of the first skip connection operation are both connected to the output of the second residual network; the output of the first skip connection operation and the output of the channel attention network are both connected to the input of the fourth merging operation, and the output of the fourth merging operation is connected to the input of the second upsampling layer.

[0060] In another exemplary embodiment of this application, the second residual network includes: a second jump connection operation, a fifth merge operation, and a fourth layer normalization unit and a VMamba model connected in sequence; the input of the fourth layer normalization unit and the input of the second jump connection operation are the inputs of the second residual network; the output of the second jump connection operation and the output of the VMamba model are connected to the input of the fifth merge operation.

[0061] The residual VMamba module in this application first organizes the features output by the feature alignment module through a fourth-layer normalization unit, and then uses the VMamba model to process the features in parallel. The merged features are then passed through a linear layer as the output features of the VMamba model. The features are then fed into a second-layer normalization unit, a first convolutional layer, and a compressed channel attention network to achieve feature fusion. Finally, the fused features are used for super-resolution reconstruction via PixelShuffle, such as... Figure 7 As shown, the specific working process of the residual VMamba module is as follows: First, ... Features after frame image alignment Organized into The form. Features After layer normalization, the data is input into the VMamba model. This process uses a residual network with a learnable parameter. The formula for controlling the residual ratio is as follows: , This indicates the second-hop join operation, which, within the VMamba model, uses two parallel branches to process features. One branch takes a linear layer. and SiLU activation function The formula is expressed as follows: The other branch uses linear layers and depthwise separable convolutions. SiLU activation function, two-dimensional state-space model And layer normalization, the formula is expressed as: The output features F1 and F2 are combined with the residual control features, and the formula is expressed as follows: .

[0062] Will and Merging The characteristics of the merger The final feature fusion is achieved through a channel attention network with a compression mechanism. Specifically, layer normalization, convolution (Conv()), and a channel attention network (CA()) with a compression mechanism are performed sequentially to obtain the fused feature map. ,in, This represents the first skip connection operation. Finally, the fused feature map is reconstructed at high resolution using PixelShuffle to obtain the reconstruction result.

[0063] In another exemplary embodiment of this application, the multi-image super-resolution reconstruction model provided in this application is based on deep learning technology and adopts a supervised learning training method to achieve end-to-end multi-image super-resolution reconstruction.

[0064] To demonstrate the effectiveness of the multi-image super-resolution reconstruction model provided in this application, an example is provided that tests the multi-image super-resolution reconstruction model provided in this application and several known multi-image super-resolution reconstruction models using a test dataset. Specifically, the test dataset is the publicly available dataset provided by the NTIRE2022 Burst Super-Resolution Challenge. This dataset provides 300 artificially synthesized low-resolution images and 882 real-world low-resolution image patches for testing. The known multi-image super-resolution reconstruction models are currently among the most advanced methods, including: HighResNet, MFIR, DBSR, BIPNet, BSRT, EBSR, Burstormer, and AFCNet.

[0065] The test results provided in this embodiment are shown in Table 1. Compared with these known models, the image patch size used when training the multi-image super-resolution reconstruction model provided in this application is [size missing]. The image patch size for training other well-known models is The size of the image patch used to train the model has a positive correlation with its performance, but the multi-image super-resolution reconstruction model provided in this application still outperforms other methods in several evaluation metrics. The multi-image super-resolution reconstruction model provided in this application achieves the following super-resolution performance on artificially synthesized datasets: Figure 8 As shown, the super-resolution reconstruction effect on real-world datasets is as follows: Figure 9 As shown.

[0066] Table 1. Results of known multi-image super-resolution reconstruction models and the multi-image super-resolution reconstruction model provided in this application on the NTIRE2022 Burst Super-Resolution Challenge dataset.

[0067]

[0068] Related technologies use convolutional neural network (CNN) structures for modeling, employing traditional CNNs in feature extraction. For feature alignment, they utilize feature-enhanced pyramid-structure deformable CNNs. During reconstruction, features from different levels are connected via long-range connections to improve the utilization of global information in super-resolution reconstruction. However, the limited global feature extraction capabilities of CNN kernels result in limited image quality in high-resolution reconstruction. This application addresses these issues by implementing inter-frame staggered attention mechanisms and inter-window staggered attention mechanisms with overlapping regions, effectively leveraging features from each frame beneficial for multi-image super-resolution reconstruction.

[0069] Related technologies use the Swing Transformer in feature extraction and reconstruction to capture long-range dependencies between global and local information. However, in the feature alignment process, optically flow-guided deformable convolutional neural networks are used, which suffer from insufficient feature alignment due to the fixed-size window of the deformable convolution. This application addresses this issue by using a VMamba model with composite sampling in the feature fusion and reconstruction modules, further focusing on features around the pixels of interest, thus improving the reconstruction results.

[0070] Related technologies employ a novel Transformer structure, employing a hierarchical, multi-scale structure to more effectively align multi-frame features. To enhance the connection between inter-frame information, this method also designs a convolutional neural network-based cyclic sampling module for multi-frame images. However, such methods, which cyclically sample every two frames of multi-frame image features, still fail to adequately address the global information across multiple frames. This application addresses these issues by implementing an inter-frame staggered attention mechanism for multi-frame image feature sampling, thereby improving reconstruction results.

[0071] Based on the same inventive concept, this application also provides a multi-image super-resolution reconstruction apparatus for implementing the multi-image super-resolution reconstruction method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the multi-image super-resolution reconstruction apparatus provided below can be found in the limitations of the multi-image super-resolution reconstruction method described above, and will not be repeated here.

[0072] In an exemplary embodiment, a multi-image super-resolution reconstruction apparatus is provided, comprising: a model building module for constructing a multi-image super-resolution reconstruction model; the multi-image super-resolution reconstruction model comprising: a feature extraction module, a feature alignment module, and a feature fusion and reconstruction module connected in sequence; the feature extraction module is constructed based on an inter-frame interleaved attention mechanism and an inter-window interleaved attention mechanism with overlapping regions; the inter-frame interleaved attention mechanism comprises N+1 inter-frame attention mechanisms, wherein the input terminals of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the output terminals of the first N inter-frame attention mechanisms are all connected to the input terminal of the N+1 inter-frame attention mechanism; the inter-window interleaved attention mechanism with overlapping regions is to replace the Swin Transformer Block in the Swin Transformer with an ST Block, the ST Block comprising a first convolution and a second convolution; the first input terminal of the second convolution is connected to the output terminal of the first convolution; the second input terminal of the second convolution is connected to the second input terminal of the first convolution, serving as the first input terminal of the ST Block; the first input terminal of the first convolution serves as the second input terminal of the ST Block; the feature fusion and reconstruction module is constructed based on a VMamba model.

[0073] The reconstruction module is used to input multiple frames of images to be reconstructed into the multi-image super-resolution reconstruction model to obtain reconstructed images.

[0074] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores multi-image super-resolution reconstruction data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a multi-image super-resolution reconstruction method.

[0075] Those skilled in the art will understand that Figure 10The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0076] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0077] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0078] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0079] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0080] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0081] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-image super-resolution reconstruction method, characterized in that, The multi-image super-resolution reconstruction method includes: A multi-image super-resolution reconstruction model is constructed. This model includes a feature extraction module, a feature alignment module, and a feature fusion and reconstruction module connected sequentially. The feature extraction module is constructed based on an inter-frame interleaved attention mechanism and an inter-window interleaved attention mechanism with overlapping regions. The inter-frame interleaved attention mechanism includes N+1 inter-frame attention mechanisms, where the inputs of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the outputs of the first N inter-frame attention mechanisms are all connected to the input of the N+1 inter-frame attention mechanism. The inter-window interleaved attention mechanism with overlapping regions replaces the Swin Transformer Block in the Swin Transformer with an ST Block. The ST Block includes a first convolution and a second convolution. The first input of the second convolution is connected to the output of the first convolution. The second input of the second convolution is connected to the second input of the first convolution, serving as the first input of the ST Block. The first input of the first convolution is the second input of the ST Block. The feature fusion and reconstruction module is constructed based on the VMamba model. The reconstructed image is obtained by inputting multiple frames of images to be reconstructed into the multi-image super-resolution reconstruction model.

2. The multi-image super-resolution reconstruction method according to claim 1, characterized in that, The multi-image super-resolution reconstruction model also includes: The first upsampling layer has an input terminal for inputting multiple frames of images to be reconstructed, and its output terminal is connected to the second input terminal of the feature fusion and reconstruction module.

3. The multi-image super-resolution reconstruction method according to claim 1, characterized in that, The feature extraction module specifically includes: An optical flow prediction network and a composite interleaved attention subsystem are provided; the input terminals of the optical flow prediction network and the composite interleaved attention subsystem are both used to input multiple frames of images to be reconstructed, and the output terminals of the optical flow prediction network and the composite interleaved attention subsystem are both connected to the input terminal of the feature alignment module. The composite interleaved attention subsystem includes: multiple sequentially connected composite interleaved attention modules; each composite interleaved attention module includes: a shallow neural network, a first inter-frame interleaved attention mechanism, an inter-window interleaved attention mechanism with overlapping regions, a first merging operation, a first residual network, and a second inter-frame interleaved attention mechanism; The input terminal of the shallow neural network is the input terminal of the composite interleaved attention module; the output terminal of the shallow neural network is connected to the input terminal of the first inter-frame interleaved attention mechanism and the input terminal of the inter-window interleaved attention mechanism with overlapping regions, respectively. The output terminals of the shallow neural network, the first inter-frame interleaved attention mechanism, and the inter-window interleaved attention mechanism with overlapping regions are all connected to the input terminal of the first merging operation. The output terminal of the first merging operation is connected to the input terminal of the first residual network. The output terminal of the first residual network is connected to the input terminal of the second inter-frame interleaved attention mechanism. The output terminal of the second inter-frame interleaved attention mechanism is the output terminal of the composite interleaved attention module.

4. The multi-image super-resolution reconstruction method according to claim 3, characterized in that, The first residual network includes: a second merging operation and a first-layer normalization unit and a multilayer perceptron connected in sequence; The input of the first normalization unit is connected to the output of the first merging operation. The input of the second merging operation is connected to the output of the multilayer perceptron and the output of the first merging operation, respectively. The output of the second merging operation is connected to the input of the second inter-frame interleaved attention mechanism.

5. The multi-image super-resolution reconstruction method according to claim 2, characterized in that, The feature fusion and reconstruction module specifically includes: a residual VMamba subsystem and a third merging operation; the input of the residual VMamba subsystem is connected to the output of the feature alignment module; the output of the residual VMamba subsystem and the output of the first upsampling layer are both connected to the input of the third merging operation. The residual VMamba subsystem includes: multiple residual VMamba modules connected in sequence; each residual VMamba module includes: a second residual network, a third residual network, and a second upsampling layer connected in sequence; the second residual network is constructed based on the VMamba model; the input of the second residual network is the input of the residual VMamba module, and the output of the second upsampling layer is the output of the residual VMamba module.

6. The multi-image super-resolution reconstruction method according to claim 5, characterized in that, The third residual network includes: a first skip connection operation, a fourth merge operation, and a second normalization unit, a first convolutional layer, and a channel attention network connected in sequence; the channel attention network has a compression mechanism. The input of the second-layer normalization unit and the input of the first jump connection operation are both connected to the output of the second residual network; the output of the first jump connection operation and the output of the channel attention network are both connected to the input of the fourth merging operation, and the output of the fourth merging operation is connected to the input of the second upsampling layer.

7. A multi-image super-resolution reconstruction device, characterized in that, The multi-image super-resolution reconstruction device includes: A model building module is used to construct a multi-image super-resolution reconstruction model. The multi-image super-resolution reconstruction model includes a feature extraction module, a feature alignment module, and a feature fusion and reconstruction module connected sequentially. The feature extraction module is constructed based on an inter-frame interleaved attention mechanism and an inter-window interleaved attention mechanism with overlapping regions. The inter-frame interleaved attention mechanism includes N+1 inter-frame attention mechanisms, where the inputs of the first N inter-frame attention mechanisms are used to input the image to be reconstructed, and the outputs of the first N inter-frame attention mechanisms are all connected to the input of the N+1 inter-frame attention mechanism. The inter-window interleaved attention mechanism with overlapping regions replaces the Swin Transformer Block in the Swin Transformer with an ST Block. The ST Block includes a first convolution and a second convolution. The first input of the second convolution is connected to the output of the first convolution. The second input of the second convolution is connected to the second input of the first convolution, serving as the first input of the ST Block. The first input of the first convolution is the second input of the ST Block. The feature fusion and reconstruction module is constructed based on a VMamba model. The reconstruction module is used to input multiple frames of images to be reconstructed into the multi-image super-resolution reconstruction model to obtain reconstructed images.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multi-image super-resolution reconstruction method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-image super-resolution reconstruction method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-image super-resolution reconstruction method as described in any one of claims 1-6.

Images