Image super-resolution method and device based on omnidirectional convolution and multi-scale mamba

Abstract

The application discloses an omnidirectional convolution and multi-scale Mamba-based image super-resolution method and relates to the field of image processing, which comprises the following steps: constructing an omnidirectional convolution and multi-scale Mamba-based image super-resolution model, wherein the image super-resolution model comprises a first convolution layer, N omnidirectional convolution-multi-scale Mamba modules and a second convolution layer connected in sequence, and the omnidirectional convolution-multi-scale Mamba module comprises a mobile end inverted bottleneck convolution layer, a multi-scale Mamba module, a first gated convolution feedforward network, an omnidirectional convolution module and a second gated convolution feedforward network connected in sequence; inputting a low-resolution image to be reconstructed into the trained image super-resolution model, first passing through the first convolution layer to extract shallow features, sequentially passing through the N omnidirectional convolution-multi-scale Mamba modules to extract deep features, and reconstructing the deep features by using the second convolution layer to obtain a high-resolution image. The application solves the content fragmentation problem in traditional Mamba calculation.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to an image super-resolution method and apparatus based on omnidirectional convolution and multi-scale Mamba. Background Technology

[0002] Image super-resolution (ISR) is an important task in the field of computer vision, aiming to restore high-resolution images from low-resolution images. Because the restored high-resolution images can reveal more key and subtle information hidden in the original images, it is widely used in computer vision fields such as satellite imaging, video surveillance, and facial recognition.

[0003] Mamba, as a rising backbone of deep learning networks in recent years, has gradually demonstrated performance comparable to Transformer networks, leading to its widespread application in computer vision. Mamba achieves long-range dependency capture through a selective state-space mechanism. Comparative studies have shown that, with the same training data, the Mamba model not only has lower computational cost than Transformer but also avoids the attention sparsity problem of Transformer, achieving better results in image super-resolution tasks. MambaIR, as the first work to apply the Mamba model to image super-resolution, showed strong competitive results compared to various Transformer variants based on image super-resolution. With in-depth research on the Mamba network architecture, various improved Mamba-based network structures have further enhanced its performance, such as global-local collaborative modeling architectures and arbitrary-scale super-resolution implementations. However, the loss of image information by the ordinary depthwise convolutional downsampling module in multi-scale Mamba and the content fragmentation caused by the 2D-to-1D transformation during multi-scale Mamba computation both affect Mamba's performance in image super-resolution. Summary of the Invention

[0004] The purpose of this application is to propose an image super-resolution method and apparatus based on omnidirectional convolution and multi-scale Mamba to address the aforementioned technical problems.

[0005] In a first aspect, the present invention provides an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, comprising the following steps:

[0006] An image super-resolution model based on omnidirectional convolution and multi-scale Mamba is constructed and trained to obtain the trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-end inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0007] The low-resolution image to be reconstructed is input into the trained image super-resolution model. It first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

[0008] Preferably, the multi-scale Mamba module includes a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module; both the first selective state scanning module and the second selective state scanning module adopt a selective state scanning structure based on Mamba.

[0009] In the multi-scale Mamba module, the input features of the multi-scale Mamba module first pass through a first depthwise separable convolutional layer to obtain the first channel features and the second channel features, as shown in the following formula:

[0010] ;

[0011] in, This indicates the first depthwise separable convolutional layer. This represents the input features of a multi-scale Mamba module. and These represent the features of the first channel and the features of the second channel, respectively.

[0012] The first channel feature is input into the Haar wavelet transform downsampling module to obtain the first intermediate feature, as shown in the following formula:

[0013] ;

[0014] in, This is represented as the Haar wavelet transform downsampling module. Indicates the first intermediate feature;

[0015] The first intermediate feature is input into the first selective state scanning module for selective scanning to obtain the first output feature, as shown in the following formula:

[0016] ;

[0017] in, This represents a selective state scan structure based on Mamba. Indicates the first output feature;

[0018] The second channel feature is input into the second selective state scanning module for selective scanning to obtain the second output feature, as shown in the following formula:

[0019] ;

[0020] in, Indicates the second output feature;

[0021] The first output feature is input into the deconvolution upsampling module to obtain the third output feature, as shown in the following formula:

[0022] ;

[0023] in, This is represented as a deconvolution upsampling module. Indicates the third output feature;

[0024] The second output feature is input into the feature fusion module to obtain the output feature of the multi-scale Mamba module, as shown in the following formula:

[0025] ;

[0026] in, This indicates the feature fusion module. This represents the output characteristics of the multi-scale Mamba module.

[0027] Preferably, the deconvolution upsampling module includes a zero-interpolation upsampling layer, a nearest-neighbor interpolation upsampling layer, and a frequency domain information reconstruction module;

[0028] In the deconvolution upsampling module, the input features are first padding to obtain the second intermediate features, as shown in the following equation:

[0029] ;

[0030] in, , This indicates a padding operation. The second intermediate feature is represented by B, C, H, W, and p, which represent the batch size, number of channels, length, width, and padding size of the input features of the Haar wavelet transform downsampling module, respectively. Represents the real number field;

[0031] The second intermediate feature is input into the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer, respectively, to obtain the third and fourth intermediate features, as shown in the following equation:

[0032] ;

[0033] ;

[0034] in, Indicates a zero-interpolation upsampling layer. This indicates the nearest neighbor interpolation upsampling layer. These represent the third and fourth intermediate features, respectively.

[0035] Create The all-zero matrix represents the parameters of the convolution kernel of the point spread function. The center point of the point spread function's convolution kernel is moved from its original position by zero-padding the top-left corner of the all-zero matrix and then performing a cyclic center shift operation. Move to the position of the all-zero matrix The first adjustment feature is obtained. Its expression is:

[0036] ;

[0037] in, This represents zero-filling and cyclic center displacement operations, where 1, C, k1, and k2 represent the batch, number of channels, length, and width of the convolution kernel parameters of the point spread matrix, respectively.

[0038] First, perform a Discrete Fourier Transform on the first adjustment feature to transform it from a time-domain variable to a frequency-domain variable, thus obtaining the optical transfer feature. Then, the optical transmission characteristics are calculated by performing complex conjugation and modulus square calculations to obtain the complex conjugation characteristics. Sum of squared features As shown in the following formula:

[0039] ;

[0040] ;

[0041] ;

[0042] in, This represents the Discrete Fourier Transform operation. This represents complex conjugate computation. Indicates the calculation of the square of the modulus. Represents the field of complex numbers;

[0043] Perform Discrete Fourier Transform on the third and fourth intermediate features respectively, transforming them from time-domain variables to frequency-domain variables, to obtain the fifth intermediate feature. and the sixth intermediate feature Wiener filtering is performed on the fifth intermediate feature based on the complex conjugate feature to obtain the filtered feature. Regularize the sixth intermediate feature to obtain the regularized feature. The filtered features and the regularized features are summed to obtain the regularized frequency domain features. As shown in the following formula:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] in, Represents the regularization parameter;

[0050] Multiplying the regularized frequency domain features with the optical transfer features yields the fused features. As shown in the following formula:

[0051] ;

[0052] The fused features and the modulus-squared features are then subjected to block averaging to obtain the first block average feature. Second block average features As shown in the following formula:

[0053] ;

[0054] ;

[0055] in, This indicates a block averaging operation;

[0056] Perform an inverse filtering operation on the first block average feature and the second block average feature to obtain the inverse filtered feature. As shown in the following formula:

[0057] ;

[0058] Upsampling is performed on the inverse filter features to obtain the extended features. The regularized frequency domain features, complex conjugate features, and extended features are then input into the frequency domain information reconstruction module to reconstruct the frequency domain information, thus obtaining the reconstructed features. As shown in the following formula:

[0059] ;

[0060] The reconstructed features are subjected to an inverse discrete Fourier transform to convert them from frequency domain variables to time domain variables, and the real part is then taken to obtain the second adjusted features. As shown in the following formula:

[0061] ;

[0062] in, This represents the inverse discrete Fourier transform operation. This indicates the operation of taking the real part;

[0063] The second adjusted feature is then depaddinged to obtain the output feature from the deconvolution upsampling. ;

[0064] The Mamba-based selective state scanning structure includes a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module. The two-dimensional selective scanning module includes a multi-directional scanning module and a Mamba-based selective state space computation module.

[0065] In the Mamba-based selective state scan structure, the input features of the Mamba-based selective state scan structure First, the channel dimension is expanded through a linear projection layer to obtain the projection features. As shown in the following formula:

[0066] ;

[0067] in, Indicates a linear projection layer;

[0068] Perform a dimensional split on the projected features to obtain the first-dimensional features. Second-dimensional features Its expression is:

[0069] ;

[0070] in, This indicates a dimension splitting operation;

[0071] The first-dimensional features are sequentially passed through a second-depth separable convolutional layer, a padding operation, and a first SiLU activation function layer to obtain convolutional features. As shown in the following formula:

[0072] ;

[0073] in, This represents the SiLU activation function. This indicates a second-depth separable convolutional layer;

[0074] The convolutional features are input into the two-dimensional selective scanning module. First, the multi-directional scanning module unfolds the convolutional features into four one-dimensional sequences along four directions, which are the first sequence. , second sequence Third sequence Fourth sequence ;in and It unfolds along the positive horizontal direction and the positive vertical direction. and It unfolds in the opposite direction of the horizontal and the opposite direction of the vertical;

[0075] Perform dimension mapping operations on the four one-dimensional sequences respectively, projecting the channel dimension C onto... , obtain the joint tensor As shown in the following formula:

[0076] ;

[0077] in, This indicates a dimension mapping operation. Correct the dimension for the time step. For state parameter dimensions;

[0078] Joint tensors along the channel dimension Split into correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g ;

[0079] Correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g The input is fed into the Mamba-based selective state-space computation module to obtain the enhanced features, as shown in the following equation:

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the input influence matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step. ;

[0086] The second-dimensional feature is passed through a second SiLU activation function layer and multiplied with the enhanced feature to obtain the output feature of the Mamba-based selective state scan structure. As shown in the following formula:

[0087] .

[0088] As a preferred embodiment, the Haar wavelet transform downsampling module includes a channel downsampling convolutional layer, a Haar wavelet transform module, a splicing layer, and a third convolutional layer, wherein the third convolutional layer adopts a convolution operation with a kernel size of 1×1.

[0089] In the Haar wavelet transform downsampling module, the input features are first fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain scaled features. As shown in the following formula:

[0090] ;

[0091] in, This indicates a channel downsampling convolutional layer with a kernel size of 1×1;

[0092] The scaling feature is input into the Haar wavelet transform module, and the scaling feature is... Convolution operations are performed with four different two-dimensional filters to obtain four sub-band features, which are the first sub-band features. Second sub-band characteristics Third sub-band characteristics and fourth sub-band features As shown in the following formula:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, This represents the convolution operation. , , and Let represent the first filter, the second filter, the third filter, and the fourth filter, respectively, and their expressions are as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] Features of the four sub-bands along the channel dimension , , and The input is processed by the stitching layer, and then the inter-channel information is fused through the third convolutional layer to obtain the output features of the Haar wavelet transform downsampling module. As shown in the following formula:

[0103] ;

[0104] in, [·|·] represents a convolution operation with a kernel size of 1×1, and [·|·] represents a concatenation operation along the channel dimension.

[0105] Preferably, the omnidirectional convolution module includes a fourth convolutional layer, a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer. The fourth convolutional layer uses a convolutional operation with a kernel size of 1×1, while the square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer use convolutional operations with kernel sizes of 3×3, 1×11, and 11×1, respectively.

[0106] In the omnidirectional convolution module, the input features are first processed by the fourth convolutional layer to remove redundant channel features, resulting in filtered features, as shown in the following equation:

[0107] ;

[0108] in, This represents the input features of the omnidirectional convolutional module. Indicates filtering features, This indicates a convolution operation with a kernel size of 1×1;

[0109] Padding operations are performed on the filter features to expand them to different dimensional sizes, resulting in the first expanded filter features. Second extended filtering feature and third extended filtering features B, C, H, W, and p represent the batch size, number of channels, length, width, and padding size of the input features in the omnidirectional convolutional module, respectively. Representing the real number field; the first, second, and third extended filter features are input into a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer, respectively, to obtain square features. Horizontal characteristics and vertical features As shown in the following formula:

[0110] ;

[0111] ;

[0112] ;

[0113] in, This indicates a convolution operation with a kernel size of 3×3. This represents a convolution operation with a kernel size of 1×11. This indicates a convolution operation with a kernel size of 11×1;

[0114] The square, horizontal, and vertical features are respectively concatenated with the filtered features using residual connections to obtain the output features of the omnidirectional convolutional module. As shown in the following formula:

[0115] .

[0116] As a preferred embodiment, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt gated convolutional feedforward networks. The gated convolutional feedforward network includes a fifth convolutional layer, a third depthwise separable convolutional layer, a dimension splitting unit, a GELU activation function layer, and a sixth convolutional layer.

[0117] In a gated convolutional feedforward network, the input features are first fed into the fifth convolutional layer for channel-wise projection expansion to obtain the initial projected expanded features. As shown in the following formula:

[0118] ;

[0119] in, This represents the input features of a gated convolutional feedforward network. This indicates that a channel-wise projection expansion operation is performed using the fifth convolutional layer. B, C, H, and W represent the batch size, number of channels, length, and width of the input features of the gated convolutional feedforward network, respectively. Represents the real number field;

[0120] After padding the initial projected extended features, a third depthwise separable convolutional layer is used for feature extraction to obtain the first projected extended features. As shown in the following formula:

[0121] ;

[0122] in, This indicates a third-depth separable convolutional layer;

[0123] Extend the first projection feature The input is fed into the dimension segmentation unit, and average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features The second projection extension feature After being input into the GELU activation function layer, it is then combined with the third projection extended features. Multiplying the features yields the fourth projective extended feature. As shown in the following formula:

[0124] ;

[0125] in, Represents the GELU activation function layer;

[0126] Extend the fourth projection feature The input is fed into the sixth convolutional layer for channel-wise projection reduction, yielding the output features of the gated convolutional feedforward network. As shown in the following formula:

[0127] ;

[0128] in, This indicates the sixth convolutional layer.

[0129] Secondly, the present invention provides an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, comprising:

[0130] The model building module is configured to build and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba, resulting in a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-side inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0131] The reconstruction module is configured to input the low-resolution image to be reconstructed into a trained image super-resolution model. The image first passes through a first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. Finally, the depth features are reconstructed using a second convolutional layer to obtain the corresponding high-resolution image.

[0132] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0133] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0134] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0135] Compared with the prior art, the present invention has the following beneficial effects:

[0136] (1) In the image super-resolution model of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba mentioned in this invention, the Haar wavelet transform downsampling module is used to replace the depth convolution downsampling in the traditional multi-scale Mamba module, which realizes lossless information preservation, so that the image entering the multi-scale scan retains more information, and the multi-scale Mamba module proposed in this invention can capture more long-distance features.

[0137] (2) The image super-resolution method based on omnidirectional convolution and multi-scale Mamba mentioned in this invention introduces an omnidirectional convolution module into the image super-resolution model to realize feature fusion between local information, so as to make up for the content fragmentation problem caused by the transformation from two-dimensional to one-dimensional during Mamba calculation.

[0138] (3) In the image super-resolution model based on omnidirectional convolution and multi-scale Mamba mentioned in this invention, a gated convolutional feedforward network is introduced after the multi-scale Mamba module and the omnidirectional convolution module to improve the reconstruction quality, thereby realizing the feature representation capability of the image super-resolution model. Attached Figure Description

[0139] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0140] Figure 1 This is a flowchart illustrating an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as an embodiment of this application.

[0141] Figure 2 This is a schematic diagram of the structure of an image super-resolution model based on omnidirectional convolution and multi-scale Mamba, which is an embodiment of this application.

[0142] Figure 3 This is a schematic diagram of the structure of the omnidirectional convolution-multi-scale Mamba module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, which is an embodiment of this application.

[0143] Figure 4 This is a schematic diagram of the structure of the multi-scale Mamba module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, which is an embodiment of this application.

[0144] Figure 5 This is a schematic diagram of the deconvolution upsampling module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as described in an embodiment of this application.

[0145] Figure 6 This is a schematic diagram of the selective state scanning structure based on Mamba, which is an embodiment of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba in this application.

[0146] Figure 7 This is a schematic diagram of the structure of the Haar wavelet transform downsampling module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as an embodiment of this application.

[0147] Figure 8 This is a schematic diagram of the structure of an omnidirectional convolution module in an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as described in an embodiment of this application.

[0148] Figure 9 This is a schematic diagram of the structure of a gated convolutional feedforward network for an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as described in an embodiment of this application.

[0149] Figure 10 This is a schematic diagram of an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, as an embodiment of this application.

[0150] Figure 11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0151] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0152] Figure 1 An embodiment of this application provides an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, comprising the following steps:

[0153] S1. Construct and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba to obtain the trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0154] For details, please refer to Figure 2 The image super-resolution model based on omnidirectional convolution and multi-scale Mamba proposed in the embodiments of this application consists of a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer. The N omnidirectional convolution-multi-scale Mamba modules are respectively represented as the first omnidirectional convolution-multi-scale Mamba module, the second omnidirectional convolution-multi-scale Mamba module, ..., the Nth omnidirectional convolution-multi-scale Mamba module. (Reference) Figure 3The omnidirectional convolutional-multi-scale Mamba module consists of five parts: a mobile inverted bottleneck convolution layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network. Both the first and second gated convolutional feedforward networks are gated convolutional feedforward networks. The computation process of the omnidirectional convolutional-multi-scale Mamba module is as follows:

[0155] Input features of omnidirectional convolutional multi-scale Mamba modules The gated features are obtained by sequentially passing the mobile-end inverted bottleneck convolutional layer, the multi-scale Mamba module, and the first gated convolutional feedforward network. As shown in the following formula:

[0156] ;

[0157] in, This represents a gated convolutional feedforward network. This represents a multi-scale Mamba module. This indicates an inverted bottleneck convolutional layer for mobile devices.

[0158] The gated features are sequentially passed through an omnidirectional convolutional module and a second gated convolutional feedforward network to obtain the output features of the omnidirectional convolutional-multi-scale Mamba module, as shown in the following equation:

[0159] ;

[0160] in, This represents an omnidirectional convolutional-multiscale Mamba module. This indicates an omnidirectional convolution module.

[0161] The structure of each module in the Omnidirectional Convolution-Multi-Scale Mamba module is described in detail below.

[0162] In a specific embodiment, the multi-scale Mamba module includes a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module; both the first selective state scanning module and the second selective state scanning module adopt a selective state scanning structure based on Mamba.

[0163] In the multi-scale Mamba module, the input features of the multi-scale Mamba module first pass through a first depthwise separable convolutional layer to obtain the first channel features and the second channel features, as shown in the following formula:

[0164] ;

[0165] in, This indicates the first depthwise separable convolutional layer. This represents the input features of a multi-scale Mamba module. and These represent the features of the first channel and the features of the second channel, respectively.

[0166] The first channel feature is input into the Haar wavelet transform downsampling module to obtain the first intermediate feature, as shown in the following formula:

[0167] ;

[0168] in, This is represented as the Haar wavelet transform downsampling module. Indicates the first intermediate feature;

[0169] The first intermediate feature is input into the first selective state scanning module for selective scanning to obtain the first output feature, as shown in the following formula:

[0170] ;

[0171] in, This represents a selective state scan structure based on Mamba. Indicates the first output feature;

[0172] The second channel feature is input into the second selective state scanning module for selective scanning to obtain the second output feature, as shown in the following formula:

[0173] ;

[0174] in, Indicates the second output feature;

[0175] The first output feature is input into the deconvolution upsampling module to obtain the third output feature, as shown in the following formula:

[0176] ;

[0177] in, This is represented as a deconvolution upsampling module. Indicates the third output feature;

[0178] The second output feature is input into the feature fusion module to obtain the output feature of the multi-scale Mamba module, as shown in the following formula:

[0179] ;

[0180] in, This indicates the feature fusion module. This represents the output characteristics of the multi-scale Mamba module.

[0181] For details, please refer to Figure 4 The multi-scale Mamba module consists of a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module; both the first and second selective state scanning modules employ a Mamba-based selective state scanning structure. First, the input features of the multi-scale Mamba module are branched using the first depthwise separable convolutional layer to obtain first-channel features and second-channel features. The first-channel features are sequentially processed by the Haar wavelet transform downsampling module, the first selective state scanning module, and the deconvolution upsampling module to obtain a third output feature; the second-channel features are processed by the second selective state scanning module, and then combined with the third output feature by the feature fusion module for feature fusion. In one embodiment of this application, the feature fusion module uses a weighted summation method for feature fusion.

[0182] In a specific embodiment, the deconvolution upsampling module includes a zero-interpolation upsampling layer, a nearest-neighbor interpolation upsampling layer, and a frequency domain information reconstruction module;

[0183] In the deconvolution upsampling module, the input features are first padding to obtain the second intermediate features, as shown in the following equation:

[0184] ;

[0185] in, , This indicates a padding operation. The second intermediate feature is represented by B, C, H, W, and p, which represent the batch size, number of channels, length, width, and padding size of the input features of the Haar wavelet transform downsampling module, respectively. Represents the real number field;

[0186] The second intermediate feature is input into the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer, respectively, to obtain the third and fourth intermediate features, as shown in the following equation:

[0187] ;

[0188] ;

[0189] in, Indicates a zero-interpolation upsampling layer. This indicates the nearest neighbor interpolation upsampling layer. These represent the third and fourth intermediate features, respectively.

[0190] Create The all-zero matrix represents the parameters of the convolution kernel of the point spread function. The center point of the point spread function's convolution kernel is moved from its original position by zero-padding the top-left corner of the all-zero matrix and then performing a cyclic center shift operation. Move to the position of the all-zero matrix The first adjustment feature is obtained. Its expression is:

[0191] ;

[0192] in, This represents zero-filling and cyclic center displacement operations, where 1, C, k1, and k2 represent the batch, number of channels, length, and width of the convolution kernel parameters of the point spread matrix, respectively.

[0193] First, perform a Discrete Fourier Transform on the first adjustment feature to transform it from a time-domain variable to a frequency-domain variable, thus obtaining the optical transfer feature. Then, the optical transmission characteristics are calculated by performing complex conjugation and modulus square calculations to obtain the complex conjugation characteristics. Sum of squared features As shown in the following formula:

[0194] ;

[0195] ;

[0196] ;

[0197] in, This represents the Discrete Fourier Transform operation. This represents complex conjugate computation. Indicates the calculation of the square of the modulus. Represents the field of complex numbers;

[0198] Perform Discrete Fourier Transform on the third and fourth intermediate features respectively, transforming them from time-domain variables to frequency-domain variables, to obtain the fifth intermediate feature. and the sixth intermediate feature Wiener filtering is performed on the fifth intermediate feature based on the complex conjugate feature to obtain the filtered feature. Regularize the sixth intermediate feature to obtain the regularized feature. The filtered features and the regularized features are summed to obtain the regularized frequency domain features. As shown in the following formula:

[0199] ;

[0200] ;

[0201] ;

[0202] ;

[0203] ;

[0204] in, Represents the regularization parameter;

[0205] Multiplying the regularized frequency domain features with the optical transfer features yields the fused features. As shown in the following formula:

[0206] ;

[0207] The fused features and the modulus-squared features are then subjected to block averaging to obtain the first block average feature. Second block average features As shown in the following formula:

[0208] ;

[0209] ;

[0210] in, This indicates a block averaging operation;

[0211] Perform an inverse filtering operation on the first block average feature and the second block average feature to obtain the inverse filtered feature. As shown in the following formula:

[0212] ;

[0213] Upsampling is performed on the inverse filter features to obtain the extended features. The regularized frequency domain features, complex conjugate features, and extended features are then input into the frequency domain information reconstruction module to reconstruct the frequency domain information, thus obtaining the reconstructed features. As shown in the following formula:

[0214] ;

[0215] The reconstructed features are subjected to an inverse discrete Fourier transform to convert them from frequency domain variables to time domain variables, and the real part is then taken to obtain the second adjusted features. As shown in the following formula:

[0216] ;

[0217] in, This represents the inverse discrete Fourier transform operation. This indicates the operation of taking the real part;

[0218] The second adjusted feature is then depaddinged to obtain the output feature from the deconvolution upsampling. ;

[0219] The Mamba-based selective state scanning structure includes a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module. The two-dimensional selective scanning module includes a multi-directional scanning module and a Mamba-based selective state space computation module.

[0220] In the Mamba-based selective state scan structure, the input features of the Mamba-based selective state scan structure First, the channel dimension is expanded through a linear projection layer to obtain the projection features. As shown in the following formula:

[0221] ;

[0222] in, Indicates a linear projection layer;

[0223] Perform a dimensional split on the projected features to obtain the first-dimensional features. Second-dimensional features Its expression is:

[0224] ;

[0225] in, This indicates a dimension splitting operation;

[0226] The first-dimensional features are sequentially passed through a second-depth separable convolutional layer, a padding operation, and a first SiLU activation function layer to obtain convolutional features. As shown in the following formula:

[0227] ;

[0228] in, This represents the SiLU activation function. This indicates a second-depth separable convolutional layer;

[0229] The convolutional features are input into the two-dimensional selective scanning module. First, the multi-directional scanning module unfolds the convolutional features into four one-dimensional sequences along four directions, which are the first sequence. , second sequence Third sequence Fourth sequence ;in and It unfolds along the positive horizontal direction and the positive vertical direction. and It unfolds in the opposite direction of the horizontal and the opposite direction of the vertical;

[0230] Perform dimension mapping operations on the four one-dimensional sequences respectively, projecting the channel dimension C onto... , obtain the joint tensor As shown in the following formula:

[0231] ;

[0232] in, This indicates a dimension mapping operation. Correct the dimension for the time step. For state parameter dimensions;

[0233] Joint tensors along the channel dimension Split into correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g ;

[0234] Correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g The input is fed into the Mamba-based selective state-space computation module to obtain the enhanced features, as shown in the following equation:

[0235] ;

[0236] ;

[0237] ;

[0238] ;

[0239] ;

[0240] in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the input influence matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step. ;

[0241] The second-dimensional feature is passed through a second SiLU activation function layer and multiplied with the enhanced feature to obtain the output feature of the Mamba-based selective state scan structure. As shown in the following formula:

[0242] .

[0243] For details, please refer to Figure 5 In the embodiments of this application, the input features of the deconvolutional upsampling module in the multi-scale Mamba module are first padded to obtain the second intermediate features, which are then passed through a zero-interpolation upsampling layer and a nearest-neighbor interpolation upsampling layer. In the zero-interpolation upsampling layer, a larger all-zero matrix is ​​constructed first, and then each element of the second intermediate feature is placed in its corresponding position in the all-zero matrix, thereby expanding the length and width of the second intermediate feature to twice its original value, resulting in the third intermediate feature. In the nearest-neighbor interpolation upsampling layer, the second intermediate feature is scaled up proportionally, thereby similarly expanding the length and width of the second intermediate feature to twice its original value, resulting in the fourth intermediate feature.

[0244] Next, regarding the parameters of the convolution kernel for the point spread function... Create The zero-matrix is ​​used to place the parameters of the point spread function convolution kernel in the top left corner of the zero-matrix and then perform a cyclic center shift operation on it, moving the center point of the original point spread convolution kernel from its original position. Move to the all-zero matrix Position is obtained by first adjustment feature .

[0245] Then, for the first adjustment feature Perform a discrete Fourier transform to convert the time-domain variables into frequency-domain variables, and obtain the optical transmission characteristics. This achieves the conversion from the first adjustment feature in the time domain to the optical transmission feature in the frequency domain. Then, the discrete Fourier transform is applied... By calculating the complex conjugate and the squared modulus, two frequency domain parameters are obtained: the complex conjugate feature and the squared modulus feature. Then, discrete Fourier transforms are performed on the third and fourth intermediate features output by the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer, respectively, to obtain the fifth intermediate feature. and the sixth intermediate feature .right Wiener filtering is used to obtain the filter characteristics. ,right Regularization is performed to obtain regular features. The filtered and regularized features are then fused to obtain regularized frequency domain features. These regularized frequency domain features are then multiplied with the optical transfer features in the frequency domain, performing an operation equivalent to frequency domain convolution, resulting in fused features. The fused features and the modulus-squared features are then subjected to block averaging, followed by inverse filtering. After the inverse filtering, the features are upsampled and expanded to obtain extended features. This prepares the signal for frequency reconstruction. The extended features obtained from the upsampling expansion are then used to reconstruct the frequency domain information. The large-scale solution is reconstructed from the small-scale solution to obtain the reconstructed features. Then reconstruct the features. Performing an inverse discrete Fourier transform converts the frequency domain variables back to time domain variables and transforms the complex tensor into a real tensor, yielding the second adjusted feature. Finally, the reconstructed second adjustment feature Perform a padding removal operation to obtain the output features from the deconvolution upsampling.

[0246] Further reference Figure 6 The selective state scanning structure used in the first and second selective state scanning modules, based on Mamba, consists of a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module (2D-SSM). The two-dimensional selective scanning module comprises a multi-directional scanning module and a Mamba-based selective state space computation module. The second depthwise separable convolutional layer is represented by DWConv in the figure. In the embodiments of this application, the first and second depthwise separable convolutional layers use the same structure, both being depthwise separable convolutions with a kernel size of 1×1. The input features of the Mamba-based selective state scanning structure are input to the linear projection layer to expand the channel dimension and obtain the projected features. Then, the obtained projected features are dimensionally split to obtain the first-dimensional features and the second-dimensional features. The first-dimensional features are then subjected to depthwise separable convolution and padding operations sequentially to ensure that the output after the convolution operation maintains the same dimension. Finally, the convolutional features are obtained by applying the SiLU activation function. Next, the convolutional features are processed by a multi-directional scanning module. The two-dimensional image is unfolded into a one-dimensional sequence along four directions, resulting in the first, second, third, and fourth sequences. Then, a linear layer is used to perform dimensional mapping on each of the four one-dimensional sequences, projecting the channel dimension C onto the image. This yields a joint tensor. Then, the joint tensor will be along the channel dimension. The parameters required for selective state-space computation of Mamba, namely the correction matrix. State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g Next, the obtained parameters are used for selective state-space computation using Mamba to obtain enhanced features.

[0247] The generation of the state transition matrix A consists of two processes: initialization and parameterization. First, a parameterization matrix is ​​created from 1 to... The sequence is copied C times to form a matrix. Because performing multiple scans will expand it to Then, take the logarithm of the result to obtain the logarithmic matrix. Its expression is:

[0248] ;

[0249] in, This is for taking the logarithm. The logarithmic matrix... Perform a forward propagation to update the parameters, and finally, exponentialize and negate the logarithmic matrix after updating the parameters to obtain... Its expression is:

[0250] ;

[0251] in, This is an exponential operation.

[0252] Each scanning direction will have its own state transition matrix A. Since the state transition matrix A is a learnable parameter, it will be adjusted during model training and eventually fixed as the most suitable state transition matrix A for that direction.

[0253] Input influence matrix B g These are parameters that control how the current input affects the state; the state-output matrix C. g It is a parameter that controls how the current state affects the output; the feedforward matrix D g It is the parameter that passes the input to the output.

[0254] Finally, through multi-directional scanning and selective state-space computation, the downsampled image is transformed from two-dimensional to one-dimensional and then back to two-dimensional to obtain enhanced features. The output features of the Mamba-based selective state scan structure are obtained. The input features of this Mamba-based selective state scan structure are in two-dimensional form, i.e., in the form of an image or picture. During the two-dimensional scan, the image is unfolded and stretched into a one-dimensional sequence by pixels. Finally, the feature-enhanced one-dimensional sequence calculated by selective state space is restored to the feature-enhanced two-dimensional image according to the original coordinates.

[0255] In a specific embodiment, the Haar wavelet transform downsampling module includes a channel downsampling convolutional layer, a Haar wavelet transform module, a splicing layer, and a third convolutional layer. The third convolutional layer uses a convolution operation with a kernel size of 1×1.

[0256] In the Haar wavelet transform downsampling module, the input features are first fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain scaled features. As shown in the following formula:

[0257] ;

[0258] in, This indicates a channel downsampling convolutional layer with a kernel size of 1×1;

[0259] The scaling feature is input into the Haar wavelet transform module, and the scaling feature is... Convolution operations are performed with four different two-dimensional filters to obtain four sub-band features, which are the first sub-band features. Second sub-band characteristics Third sub-band characteristics and fourth sub-band features As shown in the following formula:

[0260] ;

[0261] ;

[0262] ;

[0263] ;

[0264] in, This represents the convolution operation. , , and Let represent the first filter, the second filter, the third filter, and the fourth filter, respectively, and their expressions are as follows:

[0265] ;

[0266] ;

[0267] ;

[0268] ;

[0269] Features of the four sub-bands along the channel dimension , , and The input is processed by the stitching layer, and then the inter-channel information is fused through the third convolutional layer to obtain the output features of the Haar wavelet transform downsampling module. As shown in the following formula:

[0270] ;

[0271] in, [·|·] represents a convolution operation with a kernel size of 1×1, and [·|·] represents a concatenation operation along the channel dimension.

[0272] For details, please refer to Figure 7 In the Haar wavelet transform downsampling module mentioned in the embodiments of this application, the input features of the Haar wavelet transform downsampling module are first input into a channel downsampling convolutional layer with a scaling factor of 4, reducing the number of channels of the input features of the Haar wavelet transform downsampling module to one-quarter of the original number, thus obtaining the scaled features. Next, the scaling features are processed using the Haar wavelet transform module. The features are split into four sub-bands, which are then scaled using four different two-dimensional filters. The convolution operation is performed to obtain four two-dimensional filters, which are defined as follows: , , and Finally, the features of the four sub-bands are concatenated along the channel dimension, and then the information between channels is fused through a third convolutional layer with a kernel size of 1×1 to achieve 2x downsampling, thus obtaining the output features of the Haar wavelet transform downsampling module.

[0273] By integrating the Haar wavelet transform downsampling module, the Mamba-based selective state scanning structure, and the deconvolution upsampling module, a multi-scale Mamba module can be constructed.

[0274] In a specific embodiment, the omnidirectional convolution module includes a fourth convolutional layer, a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer. The fourth convolutional layer uses a convolutional operation with a kernel size of 1×1, while the square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer use convolutional operations with kernel sizes of 3×3, 1×11, and 11×1, respectively.

[0275] In the omnidirectional convolution module, the input features are first processed by the fourth convolutional layer to remove redundant channel features, resulting in filtered features, as shown in the following equation:

[0276] ;

[0277] in, This represents the input features of the omnidirectional convolutional module. Indicates filtering features, This indicates a convolution operation with a kernel size of 1×1;

[0278] Padding operations are performed on the filter features to expand them to different dimensional sizes, resulting in the first expanded filter features. Second extended filtering feature and third extended filtering features B, C, H, W, and p represent the batch size, number of channels, length, width, and padding size of the input features in the omnidirectional convolutional module, respectively. Representing the real number field; the first, second, and third extended filter features are input into a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer, respectively, to obtain square features. Horizontal characteristics and vertical features As shown in the following formula:

[0279] ;

[0280] ;

[0281] ;

[0282] in, This indicates a convolution operation with a kernel size of 3×3. This represents a convolution operation with a kernel size of 1×11. This indicates a convolution operation with a kernel size of 11×1;

[0283] The square, horizontal, and vertical features are respectively concatenated with the filtered features using residual connections to obtain the output features of the omnidirectional convolutional module. As shown in the following formula:

[0284] .

[0285] For details, please refer to Figure 8In the omnidirectional convolution-multi-scale Mamba module mentioned in the embodiments of this application, the input features of the omnidirectional convolution module first pass through a fourth convolutional layer with a kernel size of 1×1 to remove redundant channel features, thus obtaining filtered features. Ensure that the extracted features retain the characteristics of the filtered features. For the same dimensions, filtering is required first. A padding operation is performed to expand the dimensions, resulting in three expanded filter features: a first expanded filter feature, a second expanded filter feature, and a third expanded filter feature. These three expanded filter features, output after the padding operation, are then input into three convolutional layers with different orientations: a square convolutional layer, a horizontal stripe convolutional layer, and a vertical stripe convolutional layer. This extracts features in different orientations: square features, horizontal features, and vertical features. Next, residual concatenation is performed between the square features, horizontal features, and vertical features and the filter features to obtain the output features of the omnidirectional convolutional module.

[0286] In a specific embodiment, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt gated convolutional feedforward networks. The gated convolutional feedforward network includes a fifth convolutional layer, a third depthwise separable convolutional layer, a dimension splitting unit, a GELU activation function layer, and a sixth convolutional layer.

[0287] In a gated convolutional feedforward network, the input features are first fed into the fifth convolutional layer for channel-wise projection expansion to obtain the initial projected expanded features. As shown in the following formula:

[0288] ;

[0289] in, This represents the input features of a gated convolutional feedforward network. This indicates that a channel-wise projection expansion operation is performed using the fifth convolutional layer. B, C, H, and W represent the batch size, number of channels, length, and width of the input features of the gated convolutional feedforward network, respectively. Represents the real number field;

[0290] After padding the initial projected extended features, a third depthwise separable convolutional layer is used for feature extraction to obtain the first projected extended features. As shown in the following formula:

[0291] ;

[0292] in, This indicates a third-depth separable convolutional layer;

[0293] Extend the first projection feature The input is fed into the dimension segmentation unit, and average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features The second projection extension feature After being input into the GELU activation function layer, it is then combined with the third projection extended features. Multiplying the features yields the fourth projective extended feature. As shown in the following formula:

[0294] ;

[0295] in, Represents the GELU activation function layer;

[0296] Extend the fourth projection feature The input is fed into the sixth convolutional layer for channel-wise projection reduction, yielding the output features of the gated convolutional feedforward network. As shown in the following formula:

[0297] ;

[0298] in, This indicates the sixth convolutional layer.

[0299] For details, please refer to Figure 9 In the gated convolutional feedforward network of the embodiments of this application, the input features of the gated convolutional feedforward network are first projected and expanded along the channels using a fifth convolutional layer with a kernel size of 1×1 to obtain the initial projected expanded features. Then, the initial projection is expanded to include features. After padding, feature extraction is performed using a third depthwise separable convolutional layer with a kernel size of 3×3 to obtain the first projected extended features. Next, the first projection extension feature is... In the input dimension segmentation unit, average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features , for the second projection extension feature Nonlinear transformation is achieved using the GELU activation function, with third projection extending features. The gating unit is multiplied with the second projected extended feature after the GELU activation function to enhance the representational power of the feature, resulting in the fourth projected extended feature. Finally, the fourth projection feature is expanded by a sixth convolutional layer with a kernel size of 1×1. A projection reduction operation is performed along the channel to obtain the output features of the gated convolutional feedforward network.

[0300] S2: The low-resolution image to be reconstructed is input into the trained image super-resolution model. It first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

[0301] Specifically, the trained image super-resolution model is deployed, referring to... Figure 2 During the inference phase, for a given low-resolution image First, shallow features are extracted using a first convolutional layer with a kernel size of 1×1. Its expression is as follows:

[0302] ;

[0303] in, This indicates a convolution operation with a 1×1 kernel;

[0304] Secondly, deep features are extracted using N omnidirectional convolutional multi-scale Mamba modules. Its expression is as follows:

[0305] ;

[0306] in, This represents the i-th omnidirectional convolutional-multiscale Mamba module. This represents the output feature of the i-th omnidirectional convolutional-multi-scale Mamba module. N is typically set to 4. The output features of the Nth omnidirectional convolutional-multiscale Mamba module are used as depth features.

[0307] Finally, a second convolutional layer with a 1×1 kernel is used to process the depth features. Reconstruction is performed to generate the corresponding high-resolution image. .

[0308] Further reference Figure 10 As an implementation of the methods shown in the above figures, this application provides an embodiment of an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0309] This application provides an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, comprising:

[0310] Model building module 1 is configured to build and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba to obtain a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-end inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0311] Reconstruction module 2 is configured to input the low-resolution image to be reconstructed into the trained image super-resolution model. The image first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

[0312] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 7 As shown, the electronic device in this embodiment includes a processor 1101 and a memory 1102; wherein the memory 1102 is used to store computer execution instructions; and the processor 1101 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0313] Alternatively, the memory 1102 can be either standalone or integrated with the processor 1101.

[0314] When the memory 1102 is set up independently, the electronic device also includes a bus 1103 for connecting the memory 1102 and the processor 1101.

[0315] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by the processor 1101, implement the above method.

[0316] This invention also provides a computer program product, including a computer program that, when executed by a processor 1101, implements the above-described method.

[0317] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0318] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0319] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0320] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 1101 to execute some steps of the methods of the various embodiments of this application.

[0321] It should be understood that the processor 1101 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 1101 can be any conventional processor 1101. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 1101 executing the steps, or as a combination of hardware and software modules within the processor 1101 executing the steps.

[0322] The memory 1102 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0323] Bus 1103 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1103 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 1103 in the accompanying drawings of this application is not limited to only one bus 1103 or one type of bus 1103.

[0324] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0325] An exemplary storage medium is coupled to a processor 1101, enabling the processor 1101 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 1101. The processor 1101 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 1101 and the storage medium can exist as discrete components in an electronic device or a host device.

[0326] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0327] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image super-resolution method based on omnidirectional convolution and multi-scale Mamba, characterized in that, Includes the following steps: An image super-resolution model based on omnidirectional convolution and multi-scale Mamba is constructed and trained to obtain a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-end inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence. The multi-scale Mamba module includes a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module. Both the first and second selective state scanning modules adopt a Mamba-based selective state scanning structure. In the multi-scale Mamba module, the input features of the multi-scale Mamba module first pass through the first depthwise separable convolutional layer to obtain the first channel features and the second channel features. The first channel features are then input into the Haar wavelet transform downsampling module to obtain the first intermediate features. The first intermediate features are then input into the first selective state scanning module for selective scanning to obtain the first output features. The second channel features are then input into the second selective state scanning module for selective scanning to obtain the second output features. The first output features are then input into the deconvolution upsampling module to obtain the third output features. Finally, the second output features and the third output features are input into the feature fusion module to obtain the output features of the multi-scale Mamba module. The omnidirectional convolution module includes a fourth convolutional layer, a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer; In the omnidirectional convolution module, the input features of the omnidirectional convolution module first pass through the fourth convolution layer to remove redundant channel features, thus obtaining filtered features; The filtering features are each subjected to a padding operation to expand them to different dimensional sizes, resulting in a first expanded filtering feature, a second expanded filtering feature, and a third expanded filtering feature. The first expanded filtering feature, the second expanded filtering feature, and the third expanded filtering feature are then input into the square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer, respectively, to obtain square features, horizontal features, and vertical features. The square feature, horizontal feature, and vertical feature are respectively residually concatenated with the filter feature to obtain the output feature of the omnidirectional convolution module; The low-resolution image to be reconstructed is input into the trained image super-resolution model. It first passes through the first convolutional layer to extract shallow features. The shallow features are then passed through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

2. The image super-resolution method based on omnidirectional convolution and multi-scale Mamba as described in claim 1, characterized in that, The calculation formula for the first depthwise separable convolutional layer is: ； in, This indicates the first depthwise separable convolutional layer. This represents the input features of the multi-scale Mamba module. and These represent the features of the first channel and the features of the second channel, respectively. The calculation formula for the Haar wavelet transform downsampling module is as follows: ； in, This is represented as the Haar wavelet transform downsampling module. Indicates the first intermediate feature; The calculation formula for the first selective state scanning module is: ； in, This represents a selective state scan structure based on Mamba. Indicates the first output feature; The calculation formula for the second selective state scanning module is as follows: ； in, Indicates the second output feature; The calculation formula for the deconvolution upsampling module is as follows: ； in, This is represented as a deconvolution upsampling module. Indicates the third output feature; The calculation formula for the feature fusion module is as follows: ； in, Indicates the feature fusion module, This represents the output characteristics of the multi-scale Mamba module.

3. The image super-resolution method based on omnidirectional convolution and multi-scale Mamba as described in claim 2, characterized in that, The deconvolution upsampling module includes a zero-interpolation upsampling layer, a nearest-neighbor interpolation upsampling layer, and a frequency domain information reconstruction module; In the deconvolution upsampling module, the input features are first subjected to a padding operation to obtain the second intermediate features, as shown in the following equation: ； in, , This indicates a padding operation. The second intermediate feature is represented by B, C, H, W, and p, which respectively represent the batch size, number of channels, length, width, and padding size of the input feature of the Haar wavelet transform downsampling module. Represents the real number field; The second intermediate feature is input into the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer respectively to obtain the third and fourth intermediate features, as shown in the following formula: ； ； in, Indicates a zero-interpolation upsampling layer. This indicates the nearest neighbor interpolation upsampling layer. These represent the third and fourth intermediate features, respectively. Create The all-zero matrix represents the parameters of the convolution kernel of the point spread function. The center point of the point spread function's convolution kernel is moved from its original position by zero-padding the top-left corner of the all-zero matrix and then performing a cyclic center shift operation. Move to the position of the all-zero matrix The first adjustment feature is obtained. Its expression is: ； in, This represents zero-filling and cyclic center displacement operations, where 1, C, k1, and k2 represent the batch, number of channels, length, and width of the convolution kernel parameters of the point spread matrix, respectively. First, perform a Discrete Fourier Transform on the first adjustment feature to transform it from a time-domain variable to a frequency-domain variable, thus obtaining the optical transmission feature. Then, the optical transmission characteristics are calculated by performing complex conjugation and modulus square calculations to obtain the complex conjugation characteristics. Sum of squared features As shown in the following formula: ； ； ； in, This represents the Discrete Fourier Transform operation. This represents complex conjugate computation. Indicates the calculation of the square of the modulus. Represents the field of complex numbers; Perform Discrete Fourier Transform on the third and fourth intermediate features respectively, transforming them from time-domain variables to frequency-domain variables, to obtain the fifth intermediate feature. and the sixth intermediate feature Wiener filtering is performed on the fifth intermediate feature based on the complex conjugate feature to obtain the filtered feature. The sixth intermediate feature is regularized to obtain the regularized feature. The filtering features and regularization features are summed to obtain the regularized frequency domain features. As shown in the following formula: ； ； ； ； ； in, Represents the regularization parameter; Multiplying the regularized frequency domain features with the optical transfer features yields the fused features. As shown in the following formula: ； The fused features and the modulus-squared features are then subjected to block averaging operations to obtain the first block average feature. Second block average features As shown in the following formula: ； ； in, This indicates a block averaging operation; Perform an inverse filtering operation on the first block average feature and the second block average feature to obtain the inverse filtered feature. As shown in the following formula: ； The inverse filter features are upsampled and expanded to obtain the expanded features. The regularized frequency domain features, complex conjugate features, and extended features are then input into the frequency domain information reconstruction module to reconstruct the frequency domain information, thereby obtaining the reconstructed features. As shown in the following formula: ； The reconstructed features are subjected to an inverse discrete Fourier transform to convert them from frequency domain variables to time domain variables, and the real part is then taken to obtain the second adjusted features. As shown in the following formula: ； in, This represents the inverse discrete Fourier transform operation. This indicates the operation of taking the real part; The second adjusted feature is then de-paddinged to obtain the output feature of the deconvolution upsampling module. ; The Mamba-based selective state scanning structure includes a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module. The two-dimensional selective scanning module includes a multi-directional scanning module and a Mamba-based selective state space calculation module. In the Mamba-based selective state scan structure, the input features of the Mamba-based selective state scan structure First, the linear projection layer is used to expand the channel dimension, resulting in the projection features. As shown in the following formula: ； in, This represents the linear projection layer; The projected features are then subjected to dimensional splitting to obtain the first dimensional features. Second-dimensional features Its expression is: ； in, This indicates a dimension splitting operation; The first dimension feature is sequentially passed through a second depthwise separable convolutional layer, a padding operation, and a first SiLU activation function layer to obtain the convolutional feature. As shown in the following formula: ； in, This represents the SiLU activation function. This indicates a second-depth separable convolutional layer; The convolutional features are input into the two-dimensional selective scanning module. First, the multi-directional scanning module unfolds the convolutional features into four one-dimensional sequences along four directions, which are respectively the first sequence. , second sequence Third sequence Fourth sequence ;in and It unfolds along the positive horizontal direction and the positive vertical direction. and It unfolds in the opposite direction of the horizontal and the opposite direction of the vertical; Perform dimension mapping operations on the four one-dimensional sequences respectively, projecting the channel dimension C onto... , obtain the joint tensor As shown in the following formula: ； in, This indicates a dimension mapping operation. Correct the dimension for the time step. For state parameter dimensions; The joint tensor along the channel dimension Split into correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g ; The correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g The input is fed into the Mamba-based selective state-space computation module to obtain the enhanced features, as shown in the following equation: ； ； ； ； ； in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the input influence matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step. ; The second-dimensional feature is passed through a second SiLU activation function layer and multiplied with the enhanced feature to obtain the output feature of the Mamba-based selective state scan structure. As shown in the following formula: 。 4. The image super-resolution method based on omnidirectional convolution and multi-scale Mamba as described in claim 2, characterized in that, The Haar wavelet transform downsampling module includes a channel downsampling convolutional layer, a Haar wavelet transform module, a splicing layer, and a third convolutional layer. The third convolutional layer uses a convolution operation with a kernel size of 1×1. In the Haar wavelet transform downsampling module, the input features are first fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain scaled features. As shown in the following formula: ； in, This indicates a channel downsampling convolutional layer with a kernel size of 1×1; The scaling feature is input into the Haar wavelet transform module. Convolution operations are performed with four different two-dimensional filters to obtain four sub-band features, which are the first sub-band features. Second sub-band characteristics Third sub-band characteristics and fourth sub-band features As shown in the following formula: ； ； ； ； in, This represents the convolution operation. , , and Let represent the first filter, the second filter, the third filter, and the fourth filter, respectively, and their expressions are as follows: ； ； ； ； Features of the four sub-bands along the channel dimension , , and The input is processed by the stitching layer, and then the inter-channel information is fused through the third convolutional layer to obtain the output features of the Haar wavelet transform downsampling module. As shown in the following formula: ； in, [·|·] represents a convolution operation with a kernel size of 1×1, and [·|·] represents a concatenation operation along the channel dimension.

5. The image super-resolution method based on omnidirectional convolution and multi-scale Mamba as described in claim 1, characterized in that, The fourth convolutional layer uses a 1×1 kernel for convolution. The square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer use 3×3, 1×11, and 11×1 kernels, respectively. The calculation formula for the fourth convolutional layer is: ； in, This represents the input features of the omnidirectional convolution module. Indicates filtering features, This indicates a convolution operation with a kernel size of 1×1; The calculation formulas for the square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer are as follows: ； ； ； in, This represents the first extended filtering feature. This indicates the second extended filtering feature. The term represents the third extended filtering feature. B, C, H, W, and p represent the batch size, number of channels, length, width, and padding size of the input features of the omnidirectional convolutional module, respectively. Represents the real number field. Indicates square characteristics, Indicates horizontal features, Indicates vertical features. This indicates a convolution operation with a kernel size of 3×3. This represents a convolution operation with a kernel size of 1×11. This indicates a convolution operation with a kernel size of 11×1; The output features of the omnidirectional convolution module The calculation process is as follows: 。 6. The image super-resolution method based on omnidirectional convolution and multi-scale Mamba as described in claim 1, characterized in that, Both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt gated convolutional feedforward networks. The gated convolutional feedforward network includes a fifth convolutional layer, a third depthwise separable convolutional layer, a dimension splitting unit, a GELU activation function layer, and a sixth convolutional layer. In the gated convolutional feedforward network, the input features of the gated convolutional feedforward network are first fed into the fifth convolutional layer for channel-wise projection expansion to obtain initial projection expansion features. As shown in the following formula: ； in, This represents the input features of the gated convolutional feedforward network. This indicates that a channel-wise projection expansion operation is performed using the fifth convolutional layer. B, C, H, and W represent the batch size, number of channels, length, and width of the input features of the gated convolutional feedforward network, respectively. Represents the real number field; After performing a padding operation on the initial projection-expanded features, a third depthwise separable convolutional layer is used for feature extraction to obtain the first projection-expanded features. As shown in the following formula: ； in, This indicates a third-depth separable convolutional layer; The first projection extension feature The input is fed into the dimension segmentation unit, and average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features The second projection extension feature After being input into the GELU activation function layer, it is then combined with the third projection extended features. Multiplying the features yields the fourth projective extended feature. As shown in the following formula: ； in, Represents the GELU activation function layer; The fourth projection extension feature The input is fed into the sixth convolutional layer for channel-wise projection reduction to obtain the output features of the gated convolutional feedforward network. As shown in the following formula: ； in, This indicates the sixth convolutional layer.

7. An image super-resolution device based on omnidirectional convolution and multi-scale Mamba, characterized in that, The image super-resolution method based on omnidirectional convolution and multi-scale Mamba as described in any one of claims 1-6 includes: The model building module is configured to construct and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba to obtain a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-end inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence. The reconstruction module is configured to input the low-resolution image to be reconstructed into the trained image super-resolution model. The image first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The deep features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images