Image super-resolution method and device based on omnidirectional convolution and multi-scale Mama

Abstract

The invention discloses an image super-resolution method based on omni-directional convolution and multi-scale Mama, and relates to the field of image processing, and the method comprises the steps: constructing an image super-resolution model based on omni-directional convolution and multi-scale Mama, the image super-resolution model comprises a first convolution layer, N omni-directional convolution-multi-scale Mama modules and a second convolution layer which are connected in sequence, the omni-directional convolution-multi-scale Mama module comprises a mobile end inverted bottleneck convolution layer, a multi-scale Mama module, a first gating convolution feedforward network, an omni-directional convolution module and a second gating convolution feedforward network which are connected in sequence; a to-be-reconstructed low-resolution image is input into a trained image super-resolution model, shallow layer features are extracted through a first convolution layer, depth features are extracted through N omnidirectional convolution-multi-scale Mama modules in sequence, and the depth features are reconstructed through a second convolution layer to obtain a high-resolution image. According to the method, the problem of content splitting in traditional Mama calculation is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to an image super-resolution method and apparatus based on omnidirectional convolution and multi-scale Mamba. Background Technology

[0002] Image super-resolution (ISR) is an important task in the field of computer vision, aiming to restore high-resolution images from low-resolution images. Because the restored high-resolution images can reveal more key and subtle information hidden in the original images, it is widely used in computer vision fields such as satellite imaging, video surveillance, and facial recognition.

[0003] Mamba, as a rising backbone of deep learning networks in recent years, has gradually demonstrated performance comparable to Transformer networks, leading to its widespread application in computer vision. Mamba achieves long-range dependency capture through a selective state-space mechanism. Comparative studies have shown that, with the same training data, the Mamba model not only has lower computational cost than Transformer but also avoids the attention sparsity problem of Transformer, achieving better results in image super-resolution tasks. MambaIR, as the first work to apply the Mamba model to image super-resolution, showed strong competitive results compared to various Transformer variants based on image super-resolution. With in-depth research on the Mamba network architecture, various improved Mamba-based network structures have further enhanced its performance, such as global-local collaborative modeling architectures and arbitrary-scale super-resolution implementations. However, the loss of image information by the ordinary depthwise convolutional downsampling module in multi-scale Mamba and the content fragmentation caused by the 2D-to-1D transformation during multi-scale Mamba computation both affect Mamba's performance in image super-resolution. Summary of the Invention

[0004] The purpose of this application is to propose an image super-resolution method and apparatus based on omnidirectional convolution and multi-scale Mamba to address the aforementioned technical problems.

[0005] In a first aspect, the present invention provides an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, comprising the following steps:

[0006] An image super-resolution model based on omnidirectional convolution and multi-scale Mamba is constructed and trained to obtain the trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-end inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0007] The low-resolution image to be reconstructed is input into the trained image super-resolution model. It first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

[0008] Preferably, the multi-scale Mamba module includes a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module; both the first selective state scanning module and the second selective state scanning module adopt a selective state scanning structure based on Mamba.

[0009] In the multi-scale Mamba module, the input features of the multi-scale Mamba module first pass through a first depthwise separable convolutional layer to obtain the first channel features and the second channel features, as shown in the following formula:

[0010] ;

[0011] in, This indicates the first depthwise separable convolutional layer. This represents the input features of a multi-scale Mamba module. and These represent the features of the first channel and the features of the second channel, respectively.

[0012] The first channel feature is input into the Haar wavelet transform downsampling module to obtain the first intermediate feature, as shown in the following formula:

[0013] ;

[0014] in, This is represented as the Haar wavelet transform downsampling module. Indicates the first intermediate feature;

[0015] The first intermediate feature is input into the first selective state scanning module for selective scanning to obtain the first output feature, as shown in the following formula:

[0016] ;

[0017] in, This represents a selective state scan structure based on Mamba. Indicates the first output feature;

[0018] The second channel feature is input into the second selective state scanning module for selective scanning to obtain the second output feature, as shown in the following formula:

[0019] ;

[0020] in, Indicates the second output feature;

[0021] The first output feature is input into the deconvolution upsampling module to obtain the third output feature, as shown in the following formula:

[0022] ;

[0023] in, This is represented as a deconvolution upsampling module. Indicates the third output feature;

[0024] The second output feature is input into the feature fusion module to obtain the output feature of the multi-scale Mamba module, as shown in the following formula:

[0025] ;

[0026] in, This indicates the feature fusion module. This represents the output characteristics of the multi-scale Mamba module.

[0027] Preferably, the deconvolution upsampling module includes a zero-interpolation upsampling layer, a nearest-neighbor interpolation upsampling layer, and a frequency domain information reconstruction module;

[0028] In the deconvolution upsampling module, the input features are first padding to obtain the second intermediate features, as shown in the following equation:

[0029] ;

[0030] in, , This indicates a padding operation. The second intermediate feature is represented by B, C, H, W, and p, which represent the batch size, number of channels, length, width, and padding size of the input features of the Haar wavelet transform downsampling module, respectively. Represents the real number field;

[0031] The second intermediate feature is input into the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer, respectively, to obtain the third and fourth intermediate features, as shown in the following equation:

[0032] ;

[0033] ;

[0034] in, Indicates a zero-interpolation upsampling layer. This indicates the nearest neighbor interpolation upsampling layer. These represent the third and fourth intermediate features, respectively.

[0035] Create The all-zero matrix represents the parameters of the convolution kernel of the point spread function. The center point of the point spread function's convolution kernel is moved from its original position by zero-padding the top-left corner of the all-zero matrix and then performing a cyclic center shift operation. Move to the position of the all-zero matrix The first adjustment feature is obtained. Its expression is:

[0036] ;

[0037] in, This represents zero-filling and cyclic center displacement operations, where 1, C, k1, and k2 represent the batch, number of channels, length, and width of the convolution kernel parameters of the point spread matrix, respectively.

[0038] First, perform a Discrete Fourier Transform on the first adjustment feature to transform it from a time-domain variable to a frequency-domain variable, thus obtaining the optical transfer feature. Then, the optical transmission characteristics are calculated by performing complex conjugation and modulus square calculations to obtain the complex conjugation characteristics. Sum of squared features As shown in the following formula:

[0039] ;

[0040] ;

[0041] ;

[0042] in, This represents the Discrete Fourier Transform operation. This represents complex conjugate computation. Indicates the calculation of the square of the modulus. Represents the field of complex numbers;

[0043] Perform Discrete Fourier Transform on the third and fourth intermediate features respectively, transforming them from time-domain variables to frequency-domain variables, to obtain the fifth intermediate feature. and the sixth intermediate feature Wiener filtering is performed on the fifth intermediate feature based on the complex conjugate feature to obtain the filtered feature. Regularize the sixth intermediate feature to obtain the regularized feature. The filtered features and the regularized features are summed to obtain the regularized frequency domain features. As shown in the following formula:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] in, Represents the regularization parameter;

[0050] Multiplying the regularized frequency domain features with the optical transfer features yields the fused features. As shown in the following formula:

[0051] ;

[0052] The fused features and the modulus-squared features are then subjected to block averaging to obtain the first block average feature. Second block average features As shown in the following formula:

[0053] ;

[0054] ;

[0055] in, This indicates a block averaging operation;

[0056] Perform an inverse filtering operation on the first block average feature and the second block average feature to obtain the inverse filtered feature. As shown in the following formula:

[0057] ;

[0058] Upsampling is performed on the inverse filter features to obtain the extended features. The regularized frequency domain features, complex conjugate features, and extended features are then input into the frequency domain information reconstruction module to reconstruct the frequency domain information, thus obtaining the reconstructed features. As shown in the following formula:

[0059] ;

[0060] The reconstructed features are subjected to an inverse discrete Fourier transform to convert them from frequency domain variables to time domain variables, and the real part is then taken to obtain the second adjusted features. As shown in the following formula:

[0061] ;

[0062] in, This represents the inverse discrete Fourier transform operation. This indicates the operation of taking the real part;

[0063] The second adjusted feature is then depaddinged to obtain the output feature from the deconvolution upsampling. ;

[0064] The Mamba-based selective state scanning structure includes a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module. The two-dimensional selective scanning module includes a multi-directional scanning module and a Mamba-based selective state space computation module.

[0065] In the Mamba-based selective state scan structure, the input features of the Mamba-based selective state scan structure First, the channel dimension is expanded through a linear projection layer to obtain the projection features. As shown in the following formula:

[0066] ;

[0067] in, Indicates a linear projection layer;

[0068] Perform a dimensional split on the projected features to obtain the first-dimensional features. Second-dimensional features Its expression is:

[0069] ;

[0070] in, This indicates a dimension splitting operation;

[0071] The first-dimensional features are sequentially passed through a second-depth separable convolutional layer, a padding operation, and a first SiLU activation function layer to obtain convolutional features. As shown in the following formula:

[0072] ;

[0073] in, This represents the SiLU activation function. This indicates a second-depth separable convolutional layer;

[0074] The convolutional features are input into the two-dimensional selective scanning module. First, the multi-directional scanning module unfolds the convolutional features into four one-dimensional sequences along four directions, which are the first sequence. , second sequence Third sequence Fourth sequence ;in and It unfolds along the positive horizontal direction and the positive vertical direction. and It unfolds in the opposite direction of the horizontal and the opposite direction of the vertical;

[0075] Perform dimension mapping operations on the four one-dimensional sequences respectively, projecting the channel dimension C onto... , obtain the joint tensor As shown in the following formula:

[0076] ;

[0077] in, This indicates a dimension mapping operation. Correct the dimension for the time step. For state parameter dimensions;

[0078] Joint tensors along the channel dimension Split into correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g ;

[0079] Correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g The input is fed into the Mamba-based selective state-space computation module to obtain the enhanced features, as shown in the following equation:

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the input influence matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step. ;

[0086] The second-dimensional feature is passed through a second SiLU activation function layer and multiplied with the enhanced feature to obtain the output feature of the Mamba-based selective state scan structure. As shown in the following formula:

[0087] .

[0088] As a preferred embodiment, the Haar wavelet transform downsampling module includes a channel downsampling convolutional layer, a Haar wavelet transform module, a splicing layer, and a third convolutional layer, wherein the third convolutional layer adopts a convolution operation with a kernel size of 1×1.

[0089] In the Haar wavelet transform downsampling module, the input features are first fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain scaled features. As shown in the following formula:

[0090] ;

[0091] in, This indicates a channel downsampling convolutional layer with a kernel size of 1×1;

[0092] The scaling feature is input into the Haar wavelet transform module, and the scaling feature is... Convolution operations are performed with four different two-dimensional filters to obtain four sub-band features, which are the first sub-band features. Second sub-band characteristics Third sub-band characteristics and fourth sub-band features As shown in the following formula:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, This represents the convolution operation. , , and Let represent the first filter, the second filter, the third filter, and the fourth filter, respectively, and their expressions are as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] Features of the four sub-bands along the channel dimension , , and The input is processed by the stitching layer, and then the inter-channel information is fused through the third convolutional layer to obtain the output features of the Haar wavelet transform downsampling module. As shown in the following formula:

[0103] ;

[0104] in, [·|·] represents a convolution operation with a kernel size of 1×1, and [·|·] represents a concatenation operation along the channel dimension.

[0105] Preferably, the omnidirectional convolution module includes a fourth convolutional layer, a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer. The fourth convolutional layer uses a convolutional operation with a kernel size of 1×1, while the square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer use convolutional operations with kernel sizes of 3×3, 1×11, and 11×1, respectively.

[0106] In the omnidirectional convolution module, the input features are first processed by the fourth convolutional layer to remove redundant channel features, resulting in filtered features, as shown in the following equation:

[0107] ;

[0108] in, This represents the input features of the omnidirectional convolutional module. Indicates filtering features, This indicates a convolution operation with a kernel size of 1×1;

[0109] Padding operations are performed on the filter features to expand them to different dimensional sizes, resulting in the first expanded filter features. Second extended filtering feature and third extended filtering features B, C, H, W, and p represent the batch size, number of channels, length, width, and padding size of the input features in the omnidirectional convolutional module, respectively. Representing the real number field; the first, second, and third extended filter features are input into a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer, respectively, to obtain square features. Horizontal characteristics and vertical features As shown in the following formula:

[0110] ;

[0111] ;

[0112] ;

[0113] in, This indicates a convolution operation with a kernel size of 3×3. This represents a convolution operation with a kernel size of 1×11. This indicates a convolution operation with a kernel size of 11×1;

[0114] The square, horizontal, and vertical features are respectively concatenated with the filtered features using residual connections to obtain the output features of the omnidirectional convolutional module. As shown in the following formula:

[0115] .

[0116] As a preferred embodiment, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt gated convolutional feedforward networks. The gated convolutional feedforward network includes a fifth convolutional layer, a third depthwise separable convolutional layer, a dimension splitting unit, a GELU activation function layer, and a sixth convolutional layer.

[0117] In a gated convolutional feedforward network, the input features are first fed into the fifth convolutional layer for channel-wise projection expansion to obtain the initial projected expanded features. As shown in the following formula:

[0118] ;

[0119] in, This represents the input features of a gated convolutional feedforward network. This indicates that a channel-wise projection expansion operation is performed using the fifth convolutional layer. B, C, H, and W represent the batch size, number of channels, length, and width of the input features of the gated convolutional feedforward network, respectively. Represents the real number field;

[0120] After padding the initial projected extended features, a third depthwise separable convolutional layer is used for feature extraction to obtain the first projected extended features. As shown in the following formula:

[0121] ;

[0122] in, This indicates a third-depth separable convolutional layer;

[0123] Extend the first projection feature The input is fed into the dimension segmentation unit, and average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features The second projection extension feature After being input into the GELU activation function layer, it is then combined with the third projection extended features. Multiplying yields the fourth projective extended feature. As shown in the following formula:

[0124] ;

[0125] in, Represents the GELU activation function layer;

[0126] Extend the fourth projection feature The input is fed into the sixth convolutional layer for channel-wise projection reduction to obtain the output features of the gated convolutional feedforward network. As shown in the following formula:

[0127] ;

[0128] in, This indicates the sixth convolutional layer.

[0129] Secondly, the present invention provides an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, comprising:

[0130] The model building module is configured to build and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba, resulting in a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-side inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0131] The reconstruction module is configured to input the low-resolution image to be reconstructed into a trained image super-resolution model. The image first passes through a first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. Finally, the depth features are reconstructed using a second convolutional layer to obtain the corresponding high-resolution image.

[0132] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0133] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0134] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0135] Compared with the prior art, the present invention has the following beneficial effects:

[0136] (1) In the image super-resolution model of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba mentioned in this invention, the Haar wavelet transform downsampling module is used to replace the depth convolution downsampling in the traditional multi-scale Mamba module, which realizes lossless information preservation, so that the image entering the multi-scale scan retains more information, and the multi-scale Mamba module proposed in this invention can capture more long-distance features.

[0137] (2) The image super-resolution method based on omnidirectional convolution and multi-scale Mamba mentioned in this invention introduces an omnidirectional convolution module into the image super-resolution model to realize feature fusion between local information, so as to make up for the content fragmentation problem caused by the transformation from two-dimensional to one-dimensional during Mamba calculation.

[0138] (3) In the image super-resolution model based on omnidirectional convolution and multi-scale Mamba mentioned in this invention, a gated convolutional feedforward network is introduced after the multi-scale Mamba module and the omnidirectional convolution module to improve the reconstruction quality, thereby realizing the feature representation capability of the image super-resolution model. Attached Figure Description

[0139] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0140] Figure 1 This is a flowchart illustrating an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as an embodiment of this application.

[0141] Figure 2 This is a schematic diagram of the structure of an image super-resolution model based on omnidirectional convolution and multi-scale Mamba, which is an embodiment of this application.

[0142] Figure 3 This is a schematic diagram of the structure of the omnidirectional convolution-multi-scale Mamba module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, which is an embodiment of this application.

[0143] Figure 4 This is a schematic diagram of the structure of the multi-scale Mamba module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, which is an embodiment of this application.

[0144] Figure 5 This is a schematic diagram of the deconvolution upsampling module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as described in an embodiment of this application.

[0145] Figure 6 This is a schematic diagram of the selective state scanning structure based on Mamba, which is an embodiment of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba in this application.

[0146] Figure 7 This is a schematic diagram of the structure of the Haar wavelet transform downsampling module of the image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as an embodiment of this application.

[0147] Figure 8 This is a schematic diagram of the structure of an omnidirectional convolution module in an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as described in an embodiment of this application.

[0148] Figure 9 This is a schematic diagram of the structure of a gated convolutional feedforward network for an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, as described in an embodiment of this application.

[0149] Figure 10 This is a schematic diagram of an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, as an embodiment of this application.

[0150] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0151] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0152] Figure 1 An embodiment of this application provides an image super-resolution method based on omnidirectional convolution and multi-scale Mamba, comprising the following steps:

[0153] S1. Construct and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba to obtain the trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0154] For details, please refer to Figure 2 The image super-resolution model based on omnidirectional convolution and multi-scale Mamba proposed in the embodiments of this application consists of a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer. The N omnidirectional convolution-multi-scale Mamba modules are respectively represented as the first omnidirectional convolution-multi-scale Mamba module, the second omnidirectional convolution-multi-scale Mamba module, ..., the Nth omnidirectional convolution-multi-scale Mamba module. (Reference) Figure 3The omnidirectional convolutional-multi-scale Mamba module consists of five parts: a mobile inverted bottleneck convolution layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network. Both the first and second gated convolutional feedforward networks are gated convolutional feedforward networks. The computation process of the omnidirectional convolutional-multi-scale Mamba module is as follows:

[0155] Input features of omnidirectional convolutional multi-scale Mamba modules The gated features are obtained by sequentially passing the mobile-end inverted bottleneck convolutional layer, the multi-scale Mamba module, and the first gated convolutional feedforward network. As shown in the following formula:

[0156] ;

[0157] in, This represents a gated convolutional feedforward network. This represents a multi-scale Mamba module. This indicates an inverted bottleneck convolutional layer for mobile devices.

[0158] The gated features are sequentially passed through an omnidirectional convolutional module and a second gated convolutional feedforward network to obtain the output features of the omnidirectional convolutional-multi-scale Mamba module, as shown in the following equation:

[0159] ;

[0160] in, This represents an omnidirectional convolutional-multiscale Mamba module. This indicates an omnidirectional convolution module.

[0161] The structure of each module in the Omnidirectional Convolution-Multi-Scale Mamba module is described in detail below.

[0162] In a specific embodiment, the multi-scale Mamba module includes a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module; both the first selective state scanning module and the second selective state scanning module adopt a selective state scanning structure based on Mamba.

[0163] In the multi-scale Mamba module, the input features of the multi-scale Mamba module first pass through a first depthwise separable convolutional layer to obtain the first channel features and the second channel features, as shown in the following formula:

[0164] ;

[0165] in, This indicates the first depthwise separable convolutional layer. This represents the input features of a multi-scale Mamba module. and These represent the features of the first channel and the features of the second channel, respectively.

[0166] The first channel feature is input into the Haar wavelet transform downsampling module to obtain the first intermediate feature, as shown in the following formula:

[0167] ;

[0168] in, This is represented as the Haar wavelet transform downsampling module. Indicates the first intermediate feature;

[0169] The first intermediate feature is input into the first selective state scanning module for selective scanning to obtain the first output feature, as shown in the following formula:

[0170] ;

[0171] in, This represents a selective state scan structure based on Mamba. Indicates the first output feature;

[0172] The second channel feature is input into the second selective state scanning module for selective scanning to obtain the second output feature, as shown in the following formula:

[0173] ;

[0174] in, Indicates the second output feature;

[0175] The first output feature is input into the deconvolution upsampling module to obtain the third output feature, as shown in the following formula:

[0176] ;

[0177] in, This is represented as a deconvolution upsampling module. Indicates the third output feature;

[0178] The second output feature is input into the feature fusion module to obtain the output feature of the multi-scale Mamba module, as shown in the following formula:

[0179] ;

[0180] in, This indicates the feature fusion module. This represents the output characteristics of the multi-scale Mamba module.

[0181] For details, please refer to Figure 4 The multi-scale Mamba module consists of a first depthwise separable convolutional layer, a Haar wavelet transform downsampling module, a deconvolution upsampling module, a feature fusion module, a first selective state scanning module, and a second selective state scanning module; both the first and second selective state scanning modules employ a Mamba-based selective state scanning structure. First, the input features of the multi-scale Mamba module are branched using the first depthwise separable convolutional layer to obtain first-channel features and second-channel features. The first-channel features are sequentially processed by the Haar wavelet transform downsampling module, the first selective state scanning module, and the deconvolution upsampling module to obtain a third output feature; the second-channel features are processed by the second selective state scanning module, and then combined with the third output feature by the feature fusion module for feature fusion. In one embodiment of this application, the feature fusion module uses a weighted summation method for feature fusion.

[0182] In a specific embodiment, the deconvolution upsampling module includes a zero-interpolation upsampling layer, a nearest-neighbor interpolation upsampling layer, and a frequency domain information reconstruction module;

[0183] In the deconvolution upsampling module, the input features are first padding to obtain the second intermediate features, as shown in the following equation:

[0184] ;

[0185] in, , This indicates a padding operation. The second intermediate feature is represented by B, C, H, W, and p, which represent the batch size, number of channels, length, width, and padding size of the input features of the Haar wavelet transform downsampling module, respectively. Represents the real number field;

[0186] The second intermediate feature is input into the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer, respectively, to obtain the third and fourth intermediate features, as shown in the following equation:

[0187] ;

[0188] ;

[0189] in, Indicates a zero-interpolation upsampling layer. This indicates the nearest neighbor interpolation upsampling layer. These represent the third and fourth intermediate features, respectively.

[0190] Create The all-zero matrix represents the parameters of the convolution kernel of the point spread function. The center point of the point spread function's convolution kernel is moved from its original position by zero-padding the top-left corner of the all-zero matrix and then performing a cyclic center shift operation. Move to the position of the all-zero matrix The first adjustment feature is obtained. Its expression is:

[0191] ;

[0192] in, This represents zero-filling and cyclic center displacement operations, where 1, C, k1, and k2 represent the batch, number of channels, length, and width of the convolution kernel parameters of the point spread matrix, respectively.

[0193] First, perform a Discrete Fourier Transform on the first adjustment feature to transform it from a time-domain variable to a frequency-domain variable, thus obtaining the optical transfer feature. Then, the optical transmission characteristics are calculated by performing complex conjugation and modulus square calculations to obtain the complex conjugation characteristics. Sum of squared features As shown in the following formula:

[0194] ;

[0195] ;

[0196] ;

[0197] in, This represents the Discrete Fourier Transform operation. This represents complex conjugate computation. Indicates the calculation of the square of the modulus. Represents the field of complex numbers;

[0198] Perform Discrete Fourier Transform on the third and fourth intermediate features respectively, transforming them from time-domain variables to frequency-domain variables, to obtain the fifth intermediate feature. and the sixth intermediate feature Wiener filtering is performed on the fifth intermediate feature based on the complex conjugate feature to obtain the filtered feature. Regularize the sixth intermediate feature to obtain the regularized feature. The filtered features and the regularized features are summed to obtain the regularized frequency domain features. As shown in the following formula:

[0199] ;

[0200] ;

[0201] ;

[0202] ;

[0203] ;

[0204] in, Represents the regularization parameter;

[0205] Multiplying the regularized frequency domain features with the optical transfer features yields the fused features. As shown in the following formula:

[0206] ;

[0207] The fused features and the modulus-squared features are then subjected to block averaging to obtain the first block average feature. Second block average features As shown in the following formula:

[0208] ;

[0209] ;

[0210] in, This indicates a block averaging operation;

[0211] Perform an inverse filtering operation on the first block average feature and the second block average feature to obtain the inverse filtered feature. As shown in the following formula:

[0212] ;

[0213] Upsampling is performed on the inverse filter features to obtain the extended features. The regularized frequency domain features, complex conjugate features, and extended features are then input into the frequency domain information reconstruction module to reconstruct the frequency domain information, thus obtaining the reconstructed features. As shown in the following formula:

[0214] ;

[0215] The reconstructed features are subjected to an inverse discrete Fourier transform to convert them from frequency domain variables to time domain variables, and the real part is then taken to obtain the second adjusted features. As shown in the following formula:

[0216] ;

[0217] in, This represents the inverse discrete Fourier transform operation. This indicates the operation of taking the real part;

[0218] The second adjusted feature is then depaddinged to obtain the output feature from the deconvolution upsampling. ;

[0219] The Mamba-based selective state scanning structure includes a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module. The two-dimensional selective scanning module includes a multi-directional scanning module and a Mamba-based selective state space computation module.

[0220] In the Mamba-based selective state scan structure, the input features of the Mamba-based selective state scan structure First, the channel dimension is expanded through a linear projection layer to obtain the projection features. As shown in the following formula:

[0221] ;

[0222] in, Indicates a linear projection layer;

[0223] Perform a dimensional split on the projected features to obtain the first-dimensional features. Second-dimensional features Its expression is:

[0224] ;

[0225] in, This indicates a dimension splitting operation;

[0226] The first-dimensional features are sequentially passed through a second-depth separable convolutional layer, a padding operation, and a first SiLU activation function layer to obtain convolutional features. As shown in the following formula:

[0227] ;

[0228] in, This represents the SiLU activation function. This indicates a second-depth separable convolutional layer;

[0229] The convolutional features are input into the two-dimensional selective scanning module. First, the multi-directional scanning module unfolds the convolutional features into four one-dimensional sequences along four directions, which are the first sequence. , second sequence Third sequence Fourth sequence ;in and It unfolds along the positive horizontal direction and the positive vertical direction. and It unfolds in the opposite direction of the horizontal and the opposite direction of the vertical;

[0230] Perform dimension mapping operations on the four one-dimensional sequences respectively, projecting the channel dimension C onto... , obtain the joint tensor As shown in the following formula:

[0231] ;

[0232] in, This indicates a dimension mapping operation. Correct the dimension for the time step. For state parameter dimensions;

[0233] Joint tensors along the channel dimension Split into correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g ;

[0234] Correction matrix State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g The input is fed into the Mamba-based selective state-space computation module to obtain the enhanced features, as shown in the following equation:

[0235] ;

[0236] ;

[0237] ;

[0238] ;

[0239] ;

[0240] in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the input influence matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step. ;

[0241] The second-dimensional feature is passed through a second SiLU activation function layer and multiplied with the enhanced feature to obtain the output feature of the Mamba-based selective state scan structure. As shown in the following formula:

[0242] .

[0243] For details, please refer to Figure 5 In the embodiments of this application, the input features of the deconvolutional upsampling module in the multi-scale Mamba module are first padded to obtain the second intermediate features, which are then passed through a zero-interpolation upsampling layer and a nearest-neighbor interpolation upsampling layer. In the zero-interpolation upsampling layer, a larger all-zero matrix is ​​constructed first, and then each element of the second intermediate feature is placed in its corresponding position in the all-zero matrix, thereby expanding the length and width of the second intermediate feature to twice its original value, resulting in the third intermediate feature. In the nearest-neighbor interpolation upsampling layer, the second intermediate feature is scaled up proportionally, thereby similarly expanding the length and width of the second intermediate feature to twice its original value, resulting in the fourth intermediate feature.

[0244] Next, regarding the parameters of the convolution kernel for the point spread function... Create The zero-matrix is ​​used to place the parameters of the point spread function convolution kernel in the top left corner of the zero-matrix and then perform a cyclic center shift operation on it, moving the center point of the original point spread convolution kernel from its original position. Move to the all-zero matrix Position is obtained by first adjustment feature .

[0245] Then, for the first adjustment feature Perform a discrete Fourier transform to convert the time-domain variables into frequency-domain variables, and obtain the optical transmission characteristics. This achieves the conversion from the first adjustment feature in the time domain to the optical transmission feature in the frequency domain. Then, the discrete Fourier transform is applied... By calculating the complex conjugate and the squared modulus, two frequency domain parameters are obtained: the complex conjugate feature and the squared modulus feature. Then, discrete Fourier transforms are performed on the third and fourth intermediate features output by the zero-interpolation upsampling layer and the nearest-neighbor interpolation upsampling layer, respectively, to obtain the fifth intermediate feature. and the sixth intermediate feature .right Wiener filtering is used to obtain the filter characteristics. ,right Regularization is performed to obtain regular features. The filtered and regularized features are then fused to obtain regularized frequency domain features. These regularized frequency domain features are then multiplied with the optical transfer features in the frequency domain, performing an operation equivalent to frequency domain convolution, resulting in fused features. The fused features and the modulus-squared features are then subjected to block averaging, followed by inverse filtering. After the inverse filtering, the features are upsampled and expanded to obtain extended features. This prepares the signal for frequency reconstruction. The extended features obtained from the upsampling expansion are then used to reconstruct the frequency domain information. The large-scale solution is reconstructed from the small-scale solution to obtain the reconstructed features. Then reconstruct the features. Performing an inverse discrete Fourier transform converts the frequency domain variables back to time domain variables and transforms the complex tensor into a real tensor, yielding the second adjusted feature. Finally, the reconstructed second adjustment feature Perform a padding removal operation to obtain the output features from the deconvolution upsampling.

[0246] Further reference Figure 6 The selective state scanning structure used in the first and second selective state scanning modules, based on Mamba, consists of a linear projection layer, a second depthwise separable convolutional layer, a first SiLU activation function layer, a second SiLU activation function layer, and a two-dimensional selective scanning module (2D-SSM). The two-dimensional selective scanning module comprises a multi-directional scanning module and a Mamba-based selective state space computation module. The second depthwise separable convolutional layer is represented by DWConv in the figure. In the embodiments of this application, the first and second depthwise separable convolutional layers use the same structure, both being depthwise separable convolutions with a kernel size of 1×1. The input features of the Mamba-based selective state scanning structure are input to the linear projection layer to expand the channel dimension and obtain the projected features. Then, the obtained projected features are dimensionally split to obtain the first-dimensional features and the second-dimensional features. The first-dimensional features are then subjected to depthwise separable convolution and padding operations sequentially to ensure that the output after the convolution operation maintains the same dimension. Finally, the convolutional features are obtained by applying the SiLU activation function. Next, the convolutional features are processed by a multi-directional scanning module. The two-dimensional image is unfolded into a one-dimensional sequence along four directions, resulting in the first, second, third, and fourth sequences. Then, a linear layer is used to perform dimensional mapping on each of the four one-dimensional sequences, projecting the channel dimension C onto the image. This yields a joint tensor. Then, the joint tensor is along the channel dimension. The parameters required for selective state-space computation of Mamba, namely the correction matrix. State transition matrix A, input influence matrix B g State output matrix C g and feedforward matrix D g Next, the obtained parameters are used for selective state-space computation using Mamba to obtain enhanced features.

[0247] The generation of the state transition matrix A consists of two processes: initialization and parameterization. First, a parameterization matrix is ​​created from 1 to... The sequence is copied C times to form a matrix. Because performing multiple scans will expand it to Then, take the logarithm of the result to obtain the logarithmic matrix. Its expression is:

[0248] ;

[0249] in, This is for the logarithmic operation. The logarithmic matrix... Perform a forward propagation to update the parameters, and finally, exponentialize and negate the logarithmic matrix after updating the parameters to obtain... Its expression is:

[0250] ;

[0251] in, This is an exponential operation.

[0252] Each scanning direction will have its own state transition matrix A. Since the state transition matrix A is a learnable parameter, it will be adjusted during model training and eventually fixed as the most suitable state transition matrix A for that direction.

[0253] Input influence matrix B g These are parameters that control how the current input affects the state; the state-output matrix C. g It is a parameter that controls how the current state affects the output; the feedforward matrix D g It is the parameter that passes the input to the output.

[0254] Finally, through multi-directional scanning and selective state-space computation, the downsampled image is transformed from two-dimensional to one-dimensional and then back to two-dimensional to obtain enhanced features. The output features of the Mamba-based selective state scan structure are obtained. The input features of this Mamba-based selective state scan structure are in two-dimensional form, i.e., in the form of an image or picture. During the two-dimensional scan, the image is unfolded and stretched into a one-dimensional sequence by pixels. Finally, the feature-enhanced one-dimensional sequence calculated by selective state space is restored to the feature-enhanced two-dimensional image according to the original coordinates.

[0255] In a specific embodiment, the Haar wavelet transform downsampling module includes a channel downsampling convolutional layer, a Haar wavelet transform module, a splicing layer, and a third convolutional layer. The third convolutional layer uses a convolution operation with a kernel size of 1×1.

[0256] In the Haar wavelet transform downsampling module, the input features are first fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain scaled features. As shown in the following formula:

[0257] ;

[0258] in, This indicates a channel downsampling convolutional layer with a kernel size of 1×1;

[0259] The scaling feature is input into the Haar wavelet transform module, and the scaling feature is... Convolution operations are performed with four different two-dimensional filters to obtain four sub-band features, which are the first sub-band features. Second sub-band characteristics Third sub-band characteristics and fourth sub-band features As shown in the following formula:

[0260] ;

[0261] ;

[0262] ;

[0263] ;

[0264] in, This represents the convolution operation. , , and Let represent the first filter, the second filter, the third filter, and the fourth filter, respectively, and their expressions are as follows:

[0265] ;

[0266] ;

[0267] ;

[0268] ;

[0269] Features of the four sub-bands along the channel dimension , , and The input is processed by the stitching layer, and then the inter-channel information is fused through the third convolutional layer to obtain the output features of the Haar wavelet transform downsampling module. As shown in the following formula:

[0270] ;

[0271] in, [·|·] represents a convolution operation with a kernel size of 1×1, and [·|·] represents a concatenation operation along the channel dimension.

[0272] For details, please refer to Figure 7 In the Haar wavelet transform downsampling module mentioned in the embodiments of this application, the input features of the Haar wavelet transform downsampling module are first input into a channel downsampling convolutional layer with a scaling factor of 4, reducing the number of channels of the input features of the Haar wavelet transform downsampling module to one-quarter of the original number, thus obtaining the scaled features. Next, the scaling features are processed using the Haar wavelet transform module. The features are split into four sub-bands, which are then scaled using four different two-dimensional filters. The convolution operation is performed to obtain four two-dimensional filters, which are defined as follows: , , and Finally, the features of the four sub-bands are concatenated along the channel dimension, and then the information between channels is fused through a third convolutional layer with a kernel size of 1×1 to achieve 2x downsampling, thus obtaining the output features of the Haar wavelet transform downsampling module.

[0273] By integrating the Haar wavelet transform downsampling module, the Mamba-based selective state scanning structure, and the deconvolution upsampling module, a multi-scale Mamba module can be constructed.

[0274] In a specific embodiment, the omnidirectional convolution module includes a fourth convolutional layer, a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer. The fourth convolutional layer uses a convolutional operation with a kernel size of 1×1, while the square convolutional layer, the horizontal strip convolutional layer, and the vertical strip convolutional layer use convolutional operations with kernel sizes of 3×3, 1×11, and 11×1, respectively.

[0275] In the omnidirectional convolution module, the input features are first processed by the fourth convolutional layer to remove redundant channel features, resulting in filtered features, as shown in the following equation:

[0276] ;

[0277] in, This represents the input features of the omnidirectional convolutional module. Indicates filtering features, This indicates a convolution operation with a kernel size of 1×1;

[0278] Padding operations are performed on the filter features to expand them to different dimensional sizes, resulting in the first expanded filter features. Second extended filtering feature and third extended filtering features B, C, H, W, and p represent the batch size, number of channels, length, width, and padding size of the input features in the omnidirectional convolutional module, respectively. Representing the real number field; the first, second, and third extended filter features are input into a square convolutional layer, a horizontal strip convolutional layer, and a vertical strip convolutional layer, respectively, to obtain square features. Horizontal characteristics and vertical features As shown in the following formula:

[0279] ;

[0280] ;

[0281] ;

[0282] in, This indicates a convolution operation with a kernel size of 3×3. This represents a convolution operation with a kernel size of 1×11. This indicates a convolution operation with a kernel size of 11×1;

[0283] The square, horizontal, and vertical features are respectively concatenated with the filtered features using residual connections to obtain the output features of the omnidirectional convolutional module. As shown in the following formula:

[0284] .

[0285] For details, please refer to Figure 8In the omnidirectional convolution-multi-scale Mamba module mentioned in the embodiments of this application, the input features of the omnidirectional convolution module first pass through a fourth convolutional layer with a kernel size of 1×1 to remove redundant channel features, thus obtaining filtered features. Ensure that the extracted features retain the characteristics of the filtered features. For the same dimensions, filtering is required first. A padding operation is performed to expand the dimensions, resulting in three expanded filter features: a first expanded filter feature, a second expanded filter feature, and a third expanded filter feature. These three expanded filter features, output after the padding operation, are then input into three convolutional layers with different orientations: a square convolutional layer, a horizontal stripe convolutional layer, and a vertical stripe convolutional layer. This extracts features in different orientations: square features, horizontal features, and vertical features. Next, residual concatenation is performed between the square features, horizontal features, and vertical features and the filter features to obtain the output features of the omnidirectional convolutional module.

[0286] In a specific embodiment, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt gated convolutional feedforward networks. The gated convolutional feedforward network includes a fifth convolutional layer, a third depthwise separable convolutional layer, a dimension splitting unit, a GELU activation function layer, and a sixth convolutional layer.

[0287] In a gated convolutional feedforward network, the input features are first fed into the fifth convolutional layer for channel-wise projection expansion to obtain the initial projected expanded features. As shown in the following formula:

[0288] ;

[0289] in, This represents the input features of a gated convolutional feedforward network. This indicates that a channel-wise projection expansion operation is performed using the fifth convolutional layer. B, C, H, and W represent the batch size, number of channels, length, and width of the input features of the gated convolutional feedforward network, respectively. Represents the real number field;

[0290] After padding the initial projected extended features, a third depthwise separable convolutional layer is used for feature extraction to obtain the first projected extended features. As shown in the following formula:

[0291] ;

[0292] in, This indicates a third-depth separable convolutional layer;

[0293] Extend the first projection feature The input is fed into the dimension segmentation unit, and average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features The second projection extension feature After being input into the GELU activation function layer, it is then combined with the third projection extended features. Multiplying yields the fourth projective extended feature. As shown in the following formula:

[0294] ;

[0295] in, Represents the GELU activation function layer;

[0296] Extend the fourth projection feature The input is fed into the sixth convolutional layer for channel-wise projection reduction to obtain the output features of the gated convolutional feedforward network. As shown in the following formula:

[0297] ;

[0298] in, This indicates the sixth convolutional layer.

[0299] For details, please refer to Figure 9 In the gated convolutional feedforward network of the embodiments of this application, the input features of the gated convolutional feedforward network are first projected and expanded along the channels using a fifth convolutional layer with a kernel size of 1×1 to obtain the initial projected expanded features. Then, the initial projection is extended to feature. After padding, feature extraction is performed using a third depthwise separable convolutional layer with a kernel size of 3×3 to obtain the first projected extended features. Next, the first projection extension feature is... In the input dimension segmentation unit, average segmentation is performed along the channel dimension to obtain the second projected extended feature. and third projection extension features , for the second projection extension feature Nonlinear transformation is achieved using the GELU activation function, with third projection extending features. The gating unit is multiplied with the second projected extended feature after the GELU activation function to enhance the representational power of the feature, resulting in the fourth projected extended feature. Finally, the fourth projection feature is expanded by a sixth convolutional layer with a kernel size of 1×1. A projection reduction operation is performed along the channel to obtain the output features of the gated convolutional feedforward network.

[0300] S2: The low-resolution image to be reconstructed is input into the trained image super-resolution model. It first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

[0301] Specifically, the trained image super-resolution model is deployed, referring to... Figure 2 During the inference phase, for a given low-resolution image First, shallow features are extracted using a first convolutional layer with a kernel size of 1×1. Its expression is as follows:

[0302] ;

[0303] in, This indicates a convolution operation with a 1×1 kernel;

[0304] Secondly, deep features are extracted using N omnidirectional convolutional multi-scale Mamba modules. Its expression is as follows:

[0305] ;

[0306] in, This represents the i-th omnidirectional convolutional-multiscale Mamba module. This represents the output feature of the i-th omnidirectional convolutional-multi-scale Mamba module. N is typically set to 4. The output features of the Nth omnidirectional convolutional-multiscale Mamba module are used as depth features.

[0307] Finally, a second convolutional layer with a 1×1 kernel is used to process the depth features. Reconstruction is performed to generate the corresponding high-resolution image. .

[0308] Further reference Figure 10 As an implementation of the methods shown in the above figures, this application provides an embodiment of an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0309] This application provides an image super-resolution device based on omnidirectional convolution and multi-scale Mamba, comprising:

[0310] Model building module 1 is configured to build and train an image super-resolution model based on omnidirectional convolution and multi-scale Mamba to obtain a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, N omnidirectional convolution-multi-scale Mamba modules, and a second convolutional layer connected in sequence. The omnidirectional convolution-multi-scale Mamba module includes a mobile-end inverted bottleneck convolutional layer, a multi-scale Mamba module, a first gated convolutional feedforward network, an omnidirectional convolutional module, and a second gated convolutional feedforward network connected in sequence.

[0311] Reconstruction module 2 is configured to input the low-resolution image to be reconstructed into the trained image super-resolution model. The image first passes through the first convolutional layer to extract shallow features. The shallow features then pass through N omnidirectional convolutional-multi-scale Mamba modules to extract depth features. The depth features are then reconstructed using the second convolutional layer to obtain the corresponding high-resolution image.

[0312] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 7 As shown, the electronic device in this embodiment includes a processor 1101 and a memory 1102; wherein the memory 1102 is used to store computer execution instructions; and the processor 1101 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0313] Alternatively, the memory 1102 can be either standalone or integrated with the processor 1101.

[0314] When the memory 1102 is set up independently, the electronic device also includes a bus 1103 for connecting the memory 1102 and the processor 1101.

[0315] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by the processor 1101, implement the above method.

[0316] This invention also provides a computer program product, including a computer program that, when executed by a processor 1101, implements the above-described method.

[0317] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0318] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0319] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0320] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 1101 to execute some steps of the methods of the various embodiments of this application.

[0321] It should be understood that the processor 1101 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 1101 can be any conventional processor 1101. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 1101 executing the steps, or as a combination of hardware and software modules within the processor 1101 executing the steps.

[0322] The memory 1102 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0323] Bus 1103 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1103 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 1103 in the accompanying drawings of this application is not limited to only one bus 1103 or one type of bus 1103.

[0324] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0325] An exemplary storage medium is coupled to a processor 1101, enabling the processor 1101 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 1101. The processor 1101 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 1101 and the storage medium can exist as discrete components in an electronic device or a host device.

[0326] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0327] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An omnidirectional convolution and multi-scale Mamba based image super-resolution method, characterized in that, The method comprises the following steps: Constructing and training an omnidirectional convolution and multi-scale Mamba-based image super-resolution model to obtain a trained image super-resolution model; the image super-resolution model comprises a first convolution layer, N omnidirectional convolution-multi-scale Mamba modules and a second convolution layer connected in sequence, and the omnidirectional convolution-multi-scale Mamba module comprises a mobile inverted bottleneck convolution layer, a multi-scale Mamba module, a first gated convolution feedforward network, an omnidirectional convolution module and a second gated convolution feedforward network connected in sequence; inputting a low-resolution image to be reconstructed into the trained image super-resolution model, first passing through the first convolution layer to extract a shallow feature, sequentially passing the shallow feature through N omnidirectional convolution-multi-scale Mamba modules to extract a deep feature, and reconstructing the deep feature by using the second convolution layer to obtain a corresponding high-resolution image.

2. The omnidirectional convolution and multi-scale Mamba-based image super-resolution method of claim 1, wherein, The multi-scale Mamba module comprises a first depth separable convolution layer, a Haar wavelet transform down-sampling module, a deconvolution up-sampling module, a feature fusion module, a first selective state scanning module and a second selective state scanning module; the first selective state scanning module and the second selective state scanning module both adopt a Mamba-based selective state scanning structure; In the multi-scale Mamba module, the input feature of the multi-scale Mamba module first passes through the first depth separable convolution layer to obtain first channel features and second channel features, as shown in the following formula: ； wherein, represents a first depth separable convolutional layer, represents an input feature of the multi-scale Mamba module, and respectively represent a first channel feature and a second channel feature; The first channel features are input into the Haar wavelet transform down-sampling module to obtain first intermediate features, as shown in the following formula: ； wherein, denotes a Haar wavelet transform down-sampling module, denotes a first intermediate feature; The first intermediate features are input into the first selective state scanning module for selective scanning to obtain first output features, as shown in the following formula: ； wherein, represents a selective state scan structure based on Mamba, represents a first output feature; The second channel features are input into the second selective state scanning module for selective scanning to obtain second output features, as shown in the following formula: ； wherein represents the second output feature; The first output features are input into the deconvolution up-sampling module to obtain third output features, as shown in the following formula: ； wherein, denotes a deconvolution up-sampling module, denotes a third output feature; The second output features are input into the feature fusion module to obtain output features of the multi-scale Mamba module, as shown in the following formula: ； wherein, denotes a feature fusion module, denotes the output features of the multi-scale Mamba module.

3. The omnidirectional convolution and multi-scale Mamba-based image super-resolution method of claim 2, wherein, The deconvolution up-sampling module comprises a zero interpolation up-sampling layer, a nearest neighbor interpolation up-sampling layer and a frequency domain information reconstruction module; In the deconvolution up-sampling module, the input feature of the deconvolution up-sampling module is first subjected to a padding operation to obtain second intermediate features, as shown in the following formula: ； wherein, , represents a padding operation, represents a second intermediate feature, B, C, H, W and p represent batch, channel number, length, width and padding size of the input feature of the Haar wavelet transform down-sampling module, respectively, represents a real number field; The second intermediate features are respectively input into the zero interpolation up-sampling layer and the nearest neighbor interpolation up-sampling layer to obtain third intermediate features and fourth intermediate features, as shown in the following formula: ； ； wherein, denotes a zero-interpolation up-sampling layer, denotes a nearest-neighbor interpolation up-sampling layer, denotes a third intermediate feature and a fourth intermediate feature, respectively; Create The all-zero matrix represents the parameters of the convolution kernel of the point spread function. The center point of the point spread function's convolution kernel is moved from its original position by zero-padding the top-left corner of the all-zero matrix and then performing a cyclic center shift operation. Move to the position of the all-zero matrix The first adjustment feature is obtained. Its expression is: ； wherein, represents a zero padding and cyclic center shift operation, 1, C, k1, k2 represent batch size, channel number, length, width of the parameters of the convolution kernel of the point spread matrix, respectively. The first adjustment feature is first subjected to a discrete Fourier transform, which transforms it from a time domain variable to a frequency domain variable to obtain an optical transfer feature The optical transfer feature is then subjected to a complex conjugate calculation and a modulus square calculation, respectively, to obtain a complex conjugate feature and a modulus square feature as shown in the following formula: ； ； ； wherein, denotes a discrete Fourier transform operation, denotes a complex conjugate computation, denotes a modulo square computation, denotes a complex number field; Discrete Fourier transform is performed on the third intermediate feature and the fourth intermediate feature respectively, so as to transform them from time domain variables to frequency domain variables, and fifth intermediate features and sixth intermediate features are obtained Wiener filtering is performed on the fifth intermediate features based on the complex conjugate features, and filtered features are obtained Regularization is performed on the sixth intermediate features, and regularized features are obtained The filtered features and the regularized features are added, and regularized frequency domain features are obtained as shown in the following formula:​ ； ； ； ； ； wherein denotes a regularization parameter; multiplying the regularized frequency domain feature and the optical transfer feature to obtain a fusion feature as shown in the following formula: ； The fusion feature and the modulus square feature are respectively subjected to a block average operation to obtain a first block average feature and a second block average feature and a second block average feature as shown in the following formula: ； ； wherein denotes a block averaging operation; performing an inverse filtering operation on the first and second block average features to obtain inverse filtered features as shown in the following equation: ； up-sampling and expanding the inverse filtering feature to obtain an expanded feature The regularization frequency domain feature, the complex conjugate feature and the expanded feature are input into the frequency domain information reconstruction module for frequency domain information reconstruction to obtain a reconstructed feature as shown in the following formula: ； performing inverse discrete Fourier transform on the reconstructed features, converting them from frequency domain variables to time domain variables and taking real part operation to obtain second adjustment features as shown in the following formula: ； wherein, denotes an inverse discrete Fourier transform operation, denotes a take real part operation; performing a de-padding operation on the second adjustment feature to obtain an output feature of de-convolution up-sampling ; The Mamba-based selective state scanning structure comprises a linear projection layer, a second depth separable convolution layer, a first SiLU activation function layer, a second SiLU activation function layer and a two-dimensional selective scanning module, and the two-dimensional selective scanning module comprises a multi-directional scanning module and a Mamba-based selective state space calculation module; In the Mamba-based selective state scan structure, an input feature of the Mamba-based selective state scan structure First, the linear projection layer is passed to expand the channel dimension to obtain the projection feature As shown in the following formula: ； wherein represents the linear projection layer; The projection feature is subjected to a dimension splitting operation to obtain a first dimension feature and a second dimension feature and a second dimension feature The expression is: ； wherein, represents a dimension split operation; The first dimension feature is sequentially subjected to a second deep separable convolution layer, a padding operation and a first SiLU activation function layer to obtain a convolution feature As shown in the following formula: ； wherein, denotes a SiLU activation function, denotes a second depthwise separable convolutional layer; The convolution feature is input into the two-dimensional selective scanning module, and the convolution feature is expanded into four one-dimensional sequences in four directions by the multi-direction scanning module, respectively, first sequence , second sequence , third sequence , and fourth sequence ; wherein and are expanded along the positive direction of the horizontal direction and the positive direction of the vertical direction, and are expanded along the reverse direction of the horizontal direction and the reverse direction of the vertical direction. The dimension mapping operation is respectively performed on the four one-dimensional sequences, and a channel dimension C is projected to , to obtain a joint tensor , as shown in the following formula: ； wherein, represents a dimension mapping operation, is a time step correction dimension, is a state parameter dimension; splitting the joint tensor along the channel dimension into a correction matrix , a state transition matrix A, an input influence matrix B g , a state output matrix C g and a feedforward matrix D g ; The modified matrix , state transition matrix A, input influence matrix B g , state output matrix C g and feedforward matrix D g is input into the Mamba-based selective state space computation module to obtain enhanced features as follows: ； ； ； ； ； wherein, denotes the input feature at the t-th time step; denotes the hidden state at the t-1-th time step, when t = 1, , ; denotes the hidden state at the t-th time step, denote the modified state transition matrix and input influence matrix, respectively; denotes the output feature at the t-th time step, denotes the output feature at the L-th time step, denotes the augmented feature, denotes concatenation by time step, ; The second dimension feature passes through a second SiLU activation function layer and is multiplied by the enhanced feature to obtain an output feature of the Mamba-based selective state scanning structure As shown in the following formula: 。 4. The omnidirectional convolution and multi-scale Mamba-based image super-resolution method of claim 2, wherein, The Haar wavelet transform down-sampling module comprises a channel down-sampling convolution layer, a Haar wavelet transform module, a splicing layer and a third convolution layer, and the third convolution layer adopts a convolution operation with a convolution kernel size of 1*1; In the Haar wavelet transform down-sampling module, the input features of the Haar wavelet transform down-sampling module are first input to a channel down-sampling convolution layer with a scaling factor of 4 to obtain scaling features As shown in the following formula: ； wherein, denotes a channel down-sampling convolutional layer with a kernel size of 1x1; inputting the scaling features into the Haar wavelet transform module, the scaling features convolving with four different two-dimensional filters respectively to obtain four sub-band features, respectively, first sub-band features , second sub-band features , third sub-band features and fourth sub-band features , as shown in the following formula: ； ； ； ； wherein, denotes a convolution operation, , , and denote a first filter, a second filter, a third filter and a fourth filter, respectively, whose expressions are as follows: ； ； ； ； Features of the four sub-bands along the channel dimension , , and The input is processed by the stitching layer, and then the inter-channel information is fused through the third convolutional layer to obtain the output features of the Haar wavelet transform downsampling module. As shown in the following formula: ； wherein denotes a convolution operation with a kernel size of 1x1, and [·|·] denotes a concatenation operation along the channel dimension.

5. The omnidirectional convolution and multi-scale Mamba-based image super-resolution method of claim 1, wherein, The omnidirectional convolution module comprises a fourth convolution layer, a square convolution layer, a horizontal strip convolution layer and a vertical strip convolution layer, the fourth convolution layer adopts a convolution operation with a convolution kernel size of 1*1, the square convolution layer, the horizontal strip convolution layer and the vertical strip convolution layer adopt convolution operations with convolution kernel sizes of 3*3, 1*11 and 11*1 respectively; In the omnidirectional convolution module, the input features of the omnidirectional convolution module are first subjected to the fourth convolution layer to remove redundant channel features and obtain filtered features, as shown in the following formula: ； wherein, denotes an input feature of the omnidirectional convolution module, denotes a filtered feature, denotes a convolution operation with a kernel size of 1x1. Padding operation is performed on the filtering features respectively, and the filtering features are expanded to different dimension sizes to obtain first, second and third expanded filtering features B, C, H, W and p respectively represent batch, channel number, length, width and padding size of the input feature of the omnidirectional convolution module, represents a real number field; the first, second and third expanded filtering features are respectively input into the square convolution layer, the horizontal strip convolution layer and the vertical strip convolution layer to obtain square features as shown in the following formula:​​​​ ； ； ； wherein denotes a convolution operation with a kernel size of 3x3, denotes a convolution operation with a kernel size of 1x11, denotes a convolution operation with a kernel size of 11x1; The square feature, the horizontal feature and the vertical feature are respectively connected with the filtering feature by residual connection to obtain an output feature of the omnidirectional convolution module As shown in the following formula: 。 6. The omnidirectional convolution and multi-scale Mamba-based image super-resolution method of claim 1, wherein, Both the first gated convolution feedforward network and the second gated convolution feedforward network adopt a gated convolution feedforward network, and the gated convolution feedforward network comprises a fifth convolution layer, a third depth separable convolution layer, a dimension segmentation unit, a GELU activation function layer and a sixth convolution layer; In the gated convolution feedforward network, input features of the gated convolution feedforward network are first input into the fifth convolution layer for a channel projection expansion operation to obtain initial projection expansion features as shown in the following formula: ； wherein, represents an input feature of the gated convolution feedforward network, represents a channel-wise projection expansion operation with a fifth convolution layer, B, C, H, W represent batch, channel number, length, width of the input feature of the gated convolution feedforward network, respectively, represents a real number field; Padding operation is performed on the initial projection expansion feature, and a third depth separable convolution layer is used for feature extraction to obtain a first projection expansion feature As shown in the following formula: ； wherein, denotes a third depthwise separable convolutional layer; The first projection expansion feature is input into a dimension segmentation unit, and is averaged segmented along a channel dimension to obtain a second projection expansion feature and a third projection expansion feature The second projection expansion feature is input into a GELU activation function layer, and then multiplied with the third projection expansion feature to obtain a fourth projection expansion feature as shown in the following formula: ； wherein, represents a GELU activation function layer; The fourth projection extended feature Input to the sixth convolutional layer for a channel-wise projection reduction operation to obtain the output feature of the gated convolutional feedforward network As shown in the following formula: ； wherein, represents a sixth convolutional layer.

7. An omnidirectional convolution and multi-scale Mamba based image super-resolution device, characterized in that, Comprise: A model construction module configured to construct an image super-resolution model based on omnidirectional convolution and multi-scale Mamba and train the image super-resolution model to obtain a trained image super-resolution model; the image super-resolution model comprises a first convolution layer, N omnidirectional convolution-multi-scale Mamba modules and a second convolution layer connected in sequence, and the omnidirectional convolution-multi-scale Mamba module comprises a mobile end inverted bottle neck convolution layer, a multi-scale Mamba module, a first gated convolution feedforward network, an omnidirectional convolution module and a second gated convolution feedforward network connected in sequence; A reconstruction module configured to input a low-resolution image to be reconstructed into the trained image super-resolution model, first pass through the first convolution layer to extract shallow features, sequentially pass the shallow features through N omnidirectional convolution-multi-scale Mamba modules to extract deep features, and use the second convolution layer to reconstruct the deep features to obtain a corresponding high-resolution image. 8.An electronic device, comprising: one or more processors; storage configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6. The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images