Snake convolution motion deblurring method and device based on Mamba

Abstract

The present invention relates to the field of image processing technology, and provides a Mamba-based serpentine convolution motion deblurring method and device. Each serpentine processing module cascaded in the encoder and decoder of the image deblurring network used in the method includes a serpentine state space module and a serpentine channel attention module connected in sequence. The shape and path of the convolution kernel are adjusted by the serpentine state space module to more effectively capture the stripe blur characteristics and finely align the local structure. The serpentine state space module increases the perception of small blurs and improves the deblurring performance of the image deblurring network. The serpentine channel attention module is used to screen and optimize the blur information to ensure that key details are retained while removing blur, further improving the accuracy and quality of image restoration. Combining the serpentine state space module and the serpentine channel attention module solves the problem of high computational consumption of Transformer in the deblurring process.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a Mamba-based serpentine convolution motion deblurring method and device. Background Art

[0002] Images are an important way for people to obtain information. However, in the imaging process, the images obtained are often blurred due to unfavorable factors such as camera shake, movement of objects in the scene, and camera defocus, which greatly hinders the normal use and subsequent processing of the images.

[0003] Image deblurring is a basic image restoration problem in image processing and computer vision. Its goal is to restore a blurred image caused by motion factors such as camera shake and object motion to a clear image. Most existing image deblurring methods usually rely on neural network models, which implement the image restoration process by inputting a blurred image into the neural network model and outputting a clear image from the neural network model.

[0004] However, traditional neural network models are usually not good at fully restoring the clear details of the image when dealing with image blur, which in turn affects the quality of the final image restoration. Summary of the invention

[0005] The present invention provides a Mamba-based serpentine convolution motion deblurring method and device, which are used to solve the defects in the prior art.

[0006] The present invention provides a Mamba-based serpentine convolution motion deblurring method, comprising:

[0007] Get the image to be processed;

[0008] Applying an image deblurring network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0009] The image deblurring network comprises an encoder and a decoder connected in sequence, the encoder comprises an input convolution layer and a plurality of cascaded serpentine processing modules, and the decoder comprises a plurality of cascaded serpentine processing modules and an output convolution layer;

[0010] Each serpentine processing module includes a serpentine state space module and a serpentine channel attention module connected in sequence. The serpentine state space module is used to divide the input along the channel equally, and fuse them respectively after performing serpentine convolution operations to obtain a first fusion result, and the first fusion result is superimposed with the input to obtain an output after reducing the number of channels; the serpentine channel attention module is used to divide the input along the channel equally, and fuse them after performing serpentine convolution operations to obtain a second fusion result, and the channel information of the second fusion result is screened based on the channel attention mechanism to obtain fuzzy residual features, and the fuzzy residual features are superimposed with the input to obtain an output.

[0011] According to a Mamba-based serpentine convolution motion deblurring method provided by the present invention, the serpentine state space module comprises: a first normalization layer, a channel division layer, a first serpentine branch, a second serpentine branch and a fusion module;

[0012] The first normalization layer is used to perform layer normalization processing on the input to obtain the output;

[0013] The channel division layer is used to divide the output of the first normalization layer into equal parts along the channel to obtain a first equal division result and a second equal division result;

[0014] The first serpentine branch is used to perform two serpentine convolution operations on the first equally divided result in sequence and then activate the result to obtain a first activation result, and obtain an output based on the first activation result;

[0015] The second snake-shaped branch is used to perform a snake-shaped convolution operation on the second equally divided result and then activate it to obtain an output;

[0016] The fusion module is used to fuse the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result, and reduce the number of channels of the first fusion result and superimpose it with the input of the serpentine state space module to obtain the output.

[0017] According to a Mamba-based serpentine convolution motion deblurring method provided by the present invention, the fusion module includes a first product layer, a first convolution layer and a first superposition layer;

[0018] The first product layer is used to perform Hadamard product on the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result;

[0019] The first convolution layer is used to perform a convolution operation on the first fusion result to obtain a result with a reduced number of channels;

[0020] The first superposition layer is used to superimpose the channel number reduction result with the input of the serpentine state space module to obtain the output of the serpentine state space module.

[0021] According to a Mamba-based serpentine convolution motion deblurring method provided by the present invention, the first serpentine branch includes a direction scanning module;

[0022] The direction scanning module is used to scan the first activation result along multiple directions, and based on the state space equation, capture the long-range dependency in the scanning result to obtain the output of the first serpentine branch.

[0023] According to a Mamba-based serpentine convolution motion deblurring method provided by the present invention, the direction scanning module is specifically used for:

[0024] Flattening the first activation result into a one-dimensional sequence, and scanning the one-dimensional sequence along multiple directions to obtain the scanning result; the scanning result includes multiple scanning sequences;

[0025] Based on the state space equation, the long-range dependency of the multiple scanning sequences is captured, the multiple scanning sequences are merged, and the two-dimensional structure is restored through a reshaping operation to obtain the output of the first serpentine branch.

[0026] According to a Mamba-based serpentine convolution motion deblurring method provided by the present invention, the serpentine channel attention module includes a channel attention layer, and the channel attention layer includes a global pooling layer, a feature reshaping layer, a second convolution layer, and a second product layer;

[0027] The global pooling layer is used to perform global average pooling on each channel of the second fusion result to obtain a global spatial feature of each channel;

[0028] The feature reshaping layer is used to reshape the global spatial features of each channel into a one-dimensional vector;

[0029] The second convolutional layer is used to perform a convolution operation on the one-dimensional vector of each channel to obtain the weight of each channel;

[0030] The second product layer is used to multiply the global spatial feature of each channel by the weight to obtain the fuzzy residual feature.

[0031] According to a Mamba-based serpentine convolution motion deblurring method provided by the present invention, the serpentine channel attention modules in each serpentine processing module of the encoder are respectively connected to the serpentine state space modules in the serpentine processing modules at symmetrical positions in the decoder.

[0032] The present invention also provides a Mamba-based serpentine convolution motion deblurring device, comprising:

[0033] An image acquisition module, used for acquiring an image to be processed;

[0034] A deblurring module, used for applying an image deblurring network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0035] The image deblurring network comprises an encoder and a decoder connected in sequence, the encoder comprises an input convolution layer and a plurality of cascaded serpentine processing modules, and the decoder comprises a plurality of cascaded serpentine processing modules and an output convolution layer;

[0036] Each serpentine processing module includes a serpentine state space module and a serpentine channel attention module connected in sequence. The serpentine state space module is used to divide the input along the channel equally, and fuse them respectively after performing serpentine convolution operations to obtain a first fusion result, and the first fusion result is superimposed with the input to obtain an output after reducing the number of channels; the serpentine channel attention module is used to divide the input along the channel equally, and fuse them after performing serpentine convolution operations to obtain a second fusion result, and the channel information of the second fusion result is screened based on the channel attention mechanism to obtain fuzzy residual features, and the fuzzy residual features are superimposed with the input to obtain an output.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the Mamba-based serpentine convolution motion deblurring method as described above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the Mamba-based serpentine convolution motion deblurring method described above is implemented.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the Mamba-based serpentine convolution motion deblurring method is implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The Mamba-based serpentine convolution motion deblurring method and device provided by the present invention first obtains an image to be processed; then applies an image deblurring network to deblur the image to be processed, and obtains a deblurred image of the image to be processed. The image deblurring network includes an encoder and a decoder connected in sequence, and each cascaded serpentine processing module in the encoder and the decoder includes a serpentine state space module and a serpentine channel attention module connected in sequence. The shape and path of the convolution kernel can be adjusted through the serpentine state space module to more effectively capture the stripe blur characteristics and finely align the local structure. The serpentine state space module increases the perception ability of small blurs and improves the deblurring performance of the image deblurring network. The serpentine channel attention module can be used to screen and optimize the blur information to ensure that key details are retained while removing blur, further improving the accuracy and quality of image restoration. Combining the serpentine state space module and the serpentine channel attention module can solve the problem of high computational consumption of Transformer during the deblurring process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without paying any creative work.

[0043] Figure 1 It is a flowchart of the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0044] Figure 2 It is one of the structural schematic diagrams of the image deblurring network in the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0045] Figure 3 This is the second structural schematic diagram of the image deblurring network in the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0046] Figure 4 It is a structural schematic diagram of the SSSM in the image deblurring network in the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0047] Figure 5 It is a structural schematic diagram of the DSM in the image deblurring network in the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0048] Figure 6 It is a structural schematic diagram of SCA in the image deblurring network in the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0049] Figure 7 It is a schematic diagram of the operation flow of SCA in the image deblurring network in the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0050] Figure 8 It is a schematic diagram of four images to be processed input into the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0051] Fig. 9 Schematic diagram of four deblurred images outputted from the Mamba-based serpentine convolution motion deblurring method provided by the present invention;

[0052] Fig.10 It is a structural schematic diagram of a Mamba-based serpentine convolution motion deblurring device provided by the present invention;

[0053] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Figure 1 FIG. 1 is a flow chart of a Mamba-based serpentine convolution motion deblurring method provided in an embodiment of the present invention, as shown in FIG. Figure 1 As shown, the method includes:

[0056] S1, obtaining the image to be processed;

[0057] S2, applying an image deblurring network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0058] The image deblurring network comprises an encoder and a decoder connected in sequence, the encoder comprises an input convolution layer and a plurality of cascaded serpentine processing modules, and the decoder comprises a plurality of cascaded serpentine processing modules and an output convolution layer;

[0059] Each serpentine processing module includes a serpentine state space module and a serpentine channel attention module connected in sequence. The serpentine state space module is used to divide the input along the channel equally, and fuse them respectively after performing serpentine convolution operations to obtain a first fusion result, and the first fusion result is superimposed with the input to obtain an output after reducing the number of channels; the serpentine channel attention module is used to divide the input along the channel equally, and fuse them after performing serpentine convolution operations to obtain a second fusion result, and the channel information of the second fusion result is screened based on the channel attention mechanism to obtain fuzzy residual features, and the fuzzy residual features are superimposed with the input to obtain an output.

[0060] Specifically, the Mamba-based serpentine convolution motion deblurring method provided in an embodiment of the present invention is executed by a Mamba-based serpentine convolution motion deblurring device, which can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0061] First, step S1 is executed to obtain an image to be processed. The image to be processed refers to a blurred image with motion blur that needs to be deblurred. It can be a large-size substrate image or an image containing a target object such as a face or a vehicle in the monitoring field, which is not specifically limited here. The image to be processed can be acquired by an image acquisition device. The image to be processed can be a color image or a grayscale image.

[0062] Then, step S2 is performed to apply an image deblurring network to deblur the image to be processed, thereby obtaining a deblurred image of the image to be processed.

[0063] Among them, Figure 2 As shown, the image deblurring network (MSNet) includes an encoder (Encoder) and a decoder (Decoder) connected in sequence, the encoder includes an input convolution layer (Conv1) and multiple cascaded snake processing modules, and the decoder includes multiple cascaded snake processing modules and an output convolution layer (Conv2).

[0064] Each snake processing module includes a snake state space module (SSSM) and a snake channel attention module (SCA) connected in sequence. The SSSM combines the Mamba framework with snake convolution to divide the input equally along the channel, and fuses them after one or more snake convolution operations to obtain a first fusion result. The first fusion result is reduced in the number of channels and then superimposed with the input to obtain the output. The SSSM can dynamically adapt to the local features of the image and adjust the convolution direction by adjusting the shape and path of the convolution kernel to align different blurred stripe patterns.

[0065] SCA is used to divide the input equally along the channel, perform snake convolution operations on each of them, and then fuse them to obtain the second fusion result. The second fusion result is filtered for channel information based on the channel attention mechanism to obtain fuzzy residual features, and the fuzzy residual features are superimposed on the input to obtain the output. SCA can use gated filtering and adjust the weight of fuzzy information to calibrate the feature map to ensure that key details are retained while removing blur.

[0066] The Mamba-based serpentine convolution motion deblurring method provided in an embodiment of the present invention first obtains an image to be processed; then an image deblurring network is applied to deblur the image to be processed to obtain a deblurred image of the image to be processed. The image deblurring network includes an encoder and a decoder connected in sequence, and each cascaded serpentine processing module in the encoder and the decoder includes a serpentine state space module and a serpentine channel attention module connected in sequence. The shape and path of the convolution kernel can be adjusted through the serpentine state space module to more effectively capture the stripe blur characteristics and finely align the local structure. The serpentine state space module increases the perception of small blurs and improves the deblurring performance of the image deblurring network. The serpentine channel attention module can be used to screen and optimize the blur information to ensure that key details are retained while removing blur, further improving the accuracy and quality of image restoration. Combining the serpentine state space module and the serpentine channel attention module can solve the problem of high computational consumption of Transformer during the deblurring process.

[0067] On the basis of the above embodiment, the SCA in each serpentine processing module of the encoder is respectively connected to the SSSM in the serpentine processing modules at symmetrical positions in the decoder.

[0068] Specifically, Figure 3As shown, the encoder and the decoder both include three cascaded serpentine processing modules, the SCA in the first serpentine processing module of the encoder is connected to the SSSM in the third serpentine processing module of the decoder, the SCA in the second serpentine processing module of the encoder is connected to the SSSM in the second serpentine processing module of the decoder, and the SCA in the third serpentine processing module of the encoder is connected to the SSSM in the first serpentine processing module of the decoder. In this way, the deblurring performance of the image deblurring network can be further improved by increasing the input of the SSSM in each serpentine processing module of the decoder.

[0069] Based on the above embodiments, Figure 4 As shown, the SSSM includes: a first normalization layer LayerNorm1, a first channel division layer, a first serpentine branch, a second serpentine branch and a fusion module.

[0070] The first normalization layer LayerNorm1 is used to perform layer normalization on the input of SSSM, standardize the input and obtain the output.

[0071] The first channel division layer is used to divide the output of the first normalization layer equally along the channel to obtain the first equal division result. and the second equivalence results .

[0072] The first snake branch may include a first snake convolution layer SnakeConv1, a second snake convolution layer SnakeConv2, and a first activation layer Gelu1. The first snake convolution layer SnakeConv1 is used to divide the first result equally. A snake convolution operation is performed to obtain a first snake convolution operation result, and the second snake convolution layer SnakeConv2 is used to perform a snake convolution operation on the first snake convolution operation result to obtain a second snake convolution operation result. The second snake convolution operation result is a detail blur feature.

[0073] The first activation layer Gelu1 is used to activate the second snake convolution operation result to obtain a first activation result.

[0074] Afterwards, the output is obtained through the first activation result. For example, the first activation result can be directly used as the output of the first serpentine branch.

[0075] Based on the above embodiments, Figure 4As shown, the first serpentine branch includes a direction scanning module (Direction Scan Module, DSM), and the DSM can be connected between the first activation layer Gelu1 and the first product layer. The DSM is used to scan the first activation result along multiple directions, and based on the state space equation, capture the long-range dependency in the scanning result to obtain the output of the first serpentine branch. That is:

[0076] ;

[0077] in, The two-layer snake convolution operation implemented for the first snake convolution layer SnakeConv1 and the second snake convolution layer SnakeConv2, is the output of the first serpentine branch.

[0078] In the embodiment of the present invention, the DSM is used to merge multi-directional structural information, so as to enhance the image deblurring network's ability to capture global structures.

[0079] The second serpentine branch may include a third serpentine convolution layer SnakeConv3 and a second activation layer Gelu2, and the third serpentine convolution layer SnakeConv3 is used to perform a serpentine convolution operation on the second equally divided result to obtain a third serpentine convolution operation result. The third serpentine convolution operation result may be a clear latent image feature.

[0080] The second activation layer Gelu2 is used to activate the result of the third snake convolution operation to obtain the output of the second snake branch. That is:

[0081] ;

[0082] in, is the snake convolution operation of the third snake convolution layer SnakeConv3, is the output of the second serpentine branch.

[0083] Here, the first snake convolution layer SnakeConv1, the second snake convolution layer SnakeConv2, and the third snake convolution layer SnakeConv3 can all perform 3×3 snake convolution operations.

[0084] The fusion module can fuse the output of the first serpentine branch with the output of the second serpentine branch to obtain a first fusion result, and reduce the number of channels of the first fusion result and superimpose it with the input of the SSSM to obtain the output of the SSSM.

[0085] Based on the above embodiments, Figure 4 As shown, the fusion module includes the first product layer, the first convolution layer Conv3 and the first superposition layer;

[0086] The first product layer is used to perform Hadamard product on the outputs of the first serpentine branch and the second serpentine branch to obtain a first fusion result.

[0087] The first convolutional layer Conv3 is used to perform a convolution operation on the first fusion result to obtain a result with a reduced number of channels.

[0088] The first stacking layer is used to stack the channel number reduction result with the SSSM input X to obtain the output of the serpentine state space module. That is:

[0089] .

[0090] Based on the above embodiments, Figure 5 As shown, DSM is specifically used for:

[0091] The first activation result is flattened into a one-dimensional sequence, that is, the two-dimensional first activation result is flattened (Unfold) into a one-dimensional sequence.

[0092] Thereafter, the one-dimensional sequence is scanned in multiple directions to obtain the scanning results. For example, the one-dimensional sequence can be scanned in four directions, from upper left to lower right, from lower right to upper left, from lower left to upper right, and from upper right to lower left, to obtain four scanning sequences.

[0093] Afterwards, the state-space equations are used to capture the long-range dependencies of each scanning sequence, merge multiple scanning sequences (Sum), and restore the two-dimensional structure through a reshaping operation to obtain the output of the first serpentine branch.

[0094] Based on the above embodiments, Figure 6 As shown, SCA includes a second normalization layer LayerNorm2, a third convolutional layer Conv4, a second channel partitioning layer, a fourth SnakeConv4, a fifth SnakeConv5, a third product layer, a channel attention layer (Channel Attention) and a second superposition layer.

[0095] The second normalization layer LayerNorm2 is used to perform layer normalization processing on the SCA input Y, standardize the input, and obtain the output.

[0096] The convolution kernel of the third convolutional layer Conv4 is 3×3, which is used to extract the fuzzy information X1 of the output of the second normalization layer LayerNorm2.

[0097] The second channel division layer is used to divide the fuzzy information X1 equally along the channel to obtain the third equal division result And the fourth quartile results .

[0098] The fourth SnakeConv4 is used to divide the third result equally A snake convolution operation is performed to obtain a fourth snake convolution operation result.

[0099] The fifth SnakeConv5 is used to divide the fourth result into equal parts. A snake convolution operation is performed to obtain a fifth snake convolution operation result.

[0100] The third product layer is used to perform Hadamard product on the result of the fourth snake convolution operation and the result of the fifth snake convolution operation to obtain the second fusion result. .

[0101] The channel attention layer is used to perform channel attention calculation on the second fusion result to obtain the fuzzy residual features.

[0102] The second superposition layer is used to superimpose the fuzzy residual features with the SCA input Y to obtain the SCA output. That is:

[0103] ;

[0104] in, is the output of SCA, and CA is the channel attention calculation function.

[0105] Based on the above embodiment, the channel attention layer includes a global pooling layer, a feature reshaping layer, a second convolution layer and a second product layer;

[0106] The global pooling layer is used to perform global average pooling on each channel of the second fusion result to obtain a global spatial feature of each channel;

[0107] The feature reshaping layer is used to reshape the global spatial features of each channel into a one-dimensional vector;

[0108] The second convolutional layer is used to perform a convolution operation on the one-dimensional vector of each channel to obtain the weight of each channel;

[0109] The second product layer is used to multiply the global spatial feature of each channel by the weight to obtain the fuzzy residual feature.

[0110] Specifically, Figure 7 As shown, the fuzzy information X1 can be expressed as a three-dimensional feature of C*W*H. The fuzzy information X1 is divided equally along the channel through the second channel division layer to obtain the third equal division result And the fourth quartile results Among them, the third equal division result And the fourth quartile results They can all be expressed as three-dimensional features of C / 2*W*H.

[0111] The third tier results The fourth SnakeConv4 is used to perform a snake convolution operation to obtain the fourth snake convolution operation result and the fourth equal division result. The fifth SnakeConv5 performs a snake convolution operation to obtain a fifth snake convolution operation result.

[0112] The fourth snake convolution operation result and the fifth snake convolution operation result are Hadamard-multiplied by the third product layer to obtain the second fusion result. The second fusion result It can be expressed as a three-dimensional feature of C*W*H.

[0113] The second fusion result is obtained through the global pooling layer (Pooling) Perform global average pooling on each channel to obtain the global spatial features of each channel. Figure 7 Second fusion result It contains n+1 channels from 0 to n.

[0114] The global spatial features of each channel are reshaped into a one-dimensional vector through a feature reshaping layer.

[0115] The one-dimensional vector of each channel is convolved through the second convolutional layer (Conv5) to obtain the weight of each channel.

[0116] The global spatial features of each channel are multiplied by the weights through the second product layer to obtain a fuzzy residual feature. The fuzzy residual feature can be represented as a three-dimensional feature of C*W*H with n+1 channels.

[0117] On the basis of the above embodiments, ablation experiments were conducted on the GoPro dataset to verify the effectiveness of each module in the proposed MSNet. The experimental results are detailed in Table 1. Ablation analysis was performed on SSSM and SCA, and the impact of different numbers of layers on the performance of MSNet was explored. The experimental results show that adding SSSM to MSNet-16 can increase PSNR by 1.98dB, introducing SCA can increase PSNR by 1.21dB, and introducing SSSM and SCA at the same time can increase PSNR by 3.54dB, which verifies the effectiveness of the proposed module. At the same time, the number of layers of SSSM and SCA was analyzed, and the results showed that the best effect was achieved at 28 layers, so MSNet-28 was selected as the final MSNet.

[0118] Table 1. Comparison of ablation experiments

[0119]

[0120] Figure 8 The four input images to be processed.

[0121] Fig. 9 These are the four deblurred output images.

[0122] like Fig.10 As shown, based on the above embodiment, a serpentine convolution motion deblurring device based on Mamba is provided in an embodiment of the present invention, comprising:

[0123] An image acquisition module 10, used to acquire an image to be processed;

[0124] A deblurring module 11 is used to apply an image deblurring network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0125] The image deblurring network comprises an encoder and a decoder connected in sequence, the encoder comprises an input convolution layer and a plurality of cascaded serpentine processing modules, and the decoder comprises a plurality of cascaded serpentine processing modules and an output convolution layer;

[0126] Each serpentine processing module includes a serpentine state space module and a serpentine channel attention module connected in sequence. The serpentine state space module is used to divide the input along the channel equally, and fuse them respectively after performing serpentine convolution operations to obtain a first fusion result, and the first fusion result is superimposed with the input to obtain an output after reducing the number of channels; the serpentine channel attention module is used to divide the input along the channel equally, and fuse them after performing serpentine convolution operations to obtain a second fusion result, and the channel information of the second fusion result is screened based on the channel attention mechanism to obtain fuzzy residual features, and the fuzzy residual features are superimposed with the input to obtain an output.

[0127] On the basis of the above embodiments, the Mamba-based serpentine convolution motion deblurring device provided in the embodiments of the present invention, the serpentine state space module includes: a first normalization layer, a channel division layer, a first serpentine branch, a second serpentine branch and a fusion module;

[0128] The first normalization layer is used to perform layer normalization processing on the input to obtain the output;

[0129] The channel division layer is used to divide the output of the first normalization layer into equal parts along the channel to obtain a first equal division result and a second equal division result;

[0130] The first serpentine branch is used to perform two serpentine convolution operations on the first equally divided result in sequence and then activate the result to obtain a first activation result, and obtain an output based on the first activation result;

[0131] The second snake-shaped branch is used to perform a snake-shaped convolution operation on the second equally divided result and then activate it to obtain an output;

[0132] The fusion module is used to fuse the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result, and reduce the number of channels of the first fusion result and superimpose it with the input of the serpentine state space module to obtain the output.

[0133] On the basis of the above embodiment, in the Mamba-based serpentine convolution motion deblurring device provided in the embodiment of the present invention, the fusion module includes a first product layer, a first convolution layer and a first superposition layer;

[0134] The first product layer is used to perform Hadamard product on the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result;

[0135] The first convolution layer is used to perform a convolution operation on the first fusion result to obtain a result with a reduced number of channels;

[0136] The first superposition layer is used to superimpose the channel number reduction result with the input of the serpentine state space module to obtain the output of the serpentine state space module.

[0137] On the basis of the above embodiment, in the Mamba-based serpentine convolution motion deblurring device provided in the embodiment of the present invention, the first serpentine branch includes a direction scanning module;

[0138] The direction scanning module is used to scan the first activation result along multiple directions, and based on the state space equation, capture the long-range dependency in the scanning result to obtain the output of the first serpentine branch.

[0139] On the basis of the above embodiments, in the Mamba-based serpentine convolution motion deblurring device provided in the embodiments of the present invention, the direction scanning module is specifically used for:

[0140] Flattening the first activation result into a one-dimensional sequence, and scanning the one-dimensional sequence along multiple directions to obtain the scanning result; the scanning result includes multiple scanning sequences;

[0141] Based on the state space equation, the long-range dependency of the multiple scanning sequences is captured, the multiple scanning sequences are merged, and the two-dimensional structure is restored through a reshaping operation to obtain the output of the first serpentine branch.

[0142] On the basis of the above embodiment, in the Mamba-based serpentine convolution motion deblurring device provided in the embodiment of the present invention, the serpentine channel attention module includes a channel attention layer, and the channel attention layer includes a global pooling layer, a feature reshaping layer, a second convolution layer and a second product layer;

[0143] The global pooling layer is used to perform global average pooling on each channel of the second fusion result to obtain a global spatial feature of each channel;

[0144] The feature reshaping layer is used to reshape the global spatial features of each channel into a one-dimensional vector;

[0145] The second convolutional layer is used to perform a convolution operation on the one-dimensional vector of each channel to obtain the weight of each channel;

[0146] The second product layer is used to multiply the global spatial feature of each channel by the weight to obtain the fuzzy residual feature.

[0147] On the basis of the above embodiments, in the Mamba-based serpentine convolution motion deblurring device provided in the embodiments of the present invention, the serpentine channel attention modules in each serpentine processing module of the encoder are respectively connected to the serpentine state space modules in the serpentine processing modules at symmetrical positions in the decoder.

[0148] Specifically, the functions of each module in the Mamba-based serpentine convolution motion deblurring device provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further description will be given in the embodiment of the present invention.

[0149] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor (Processor) 110, a communication interface (Communications Interface) 120, a memory (Memory) 130 and a communication bus 140, wherein the processor 110, the communication interface 120, and the memory 130 communicate with each other through the communication bus 140. The processor 110 may call the logic instructions in the memory 130 to execute the Mamba-based serpentine convolution motion deblurring method provided in the above embodiments.

[0150] In addition, the logic instructions in the above-mentioned memory 130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0151] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the Mamba-based serpentine convolution motion deblurring method provided in the above embodiments.

[0152] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the Mamba-based serpentine convolution motion deblurring method provided in the above embodiments.

[0153] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A snake-like convolution motion deblurring method based on Mamba, characterized in that: include: Get the image to be processed; Applying an image deblurring network to deblur the image to be processed to obtain a deblurred image of the image to be processed; The image deblurring network comprises an encoder and a decoder connected in sequence, the encoder comprises an input convolution layer and a plurality of cascaded serpentine processing modules, and the decoder comprises a plurality of cascaded serpentine processing modules and an output convolution layer; Each serpentine processing module includes a serpentine state space module and a serpentine channel attention module connected in sequence. The serpentine state space module is used to combine the Mamba framework with the serpentine convolution, divide the input equally along the channel, and fuse them respectively after the serpentine convolution operation to obtain a first fusion result, and the first fusion result is reduced in the number of channels and then superimposed with the input to obtain an output; the serpentine channel attention module is used to divide the input equally along the channel, and fuse them after the serpentine convolution operation to obtain a second fusion result, and the second fusion result is filtered for channel information based on the channel attention mechanism to obtain a fuzzy residual feature, and the fuzzy residual feature is superimposed with the input to obtain an output; The serpentine state space module includes: a first normalization layer, a channel division layer, a first serpentine branch, a second serpentine branch and a fusion module; The first normalization layer is used to perform layer normalization processing on the input to obtain the output; The channel division layer is used to divide the output of the first normalization layer into equal parts along the channel to obtain a first equal division result and a second equal division result; The first serpentine branch is used to perform two serpentine convolution operations on the first equally divided result in sequence and then activate the result to obtain a first activation result, and obtain an output based on the first activation result; The second snake-shaped branch is used to perform a snake-shaped convolution operation on the second equally divided result and then activate it to obtain an output; The fusion module is used to fuse the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result, and reduce the number of channels of the first fusion result and superimpose it with the input of the serpentine state space module to obtain the output.

2. The Mamba-based serpentine convolution motion deblurring method according to claim 1, characterized in that: The fusion module includes a first product layer, a first convolution layer and a first superposition layer; The first product layer is used to perform Hadamard product on the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result; The first convolution layer is used to perform a convolution operation on the first fusion result to obtain a result with a reduced number of channels; The first superposition layer is used to superimpose the channel number reduction result with the input of the serpentine state space module to obtain the output of the serpentine state space module.

3. The Mamba-based serpentine convolution motion deblurring method according to claim 1, characterized in that: The first serpentine branch includes a direction scanning module; The direction scanning module is used to scan the first activation result along multiple directions, and based on the state space equation, capture the long-range dependency in the scanning result to obtain the output of the first serpentine branch.

4. The Mamba-based serpentine convolution motion deblurring method according to claim 3, characterized in that: The direction scanning module is specifically used for: Flattening the first activation result into a one-dimensional sequence, and scanning the one-dimensional sequence along multiple directions to obtain the scanning result; the scanning result includes multiple scanning sequences; Based on the state space equation, the long-range dependency of the multiple scanning sequences is captured, the multiple scanning sequences are merged, and the two-dimensional structure is restored through a reshaping operation to obtain the output of the first serpentine branch.

5. The Mamba-based serpentine convolution motion deblurring method according to any one of claims 1 to 4, characterized in that: The serpentine channel attention module includes a channel attention layer, and the channel attention layer includes a global pooling layer, a feature reshaping layer, a second convolutional layer, and a second product layer; The global pooling layer is used to perform global average pooling on each channel of the second fusion result to obtain a global spatial feature of each channel; The feature reshaping layer is used to reshape the global spatial features of each channel into a one-dimensional vector; The second convolution layer is used to perform a convolution operation on the one-dimensional vector of each channel to obtain the weight of each channel; The second product layer is used to multiply the global spatial feature of each channel by the weight to obtain the fuzzy residual feature.

6. The Mamba-based serpentine convolution motion deblurring method according to any one of claims 1 to 4, characterized in that: The serpentine channel attention modules in each serpentine processing module of the encoder are respectively connected to the serpentine state space modules in the serpentine processing modules at symmetrical positions in the decoder.

7. A Mamba-based serpentine convolution motion deblurring device, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; A deblurring module, used for applying an image deblurring network to deblur the image to be processed to obtain a deblurred image of the image to be processed; The image deblurring network comprises an encoder and a decoder connected in sequence, the encoder comprises an input convolution layer and a plurality of cascaded serpentine processing modules, and the decoder comprises a plurality of cascaded serpentine processing modules and an output convolution layer; Each serpentine processing module includes a serpentine state space module and a serpentine channel attention module connected in sequence. The serpentine state space module is used to combine the Mamba framework with the serpentine convolution, divide the input equally along the channel, and fuse them respectively after the serpentine convolution operation to obtain a first fusion result, and the first fusion result is reduced in the number of channels and then superimposed with the input to obtain an output; the serpentine channel attention module is used to divide the input equally along the channel, and fuse them after the serpentine convolution operation to obtain a second fusion result, and the second fusion result is filtered for channel information based on the channel attention mechanism to obtain a fuzzy residual feature, and the fuzzy residual feature is superimposed with the input to obtain an output; The serpentine state space module includes: a first normalization layer, a channel division layer, a first serpentine branch, a second serpentine branch and a fusion module; The first normalization layer is used to perform layer normalization processing on the input to obtain the output; The channel division layer is used to divide the output of the first normalization layer into equal parts along the channel to obtain a first equal division result and a second equal division result; The first serpentine branch is used to perform two serpentine convolution operations on the first equally divided result in sequence and then activate the result to obtain a first activation result, and obtain an output based on the first activation result; The second snake-shaped branch is used to perform a snake-shaped convolution operation on the second equally divided result and then activate it to obtain an output; The fusion module is used to fuse the outputs of the first serpentine branch and the second serpentine branch to obtain the first fusion result, and reduce the number of channels of the first fusion result and superimpose it with the input of the serpentine state space module to obtain the output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the Mamba-based serpentine convolution motion deblurring method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the Mamba-based serpentine convolution motion deblurring method according to any one of claims 1 to 6 is implemented.

Images