An image deraining method based on multi-scale state space model

Abstract

Embodiments of the present invention provide an image deraining method based on a multiscale state-space model, belonging to the fields of computer vision and image processing. The method comprises: acquiring an image with rain streaks and decomposing the image into a multiscale pyramid image through downsampling; inputting the rain streak image and the multiscale pyramid image into a pre-constructed multiscale state-space model, performing feature extraction and image reconstruction through convolutional layers and an encoder-decoder network, and obtaining a reconstructed residual image; and adding the reconstructed residual image to the rain streak image to obtain the target derained image. The state-space model is used to model global information, while a multiscale framework is combined to effectively utilize multiscale complementary information, explicitly mining cross-scale complementary features, and more accurately removing rain streaks. This method gradually restores a high-quality, clear image from the rain-stained image, ensuring the clarity and visual quality of the derained image.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image deraining method based on a multi-scale state space model. Background Art

[0002] Rain, a common weather phenomenon, not only significantly reduces image visibility but can also obscure critical visual details, posing a significant challenge to downstream vision tasks in applications such as autonomous driving and video surveillance. Image deraining aims to remove undesirable rain-induced degradation from input images, thereby improving their visual quality and enhancing the accuracy of perception systems. Therefore, effective image deraining research not only improves image visual quality but also holds significant research significance.

[0003] Numerous methods have been proposed for image deraining. Early methods based on prior knowledge typically relied on model-driven strategies, leveraging prior information about clean images or specific features of rain streaks to guide the restoration process. Due to the ill-posed nature of image deraining, traditional approaches typically rely on image decomposition, low-rank representations, discriminative sparse coding, and Gaussian mixture models. However, these methods often rely on empirical observations and lead to complex optimization problems.

[0004] Subsequently, the rise of deep learning has greatly promoted the development of image deraining. Convolutional neural network-based methods excel at capturing the complex mapping relationship between rainy and clear images, enabling them to effectively handle rain streaks of varying shapes, sizes, and densities. However, the fixed receptive field of convolutional neural networks limits their ability to capture global features and multi-scale information.

[0005] Zamir et al. designed an efficient Transformer model by estimating self-attention along the channel dimension and achieved remarkable performance. Xiao et al. first introduced a Transformer for image deraining based on spatial and window self-attention modules. Chen et al. developed a sparse Transformer to fully utilize the most useful features to achieve better image restoration. However, the self-attention mechanism introduces quadratic spatial and temporal complexity, which significantly increases the computational cost when processing high-resolution images, which is unacceptable. Therefore, it is necessary to explore a method that does not significantly increase the computational cost while effectively capturing non-local information to achieve high-quality image deraining. Summary of the Invention

[0006] The purpose of the embodiment of the present invention is to provide an image deraining method based on a multi-scale state space model.

[0007] To achieve the above objectives, an embodiment of the present invention provides an image deraining method based on a multi-scale state space model, comprising:

[0008] Acquire an image with rain streaks, and decompose the image with rain streaks into a multi-scale pyramid image by downsampling;

[0009] The image with rain streaks and the multi-scale pyramid image are respectively input into a pre-built multi-scale state space model, and feature extraction and image reconstruction are performed through a convolutional layer and an encoder-decoder network to obtain a reconstructed residual image, wherein the encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module;

[0010] The reconstructed residual image is added to the image with rain streaks to obtain a target rain-free image.

[0011] Optionally, the multi-scale Mamba block is used to combine the multi-scale 2D selective scanning mechanism to apply geometric transformation operations to the multi-scale pyramid image to generate scanning sequences in different directions to represent global feature information;

[0012] The frequency feature enhancement module is used to map the input features to the frequency domain through Fourier transform, separate the real part and the imaginary part and splice them into channel dimensions, extract the frequency domain features through convolution and nonlinear activation, and restore them to the spatial domain through inverse Fourier transform to achieve deep feature extraction.

[0013] Optionally, the multi-scale 2D selective scanning mechanism is expressed according to the following formula:

[0014]

[0015] Where k represents the scale, k = 1 represents a small scale, k = 2 represents a medium scale, k = 4 represents a large scale, X represents the input image, Stack() represents the stacking operation of the image, T() represents the transposition operation of the image, F() represents the pixel flipping operation, and Cat() represents the image splicing.

[0016] Optionally, the encoder-decoder network further includes a gated feature fusion module for adaptively aggregating complementary features of multi-scale branches.

[0017] Optionally, the image with rain streaks and the multi-scale pyramid image are respectively input into a pre-built multi-scale state space model, and feature extraction and image reconstruction are performed through a convolutional layer and an encoder-decoder network to obtain a reconstructed residual image, including:

[0018] Inputting the image with rain streaks and the multi-scale pyramid image into the convolution layer of a pre-built multi-scale state space model respectively to extract shallow features;

[0019] The shallow features are input into the multi-scale Mamba block in the encoder-decoder network of each scale branch, and two parallel branches are processed. The features of the two branches are aggregated and output through the Hadamard product. In the first branch, the feature channel is processed by depth convolution and SiLU activation function, and combined with a multi-scale 2D selective scanning strategy and layer normalization operation. In the second branch, layer normalization and SiLU activation operations are performed.

[0020] Aggregate features at different scales through the gated feature fusion module to obtain aggregated features;

[0021] The aggregated features are input into the decoder network for image reconstruction to obtain a reconstructed residual image; wherein the multi-scale state space model network architecture includes three scale branches, each scale branch includes an encoder and a decoder.

[0022] Optionally, features at different scales are aggregated through a gated feature fusion module, and the aggregated features are obtained according to the following formula:

[0023] ;

[0024] ;

[0025] Where, Input features for different scales, express activation function, represents a 3×3 convolution, represents 1×1 convolution, ⊙ represents element-wise multiplication, represents the input features, represents the features after aggregation, Represents a gate control unit.

[0026] Optionally, the loss function used by the multi-scale state-space model is as follows:

[0027]

[0028] Where, represents the Charbonnier loss, represents the edge loss, represents the frequency loss, 、 and Represent weights respectively.

[0029] In a second aspect, the present invention provides an image rain removal system based on a multi-scale state space model, comprising:

[0030] an acquisition unit, configured to acquire an image with rain streaks, and decompose the image with rain streaks into a multi-scale pyramid image by downsampling;

[0031] a reconstruction unit, configured to input the image with rain streaks and the multi-scale pyramid image into a pre-built multi-scale state space model, perform feature extraction and image reconstruction through a convolutional layer and an encoder-decoder network, and obtain a reconstructed residual image, wherein the encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module;

[0032] The obtaining unit is configured to add the reconstructed residual image to the image with rain streaks to obtain a target derained image.

[0033] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned image deraining method based on the multi-scale state space model are implemented.

[0034] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned image deraining method based on the multi-scale state space model.

[0035] Through the above technical solution, the state space model is used to model global information. At the same time, the multi-scale framework is combined to effectively utilize multi-scale complementary information, explicitly mine cross-scale complementary features, and more accurately remove rain streaks. High-quality clear images are gradually restored from rainy images, ensuring the clarity and visual quality of the derained images.

[0036] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings:

[0038] Figure 1 This is a flowchart of an implementation of an image deraining method based on a multi-scale state space model provided by an embodiment of the present invention;

[0039] Figure 2Schematic diagram of a multi-scale 2D selective scanning strategy provided by an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of a frequency feature enhancement module and a gated feature fusion module provided by an embodiment of the present invention;

[0041] Figure 4 Schematic diagram of the overall network structure of an image deraining method based on a multi-scale state space model provided by an embodiment of the present invention;

[0042] Figure 5 1 is a schematic structural diagram of an image rain removal system based on a multi-scale state space model provided by an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following describes the specific implementation of the embodiment of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present invention and is not used to limit the embodiment of the present invention.

[0045] See Figure 1 FIG. 1 is a flowchart of an implementation of an image deraining method based on a multi-scale state space model provided by an embodiment of the present invention, including the following steps:

[0046] Step 100: Acquire an image with rain streaks, and decompose the image with rain streaks into multi-scale pyramid images by downsampling.

[0047] Specifically, the input image is down-sampled to generate three scale branches of the pyramid scale, namely the large scale, i.e., the original image, the middle scale, i.e., the 1 / 2 sampling branch, and the small scale, i.e., the 1 / 4 sampling branch.

[0048] Step 101: The image with rain streaks and the multi-scale pyramid image are respectively input into a pre-built multi-scale state space model, and feature extraction and image reconstruction are performed through a convolutional layer and an encoder-decoder network to obtain a reconstructed residual image.

[0049] The encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module.

[0050] Specifically, the multi-scale Mamba block is used to combine the multi-scale 2D selective scanning mechanism to perform geometric transformation operations on the multi-scale pyramid image, generating scanning sequences in different directions to represent global feature information.

[0051] Furthermore, the multi-scale Mamba block models global features through a state-space model, as follows: The state-space model is a mathematical framework commonly used in time series analysis and control systems. The state equation describes the evolution of the underlying system over time and characterizes the relationship between the system's hidden state and its time dynamics. The input signal x(t) ∈ R is represented by the implicit latent state Mapped to the output response y(t) ∈ R. It can be expressed as a set of first-order linear ordinary differential equations as follows:

[0052] h′(t) = Ah(t) + Bx(t)

[0053] y(t) = Ch(t) + Dx(t)

[0054] Where N represents the state size, and A, B, C, and D are learnable weight matrices.

[0055] Subsequently, the above equations are usually integrated into the actual deep learning algorithm by using a discretization process. Specifically, let Δ be the time scale parameter used to convert the continuous parameters A and B into discrete parameters. , The commonly used discretization method is the zero-order hold rule, which is expressed as follows:

[0056]

[0057]

[0058]

[0059] However, the above formulation is primarily for linear time-invariant systems whose parameters remain constant under varying inputs. To overcome this limitation, Mamba improves the state-space module and proposes a selective scanning mechanism (S6). This mechanism improves the original state-space module by introducing specific restoration priors to achieve both input-dependent weights and linear computational complexity.

[0060] Specifically, local patch repetitiveness and inter-channel interactions are taken into account to assist long-range spatial modeling in Mamba. Given the input deep features , where H and W represent the height and width of the image, C represents the channel, first layer normalization (LN) is used, and then visual state space block (VSSB) is used to capture spatial long-range dependencies. In order to improve network performance, a learnable scaling factor is used in the residual connection , the specific expression is:

[0061]

[0062] Then, another normalization layer (LN) is used to Normalization is performed and a convolutional layer is used to model the spatial local similarity prior. Then a forward propagation layer (FFL) is used to obtain the final output of VSSB. , which can be expressed as:

[0063]

[0064] For example, in a multi-scale network architecture, different scale branches contain different rain degradation features. Compared with small-scale branches, large-scale branches contain richer feature information. If the same scanning strategy is used for branches of different scales, it may lead to information redundancy and waste of computing resources. Figure 2 As shown in , the multi-scale Mamba block adopts a multi-scale 2D selective scanning strategy to better extract the potential explicit information in different scales. Figure 2 As shown in Figure 2, this module assigns different numbers of scanning directions to branches of different scales through geometric transformation, thereby achieving efficient sequential scanning. Specifically, larger scales use more scanning directions (four scanning directions), medium scales are assigned one scanning direction, and smaller scales use fewer scanning directions. The multi-scale 2D selective scanning mechanism is expressed according to the following formula:

[0065]

[0066] Where k represents the scale, k = 1 represents a small scale, k = 2 represents a medium scale, k = 4 represents a large scale, X represents the input image, Stack() represents the stacking operation of the image, T() represents the transposition operation of the image, F() represents the pixel flipping operation, and Cat() represents the image splicing.

[0067] Specifically, the frequency feature enhancement module is used to map the input features to the frequency domain through Fourier transform, separate the real part and the imaginary part and splice them into channel dimensions, extract the frequency domain features through convolution and nonlinear activation, and restore them to the spatial domain through inverse Fourier transform to achieve deep feature extraction.

[0068] Furthermore, the frequency feature enhancement module is as follows: Figure 3 As shown, the frequency feature enhancement module is specifically: input It is divided into two branches. One branch extracts local features in the spatial domain through time domain convolution operation, while the other branch uses two-dimensional real fast Fourier transform to map the input features to the frequency domain, obtaining F(X)∈R H ×W / 2×CSubsequently, the real and imaginary features of F(X) are concatenated in the channel dimension to obtain Y∈RH× W / 2 ×2C, and then the processed features are restored to the time domain through inverse Fourier transform to obtain the output Finally, through the residual connection, the spatial domain features, frequency domain features and the original input are added to obtain the final output.

[0069] Preferably, the encoder-decoder network further comprises a gated feature fusion module for adaptively aggregating complementary features of multi-scale branches.

[0070] Specifically, when executing step 101, the following steps may be specifically performed:

[0071] S1010: Inputting the image with rain streaks and the multi-scale pyramid image into the convolution layer of a pre-built multi-scale state space model respectively to extract shallow features.

[0072] S1011: Input the shallow features into the multi-scale Mamba block in the encoder-decoder network of each scale branch, perform two parallel branch processing, and aggregate and output the features of the two branches through the Hadamard product.

[0073] Among them, in the first branch, the feature channel undergoes depth convolution and SiLU activation function processing, and is combined with a multi-scale 2D selective scanning strategy and layer normalization operation. In the second branch, layer normalization and SiLU activation operations are performed.

[0074] In this embodiment, the multi-scale Mamba block is specifically implemented as follows: Input feature The processing is done in two parallel branches. In the first branch, the feature channels are processed with depthwise convolution and SiLU activation functions, combined with a multi-scale 2D selective scanning strategy and layer normalization. In the second branch, layer normalization and SiLU activation are performed. Afterwards, the features of the two branches are aggregated using the Hadamard product. The final output is , the specific process can be expressed as:

[0075]

[0076] Where DWConv represents depthwise separable convolution and ⊙ represents Hadamard product.

[0077] S1012: Aggregate features at different scales through a gated feature fusion module to obtain aggregated features.

[0078] For details, see the gated feature fusion module. Figure 3As shown in Figure 1, the gated feature fusion module is designed to better promote the fusion of features at different scales. For input features from three scales, a dual-input gating unit is used, which can dynamically adapt to inputs at different scales. The formula of the gating unit can be expressed as:

[0079] ;

[0080] in, represents a 3×3 convolution, Represents 1×1 convolution, Tanh() represents the tanh activation function, and ⊙ represents element-wise multiplication. By combining these two gating units, different features can be effectively fused. The process can be expressed as:

[0081] ;

[0082] Where, represents the input features, represents the features after aggregation, Represents a gate control unit.

[0083] Furthermore, the aggregated features of small and medium scales are spliced ​​to the previous scale through the decoder network composed of multi-scale Mamba blocks and frequency feature enhancement modules, and finally fused into the large-scale network for decoding. Each scale is then subjected to 3×3 convolution to obtain the reconstructed residual image. .

[0084] S1013: Input the aggregated features into the decoder network for image reconstruction to obtain a reconstructed residual image.

[0085] The multi-scale state-space model network architecture comprises three scale branches, each containing an encoder and a decoder. Each branch utilizes a multi-scale Mamba module combined with a multi-scale 2D selective scanning strategy to capture global feature information. A frequency feature enhancement module then extracts local feature information in the frequency domain, further facilitating image restoration. Furthermore, a gated feature fusion module adaptively aggregates complementary features across scales, further enhancing high-quality image restoration and achieving efficient deraining performance.

[0086] More specifically, during the model training process, in order to supervise the network training process and enhance its ability to capture details, a weighted sum of three losses is selected as the basis for guiding network training. The loss function used in the multi-scale state space model is as follows:

[0087]

[0088] Where, represents the Charbonnier loss, represents the edge loss, represents the frequency loss, 、 and Represent weights respectively.

[0089] For example, scalar weight 、 and They are empirically set to 1, 0.05, and 0.01, respectively.

[0090] Step 102: Add the reconstructed residual image to the image with rain streaks to obtain a target rain-free image.

[0091] Furthermore, the reconstructed residual image is added to the original rainy image to obtain the final rain-free image: .

[0092] In this embodiment, the overall network structure diagram of the image deraining method based on the multi-scale state space model is shown in FIG. Figure 4 As shown, the multi-scale state space model network architecture contains three scale branches, each branch contains an encoder and a decoder. For a given rainy day input image ,First, the coarse-to-fine technique is used to divide it into three scales, that is, the input image is interpolated Downsample to 1 / 2 and 1 / 4 scale to generate pyramid-shaped multi-scale rainy day images. These images are used as inputs of each scale and shallow features are extracted through 3×3 convolutional layers. , where H×W represents the spatial dimension and C is the number of feature channels. Next, the shallow features are input into the encoder-decoder network of each scale branch. The encoder / decoder of each branch consists of a multi-scale Mamba block and a frequency feature enhancement module. The multi-scale Mamba block combines the multi-scale 2D selective scanning mechanism to capture global feature information with linear computational complexity, while the frequency feature enhancement module focuses on local feature information through the frequency domain to achieve deep feature extraction. In addition, in this network, a gated feature fusion module is introduced to aggregate feature selection information between different scales and representations of a specific scale, and input it into the decoder network to reconstruct a high-quality output image to obtain a reconstructed residual image. Finally, the reconstructed residual image is added to the original rainy image to obtain the final derained image. .

[0093] Furthermore, to verify the effectiveness of this application, the model's deraining performance was evaluated on common image deraining benchmark datasets, such as Rain200H. On these datasets, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the Y channel in the YCbCr color space were calculated as benchmark evaluation metrics for quantitative comparison.

[0094] In this embodiment, the peak signal-to-noise ratio (PSNR) is used to measure the difference between two images, such as the difference between a compressed image and the original image, to evaluate the quality of the compressed image; and the difference between a restored image and the ground truth, to evaluate the performance of the restoration algorithm. Its calculation formula is:

[0095]

[0096] Where MSE is the mean square error of the two images pixel by pixel.

[0097] In this embodiment, the structural similarity index (SSIM) is based on the perceptual model of the human visual system and is used to measure the similarity between two images in terms of brightness, contrast, and structure. Unlike PSNR, SSIM is closer to the perception of the human visual system and can more accurately reflect image quality. The formula for SSIM can be expressed as:

[0098]

[0099] in, and are the average values ​​of image x and y in the local window, indicating the brightness level of the image, and are the variances of images x and y in the local window, respectively, indicating the contrast of the image, is the covariance of images x and y in the local window, indicating the structural similarity of the images, , , usually K1=0.01, K2=0.03, L is the dynamic range of pixel values.

[0100] The technical effects achieved by this application are as follows:

[0101] 1. Image deraining is performed using a deep learning algorithm based on a multi-scale state-space model. This model models global information and, combined with a multi-scale framework, effectively utilizes multi-scale complementary information. This method explicitly mines cross-scale complementary features, more accurately removing rain streaks and gradually restoring high-quality, clear images from rainy images, ensuring the clarity and visual quality of derained images.

[0102] 2. By combining a multi-scale 2D selective scanning strategy with a state-space model, the computational complexity is significantly reduced compared to the Transformer model approach used in existing technologies. While maintaining high performance, it is more suitable for deployment on resource-constrained mobile or embedded devices.

[0103] See Figure 5 FIG. 1 is a schematic diagram of the structure of an image rain removal system based on a multi-scale state space model provided by an embodiment of the present invention, including:

[0104] an acquisition unit, configured to acquire an image with rain streaks, and decompose the image with rain streaks into a multi-scale pyramid image by downsampling;

[0105] a reconstruction unit, configured to input the image with rain streaks and the multi-scale pyramid image into a pre-built multi-scale state space model, perform feature extraction and image reconstruction through a convolutional layer and an encoder-decoder network, and obtain a reconstructed residual image, wherein the encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module;

[0106] The obtaining unit is configured to add the reconstructed residual image to the image with rain streaks to obtain a target derained image.

[0107] Figure 6 It is a schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0108] The product price version management method for public cloud scenarios provided in the embodiments of the present application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. In the embodiments of the present invention, electronic devices include but are not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0109] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a button, a camera, a display, and a SIM card interface, etc.

[0110] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0112] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0113] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0114] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of an electronic device. The external memory card communicates with the processor through the external memory interface, enabling data storage. For example, files such as music and videos can be stored on the external memory card.

[0115] Internal memory can be used to store computer-executable program code, which includes instructions. The processor executes the instructions stored in the internal memory to perform various functional applications and data processing of the electronic device. The internal memory can include a program storage area and a data storage area. The internal memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0116] The wireless communication function of an electronic device can be implemented through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0117] Wireless communication modules can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0118] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0119] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0120] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0121] A GPU is a microprocessor for image processing that connects the display screen to the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0122] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0123] The storage medium provided in this application stores a program product that can implement a product price version management method for public cloud scenarios.

[0124] An image deraining method based on a multi-scale state-space model includes: obtaining an image with rain streaks and decomposing the image with rain streaks into a multi-scale pyramid image through downsampling; inputting the image with rain streaks and the multi-scale pyramid image into a pre-built multi-scale state-space model respectively, performing feature extraction and image reconstruction through a convolutional layer and an encoder-decoder network to obtain a reconstructed residual image, wherein the encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module; and adding the reconstructed residual image to the image with rain streaks to obtain a target derained image.

[0125] In some possible implementations, the subject matter of the present disclosure, namely, a method and system for managing product price and version for public cloud scenarios, can be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of this specification.

[0126] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0127] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image deraining method based on a multi-scale state space model, characterized in that: include: Acquire an image with rain streaks, and decompose the image with rain streaks into a multi-scale pyramid image by downsampling; The image with rain streaks and the multi-scale pyramid image are respectively input into a pre-built multi-scale state space model, and feature extraction and image reconstruction are performed through a convolutional layer and an encoder-decoder network to obtain a reconstructed residual image, wherein the encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module; The multi-scale Mamba block is used to combine the multi-scale 2D selective scanning mechanism to perform geometric transformation operations on the multi-scale pyramid image to generate scanning sequences in different directions to represent global feature information; The frequency feature enhancement module is used to map the input features to the frequency domain through Fourier transform, separate the real and imaginary parts and splice them into channel dimensions, extract the frequency domain features through convolution and nonlinear activation, and restore them to the spatial domain through inverse Fourier transform to achieve deep feature extraction; The multi-scale 2D selective scanning mechanism is expressed according to the following formula: Where k represents the scale, k = 1 represents a small scale, k = 2 represents a medium scale, k = 4 represents a large scale, X represents the input image, Stack() represents the stacking operation of the image, T() represents the transposition operation of the image, F() represents the pixel flipping operation, and Cat() represents the image splicing; The encoder-decoder network also includes a gated feature fusion module for adaptively aggregating complementary features of multi-scale branches; The reconstructed residual image is added to the image with rain streaks to obtain a target rain-free image.

2. The image deraining method based on the multi-scale state space model according to claim 1, characterized in that: The image with rain streaks and the multi-scale pyramid image are respectively input into a pre-built multi-scale state space model. After feature extraction and image reconstruction through the convolution layer and the encoder-decoder network, a reconstructed residual image is obtained, including: Inputting the image with rain streaks and the multi-scale pyramid image into the convolution layer of a pre-built multi-scale state space model respectively to extract shallow features; The shallow features are input into the multi-scale Mamba block in the encoder-decoder network of each scale branch, and two parallel branches are processed. The features of the two branches are aggregated and output through the Hadamard product. In the first branch, the feature channel is processed by depth convolution and SiLU activation function, and combined with a multi-scale 2D selective scanning strategy and layer normalization operation. In the second branch, layer normalization and SiLU activation operations are performed. Aggregate features at different scales through the gated feature fusion module to obtain aggregated features; The aggregated features are input into the decoder network for image reconstruction to obtain a reconstructed residual image; wherein the network architecture of the multi-scale state space model includes three scale branches, each scale branch includes an encoder and a decoder.

3. The image deraining method based on the multi-scale state space model according to claim 2, characterized in that: The features of different scales are aggregated through the gated feature fusion module, and the aggregated features are obtained according to the following formula: ； ； Where, Input features for different scales, express activation function, represents a 3×3 convolution, represents 1×1 convolution, ⊙ represents element-wise multiplication, represents the input features, represents the features after aggregation, Represents a gate control unit.

4. The image deraining method based on a multi-scale state space model according to claim 1, characterized in that: The loss function used in the multi-scale state-space model is as follows: Where, represents the Charbonnier loss, represents the edge loss, represents the frequency loss, 、 and Represent weights respectively.

5. An image deraining system based on a multi-scale state space model, characterized in that: The system is used to implement the image deraining method based on the multi-scale state space model as described in any one of claims 1 to 4; The system includes: an acquisition unit, configured to acquire an image with rain streaks, and decompose the image with rain streaks into a multi-scale pyramid image by downsampling; a reconstruction unit, configured to input the image with rain streaks and the multi-scale pyramid image into a pre-built multi-scale state space model, perform feature extraction and image reconstruction through a convolutional layer and an encoder-decoder network, and obtain a reconstructed residual image, wherein the encoder-decoder network includes a multi-scale Mamba block and a frequency feature enhancement module; The obtaining unit is configured to add the reconstructed residual image to the image with rain streaks to obtain a target derained image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the image deraining method based on the multi-scale state space model are implemented as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image deraining method based on a multi-scale state space model are implemented.

Images