An image restoration method and device based on discrete wavelet transform

Abstract

This invention relates to the field of image restoration technology, specifically disclosing an image restoration method and device based on discrete wavelet transform. An image restoration network, called Multi-Wavelet Attention Memory Neural Network (MWA-MNN), is designed, consisting of three sub-networks. These sub-networks include Discrete Wavelet Attention Blocks (DWABs) and Coordinate Channel Attention Blocks (CCABs) to extract spatial and channel feature information, effectively separating degraded image layers such as rain streaks from the image to obtain a clean image. The invention leverages the expressive power of wavelet transform for high-frequency information, thereby improving image restoration performance. The sub-networks also include Supervised Selective Kernel Blocks (SSKBs) for filtering effectively extracted features in each sub-network. This invention also proposes a memristor-based cross-array MWA-MNN circuit implementation scheme. Memristors are a novel type of non-volatile memory with faster read speeds and lower power consumption. Experimental results show that this method has good performance and versatility in various image restoration tasks.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to an image restoration method and device based on discrete wavelet transform. Background Technology

[0002] Computer vision offers effective solutions to many problems that previously relied on human visual processing, one of which is image restoration. Image restoration uses computer vision techniques to automatically remove defects and noise from images, resulting in higher-quality images. Image restoration can improve image sharpness, remove noise, restore missing parts, and make images more suitable for various computer vision tasks. In recent years, research on image restoration has received increasing attention, especially after the introduction of deep learning techniques, which has significantly improved the performance of image restoration algorithms. Deep learning-based image restoration methods use convolutional neural networks to learn the structure and features of images, automatically removing defects and noise. These methods have been applied to various practical applications, such as image demodulation, image dehazing, and low-light image enhancement. However, current image restoration methods still face some challenges. Using simple multi-scale coarse-fine networks to reduce image resolution leads to the loss of detailed features, making it difficult to improve the quality of image restoration. Furthermore, large-size filters result in too many parameters in the network. Many methods ignore the frequency information of the image, which is crucial for image denoising. In addition, due to computational complexity, most methods are difficult to deploy for image restoration on edge devices. Summary of the Invention

[0003] This invention provides an image restoration method and device based on discrete wavelet transform, and the technical problem it solves is: how to improve the image restoration performance.

[0004] To address the above technical problems, this invention provides an image restoration method based on discrete wavelet transform, comprising the following steps:

[0005] S1. Construct an image restoration network;

[0006] The image restoration network includes a first sub-network, a second sub-network, and a third sub-network. The first sub-network includes a first discrete wavelet attention block, a first coordinate channel attention block, and a first supervised selective kernel block connected in sequence. The second sub-network includes a second discrete wavelet attention block, a second coordinate channel attention block, and a second supervised selective kernel block connected in sequence. The third sub-network includes a third discrete wavelet attention block, a third coordinate channel attention block, a fourth coordinate channel attention block, and a fifth coordinate channel attention block connected in sequence.

[0007] The first degraded image, obtained by performing 2×2 segmentation patching on the original degraded image, is input to the first discrete wavelet attention block and the first supervised selective kernel block. The first discrete wavelet attention block performs frequency decomposition on the first degraded image to obtain multiple first frequency sub-bands, which are then input to the first coordinate channel attention block. The first coordinate channel attention block extracts positional spatial information and channel information from the multiple first frequency sub-bands to obtain a first spatial channel feature map, which is then input to the first supervised selective kernel block. The first supervised selective kernel block filters the first spatial channel feature map and combines it with the first degraded image to obtain a first output feature map.

[0008] The original degraded image is padded left and right to obtain a second degraded image, which is then input into the second discrete wavelet attention block and the second supervised selective kernel block. The second discrete wavelet attention block performs frequency decomposition on the second degraded image to obtain multiple second frequency sub-bands, which are then concatenated with the first output feature map and input into the second coordinate channel attention block. The second coordinate channel attention block extracts the spatial and channel information of the input image to obtain a second spatial channel feature map, which is then fed into the second supervised selective kernel block. The second supervised selective kernel block filters the second spatial channel feature map and combines it with the second degraded image to obtain a second output feature map.

[0009] The original degraded image is subjected to spatial feature extraction by the third discrete wavelet attention block, and then concatenated with the second output feature map before being input into the third coordinate channel attention block. After sequential channel feature extraction by the third coordinate channel attention block, the fourth coordinate channel attention block, and the fifth coordinate channel attention block, the third spatial channel feature map is obtained and added to the original degraded image to obtain the restored image.

[0010] S2. Train the image restoration network;

[0011] S3. Input the degraded image that needs to be repaired into the trained image restoration network, and output the restored image.

[0012] Furthermore, during the training process in step S2, the loss function of the image restoration network is designed as follows:

[0013] L mixture =αL MSE +(1-α)L SSIM

[0014] Among them, L mixture L represents the mixed loss. MSE L represents the mean squared error loss. SSIM The structural similarity index represents the loss, and α is used to balance L. MSE and LSSIM hyperparameters;

[0015] L MSE The calculation formula is as follows:

[0016]

[0017] Where C, H, and W represent the number of channels, height, and width of the restored or real image, respectively, and y c ,h, w Indicates the tag value. Indicates the predicted value;

[0018] L SSIM The calculation formula is as follows:

[0019]

[0020] Where, μ x μ y Let σ represent the image mean of the restored image and the original image, respectively. xy It is the covariance between the restored image and the real image, σ x σ y C1 and C2 represent the standard deviations of the restored image and the original image, respectively, and are different constants.

[0021] Furthermore, the first discrete wavelet attention block, the second discrete wavelet attention block, and the third discrete wavelet attention block all employ the same discrete wavelet attention block structure. The operation process of the discrete wavelet attention block is represented by the following function:

[0022] I out =Conv 1×1 (I in )+Conv 3×3 (Concat(I o ,(IWT(Attention(Concat(DWT(I w ))))))

[0023] Among them, I in Io ut These represent the input and output images of the discrete wavelet attention block, respectively. Indicates that I in Feature map obtained after 3×3 convolution The two parts obtained after separation are: DWT() (Discrete Wavelet Transform), Concat() (Concatenation), Attention() (Spatial and Channel Attention), and IWT() (Inverse Wavelet Transform). 1×1 () represents a 1×1 convolution operation, Conv3×3 () indicates a 3×3 convolution operation.

[0024] Furthermore, the third, fourth, and fifth coordinate channel attention blocks have the same structure, each including 12 sequentially connected coordinate channel attention blocks, one 3×3 convolutional layer, and one splicing layer. The splicing layer is used to splice the output of the 3×3 convolutional layer with the input of the first coordinate channel attention block. The first and second coordinate channel attention blocks each include five sequentially connected coordinate channel attention blocks.

[0025] The operation process of the coordinate channel attention block is represented by the following function:

[0026] X out =ChannelA(CoordinateA(DSConv 3×3 (GELU(DSConv 3×3 (LN(X in )))))+X in )

[0027] X in Xo ut These represent the input and output images of the coordinate channel attention block, respectively. LN() represents the layer normalization operation, and DSConv... 3×3 () indicates a depthwise separable 3×3 convolution operation, GELU() indicates the GELU activation function operation, CoordinateA() indicates the coordinate attention operation, and ChannelA() indicates the channel attention operation.

[0028] Furthermore, the first supervisory selective core block and the second supervisory selective core block adopt the same structure of supervisory selective core block, wherein the supervisory selective core block includes a supervisory block and a selective core block;

[0029] In the supervision module, input features Figure X in After passing through a 1×1 convolutional layer, it is then added to the degraded image to produce the restored image I. r Then, a supervised attention feature map is generated through a 1×1 convolutional layer and a sigmoid function;

[0030] In the selective kernel module, the input features are first... Figure X in Feature extraction was performed using four different kernel sizes to obtain X1, X2, X3, and X4; then X1, X2, X3, and X4 were added together to obtain new features. Figure X Then, perform global average pooling on X′ to obtain a 1×1 channel feature map. C′ represents the number of channels in S; then S is passed through a fully connected layer to generate a feature map. The value of d is calculated as follows:

[0031]

[0032] Where L represents the lower bound of dimensionality reduction, and r represents the ratio of dimensionality reduction;

[0033] Next, perform Softmax on the feature map Z along the channel dimension to select features from multiple scales; then multiply the selected features element-wise by the minimum value and sum the results element-wise.

[0034] Finally, the summed features are multiplied by the supervised attention feature map generated by the supervised module to obtain the output feature map.

[0035] Furthermore, in channel convolution, the three-channel image signals are converted into voltage signals and input row by row into the first memristor cross array, with bit lines controlling the sequential input of each 3×3 image signal;

[0036] In depthwise separable convolution, a C×N×N second memristor cross array is used to achieve depthwise convolution. Samples at the same position are input through multiple channels, and 1×1 pointwise convolution is performed using each row of the second memristor cross array.

[0037] Furthermore, a third memristor cross array is used to perform global average pooling calculation in Attention(), expressed by the formula:

[0038]

[0039] V avgC V represents the result of global average pooling. c,k M represents the k-th value on the c-th channel. C R represents the memristor value, and R represents the resistance value.

[0040] The present invention also provides an image restoration device based on discrete wavelet transform, applied to step S3 of the above method, the key feature of which is: it includes an image acquisition unit and an image restoration unit, wherein the image restoration unit is equipped with the trained image restoration network;

[0041] The image acquisition unit is used to acquire the degraded image that needs to be repaired, and the image restoration unit is used to restore the degraded image that needs to be repaired using the image restoration network and output the restored image.

[0042] This invention provides an image restoration method and device based on discrete wavelet transform, and designs an image restoration network called Multi-Wavelet Attention Memory Neural Network (MWA-MNN), which consists of three sub-networks. This invention introduces a multi-patch method for fine-to-coarse image restoration. For each fine level, a hardware-software co-operated image restoration method is proposed, contributing its residual image to the coarse level, allowing each level to focus on a different restoration scale. This invention designs a Discrete Wavelet Attention Block (DWAB) and a Coordinate Channel Attention Block (CCAB) to extract spatial and channel feature information, thereby effectively separating degraded image layers such as rain streaks from the image to obtain a clean image. The invention utilizes the expressive power of wavelet transform for high-frequency information, thereby improving image restoration performance. This invention also designs a Supervised Selective Kernel Block (SSKB) to filter effectively extracted features in each sub-network. This invention also proposes a memristor-based cross-array MWA-MNN circuit implementation scheme. Memristors are a novel type of non-volatile memory with faster read speeds and lower power consumption. Experimental results show that this method has good performance and versatility in various image restoration tasks. In summary:

[0043] The MWA-MNN proposed in this invention realizes multi-scale image restoration tasks, extracts global and local feature information, and has high effectiveness and strong versatility.

[0044] The DWAB and CCAB proposed in this invention are beneficial for enhancing the ability to extract contextual features and capture details, while the proposed SSKB is beneficial for reducing redundant and invalid features, thereby achieving the goal of selecting effective features.

[0045] This invention has conducted extensive experiments on image deduplication, image dehazing, and low-light image enhancement. The results show that the method achieves better results in image restoration tasks than many other methods.

[0046] This invention designs a hardware and software co-processing method based on memristors and a circuit implementation scheme, which can restore images faster and more effectively on edge devices. Attached Figure Description

[0047] Figure 1 This is an overall architecture diagram of MWA-MNN provided in the embodiments of the present invention;

[0048] Figure 2 This is an architecture diagram of the discrete wavelet attention block provided in an embodiment of the present invention;

[0049] Figure 3This is a schematic diagram illustrating the operation process of DWT and IWT provided in an embodiment of the present invention;

[0050] Figure 4 This is an architecture diagram of the coordinate channel attention block provided in an embodiment of the present invention;

[0051] Figure 5 This is an architecture diagram of the supervised selective kernel block provided in an embodiment of the present invention;

[0052] Figure 6 This is a diagram illustrating the hardware implementation of MWA-MNN and its deployment on edge devices provided in this embodiment of the invention.

[0053] Figure 7 This is a diagram illustrating the implementation of the MWA-MNN convolution operation based on a memristor cross array and its peripheral circuitry provided in an embodiment of the present invention.

[0054] Figure 8 This is a diagram of the memristor peripheral circuit in the MWA-MNN for edge deployment provided in an embodiment of the present invention;

[0055] Figure 9 This is a visual comparison result of image deraining provided in an embodiment of the present invention;

[0056] Figure 10 This is a visual comparison result diagram of low-light image enhancement provided in an embodiment of the present invention;

[0057] Figure 11 This is a visual comparison result of image dehazing provided in an embodiment of the present invention;

[0058] Figure 12 This is a visual comparison result diagram of various input conditions of MWA-MNN provided in the embodiments of the present invention;

[0059] Figure 13 This is a visual comparison result diagram of object detection before and after using the method provided by the present invention, provided in an embodiment of the present invention. Detailed Implementation

[0060] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0061] This invention provides an image restoration method based on discrete wavelet transform, such as... Figure 1 As shown, the steps include:

[0062] S1. Construct an image restoration network;

[0063] S2. Train the image restoration network;

[0064] S3. Input the degraded image that needs to be repaired into the trained image restoration network, and output the restored image.

[0065] The architecture of image restoration networks is as follows Figure 1 As shown, the network includes a first subnetwork (subnetwork 1), a second subnetwork (subnetwork 2), and a third subnetwork (subnetwork 3). The first subnetwork comprises a first discrete wavelet attention block, a first coordinate channel attention block, and a first supervised selective kernel block, all connected in sequence. The second subnetwork comprises a second discrete wavelet attention block, a second coordinate channel attention block, and a second supervised selective kernel block, all connected in sequence. The third subnetwork comprises a third discrete wavelet attention block, a third coordinate channel attention block, a fourth coordinate channel attention block, and a fifth coordinate channel attention block, all connected in sequence.

[0066] The first degraded image, obtained by 2×2 segmentation and patching of the original degraded image, is input to the first discrete wavelet attention block and the first supervised selective kernel block. The first discrete wavelet attention block performs frequency decomposition on the first degraded image to obtain multiple first frequency sub-bands, which are input to the first coordinate channel attention block. The first coordinate channel attention block extracts positional spatial information and channel information from the multiple first frequency sub-bands to obtain a first spatial channel feature map, which is then input to the first supervised selective kernel block. The first supervised selective kernel block is used to filter the first spatial channel feature map and combine it with the first degraded image to obtain a first output feature map.

[0067] The original degraded image is segmented into left and right parts to obtain a second degraded image, which is then input into a second discrete wavelet attention block and a second supervised selective kernel block. The second discrete wavelet attention block performs frequency decomposition on the second degraded image to obtain multiple second frequency sub-bands, which are then concatenated with the first output feature map and input into a second coordinate channel attention block. The second coordinate channel attention block extracts the spatial and channel information of the input image to obtain a second spatial channel feature map, which is then input into the second supervised selective kernel block. The second supervised selective kernel block filters the second spatial channel feature map and combines it with the second degraded image to obtain a second output feature map.

[0068] After spatial feature extraction by the third discrete wavelet attention block, the original degraded image is concatenated with the second output feature map and then input into the third coordinate channel attention block. After sequential channel feature extraction by the third coordinate channel attention block, the fourth coordinate channel attention block, and the fifth coordinate channel attention block, the third spatial channel feature map is obtained and added to the original degraded image to obtain the restored image.

[0069] from Figure 1 As can be seen, the first, second, and third discrete wavelet attention blocks all use the same discrete wavelet attention block structure. For example... Figure 2 As shown, the discrete wavelet attention block mainly includes Discrete Wavelet Transform (DWT), Inverse Wavelet Transform (IWT), 3×3 convolution, 1×1 convolution, Global Average Pooling (GAP), Global Max Pooling (GMP), and element-wise addition. element-wise multiplication splicing Processing units, etc. Figure 3 The diagram shows DWT and IWT, where ↑ and ↓ represent upsampling and downsampling, respectively.

[0070] Specifically, the operation of the discrete wavelet attention block can be represented by the following function:

[0071] I out =Conv 1×1 (I in )+Conv 3×3 (Concat(I o ,(IWT(Attention(Concat(DWT(I w ))))))

[0072] Among them, I in Io ut These represent the input and output images of the discrete wavelet attention block, respectively. H×W represents I in Height and width, C represents Io ut The number of channels, Indicates that I in Feature map obtained after 3×3 convolution The two parts obtained after separation are: DWT() (Discrete Wavelet Transform), Concat() (Concatenation), Attention() (Spatial and Channel Attention), and IWT() (Inverse Wavelet Transform). 1×1 () represents a 1×1 convolution operation, Conv 3×3 () indicates a 3×3 convolution operation. w The generation of Io can be expressed by the following formula:

[0073] I w Io = Split(Conv 3×3 (I in ))

[0074] Split() represents a splitting operation along the channel dimension.

[0075] Among them, for spatial and channel attention operations, Attention(), such as Figure 2 As shown, Concat(DWT(Iw Image I obtained d After 3×3 convolution, simple gating (SG), and 3×3 convolution, different attention extraction methods are applied. For spatial attention, the input image is first subjected to global average pooling (GAP) and global max pooling (GMP), then concatenated, followed by a 1×1 convolution with ReLU and Sigmoid functions, and finally multiplied with the input image. For channel attention, the input image is first subjected to global average pooling (GAP), then a 1×1 convolution, and multiplied with the input image. Finally, the two attention feature maps are concatenated and subjected to a 1×1 convolution, and then multiplied with image I. d Adding them together yields image I. d ′.

[0076] This invention performs channel and spatial attention extraction after DWT, and uses SimpleGate instead of ReLU as the activation function, and only uses global average pooling and 1×1 convolution to construct channel attention, thereby reducing the number of parameters while maintaining performance.

[0077] Wavelet transform can decompose a signal into subbands of different frequencies, providing contextual and textural information at multiple levels. By analyzing and processing these subbands, image processing operations such as denoising, compression, and edge detection can be performed.

[0078] like Figure 1 As shown, the third, fourth, and fifth coordinate channel attention blocks have the same structure, each including 12 sequentially connected coordinate channel attention blocks, one 3×3 convolutional layer, and one splicing layer. The splicing layer is used to splice the output of the 3×3 convolutional layer with the input of the first coordinate channel attention block. The first and second coordinate channel attention blocks each include five sequentially connected coordinate channel attention blocks.

[0079] like Figure 4 As shown, the coordinate channel attention block mainly includes operation units such as layer normalization, 3×3 convolution, 1×1 convolution, global average pooling (GAP), Gaussian error linear unit activation function (GELU), and Sigmoid activation function. The operation process of the coordinate channel attention block can be represented by the following function:

[0080] X out =ChannelA(CoordinateA(DSConv 3×3 (GELU(DSConv 3×3 (LN(X in )))))+X in )

[0081] X in Xo utThese represent the input and output images of the coordinate channel attention block, respectively. LN() represents the layer normalization operation, and DSConv... 3×3 () indicates a depthwise separable 3×3 convolution operation, GELU() indicates the GELU activation function operation, CoordinateA() indicates the coordinate attention operation, and ChannelA() indicates the channel attention operation.

[0082] Channel attention mechanisms have been widely used in computer vision and have proven to play a crucial role in improving model performance. However, they generally only focus on channel information while neglecting positional information, which is critical in image processing. To address this issue, this invention proposes a Coordinate Channel Attention Block (CCAB) that combines channel attention blocks and coordinate attention. By introducing coordinate attention blocks, long-term dependencies along one spatial direction can be captured while maintaining precise positional features along another spatial direction, thereby enhancing the representational power of learned features. Furthermore, the channel attention proposed in this invention can extract rich contextual feature information. This is achieved by normalizing the input feature map (denoted as...). This process is initiated using layer normalization to facilitate smooth training. Then, this invention utilizes 3×3 depthwise separable convolutions for feature extraction, significantly reducing network parameters and computational complexity while maintaining network effectiveness. Finally, GELU is used as the activation function to derive the feature maps. Then, a 3×3 depthwise separable convolution is performed. Next, spatial and channel information are extracted using coordinate and channel attention respectively, yielding the residual feature X. r Finally, X r With X in Add them together to obtain the output features.

[0083] like Figure 1 As shown, the first and second supervised selective kernel blocks employ the same supervised selective kernel block structure. Figure 5 As shown, the supervised selective kernel block mainly consists of two parts: the supervised kernel and the selective kernel.

[0084] In the supervision section, feature map The image is passed through a 1×1 convolutional layer and then added to the degraded input image to produce a restored image. It is used to calculate the loss using real images. Then, a supervised attention feature map is generated using a 1×1 algorithm and a sigmoid function.

[0085] In the selective kernel section, feature maps Feature extraction was performed using four different kernel sizes (1, 3, 5, 7) to obtain X1, X2, X3, and X4. Then, X1, X2, X3, and X4 were summed to obtain a new feature map. Then, perform global average pooling on X′ to obtain a 1×1 channel feature map. C′ represents the number of channels in S. To capture more efficient features, S undergoes a fully connected layer (PC) to reduce its dimensionality, resulting in compact features. The value of d is calculated as follows:

[0086]

[0087] Where L represents the lower bound of dimensionality reduction, and r represents the ratio of dimensionality reduction. In the specific implementation, r is set to 16, and L is set to 32.

[0088] During the selection phase, the attention vector is softmaxed along the channel dimension to select features from multiple scales. Then, the selected features are element-wise multiplied by the minimum value, and the results are summed element-wise. Finally, the features obtained above are multiplied by the supervised attention map to derive the output feature map.

[0089] Mean Squared Error (MSE) is widely used in image restoration. However, MSE only considers the sum of squared differences and minimizes the mean squared error between the predicted and ground truth values, which easily ignores noise in the data, leading to model overfitting. Therefore, in some cases, it may lead to average error, thus ignoring important individual differences. To address these issues, this invention proposes a hybrid loss function that combines MSE and the Structural Similarity Index (SSIM) measure, which can evaluate not only the pixel-level differences between the ground truth and the restored image but also their structural similarity. MSE can be expressed as:

[0090]

[0091] Among them, y c ,h, w Indicates the tag value. This represents the predicted value.

[0092] SSIM can be expressed as the following equation:

[0093]

[0094] Where, μ x μ y Let σ represent the image mean of the restored image and the original image, respectively. xy It is the covariance between the restored image and the real image, σ x σ yLet C1 and C2 represent the standard deviations of the restored image and the original image, respectively. C1 and C2 are different constants (constants added to prevent instability when the denominator is close to zero). Therefore, the SSIM loss can be defined as:

[0095] L SSIM =1-SSIM(f(x) in ),x gt )

[0096] f(x in ) represents the restored image output by the image restoration network, x gt Represents a real image.

[0097] The final formula for the hybrid loss function is as follows:

[0098] L mixture =αL MSE +(1-α)L SSIM

[0099] Here, α is a hyperparameter used to balance the two types of losses mentioned above, and in specific implementations, it is set to 0.2.

[0100] In outdoor environments, surveillance devices such as cameras, drones, and mobile phones are used for object detection and other advanced vision tasks, inevitably capturing low-quality images due to harsh weather conditions. Deploying image restoration algorithms on hardware can improve image quality and accuracy. However, neural networks typically have complex structures and require significant computational resources, especially for tasks such as image restoration involving processing large amounts of data and parameters. Therefore, deploying image restoration algorithms on typical edge devices is challenging. To overcome this problem, this invention proposes a hardware-friendly MWA-MNN and its edge implementation scheme. Figure 6 This paper illustrates the hardware implementation of the MWA-MNN on an edge device proposed in this invention, which employs a memristor crossbar array. Based on a memristor neuromorphic computing system, MWA-MNN enables image restoration and in-memory computation on the edge device, making inference on the edge device more efficient and faster while reducing storage and computing resource costs. This improves the overall system performance and efficiency and provides a promising software and hardware implementation solution for applications such as autonomous driving, robotics, and road monitoring. In practical applications, the edge device preprocesses the acquired image data and inputs it into the memristor neural computing system. The memristor neural computing system, through the cooperation of peripheral circuits and a controller, performs memristor-based in-memory computation to obtain the restored image.

[0101] Based on Kirchhoff's laws and Ohm's law, memristors can achieve efficient memory computation without additional data communication overhead. Therefore, this invention maps the weights of depthwise separable convolution kernels and channel convolution kernels onto a memristor cross array to achieve more efficient convolution operations. In channel convolution, the three-channel image signals are converted into voltage signals and input row-by-row into the first memristor cross array, using bit lines (BL) to control the sequential input of each 3×3 image signal. In depthwise separable convolution, a C×N×N second memristor cross array (N represents the depth convolution kernel size) is used to implement depthwise convolution. By inputting samples at the same location through multiple channels, 1×1 pointwise convolution is performed using each row of the second memristor cross array. Specifically, the output current and output voltage of the two columns of memristor cross arrays are converted into digital signals by an IV converter for further processing. The following equation illustrates the complete voltage conversion process:

[0102]

[0103]

[0104] in, It is the positive and negative input current of the memristor, V m It is the conversion voltage, V j The output voltage corresponds to the output of two memristors, where i represents the dimension of the input signal, j represents the dimension of the output signal, and W... n,m It represents the weights of each pair of memristors, where n and m represent the nth row and mth column of the memristor cross array, and V n This represents the input voltage in the nth row, and R represents the resistance value.

[0105] Therefore, each pair of memristors is used to form weights. Additionally, due to the inconsistency in the row voltage dimensions input to the memristor crossbar array during depthwise separable convolution, data for dimensionality transformation between depthwise and pointwise convolutions must be temporarily stored. Figure 7 This paper presents an implementation of MWA-MNN convolution operations based on a memristor cross array and its peripheral circuitry. Utilizing a memristor cross array for convolution operations effectively accelerates image restoration and reduces circuit complexity. Furthermore, the MWA-MNN implementation based on a memristor-based neural computing system reduces the difficulty of implementing image restoration tasks on edge devices due to its lightweight circuit structure. It significantly improves the efficiency of limited hardware resources and minimizes the substantial memory overhead associated with implementing multiply-accumulate operations in hardware.

[0106] Furthermore, the hardware implementation details are in Figure 8As shown, these are the main units for intermediate voltage processing, compatible with memristor cross arrays. Since global average pooling calculates the average eigenvalue of each channel of the feature map, this invention employs a parallel memristor array to perform calculations for multiple channels, such as... Figure 8 As shown in (d), the global average pooling circuit can be represented as follows:

[0107]

[0108] V avgC V represents the result of global average pooling. c,k M represents the k-th value on the c-th channel. C This represents the memristor value. The size of each pooling window for the channel is k = H × W, therefore the value of each memristor is M. gap = k×R. For global max pooling, this invention uses two MOS transistors to construct a maximum voltage selector. Based on this selector, this invention achieves global max pooling, as shown below. Figure 8 As shown in (f).

[0109] Furthermore, this invention employs a diode to implement the ReLU activation function. The implementations of Sigmoid and Softmax are as follows: Figure 8 As shown in (b) and (e). The outputs of these two circuits are as follows:

[0110]

[0111]

[0112] In the formula, V bias Indicates the bias voltage, V d This represents the input voltage of the sigmoid function, V. s1 Exp(·) represents the input voltage of Softmax. Figure 8 The exponential circuit shown in (a) is an example.

[0113] The Hadamard product is used to perform computations between the feature map and the attention matrix. The Hadamard circuit implementation is built upon the integration of multiple sets of parallel multipliers and operational amplifiers, such as... Figure 8 As shown in (c), each set of multipliers and operational amplifiers performs multiplication at corresponding positions between the two matrices. The entire network layer of the model, including convolution, pooling, and activation augmentation, can be implemented in hardware using these processing units and memristor cross-switch arrays. These memristor circuits are the main components for implementing MWA-MNN computations on edge devices. Furthermore, image restoration tasks on edge devices can be coupled with… Figure 6 The controllers shown cooperate to achieve this.

[0114] Based on the above-mentioned hardware and software integration, such as Figure 6 , Figure 7 , Figure 8 As shown, this invention also provides an image restoration device based on discrete wavelet transform, including an image acquisition unit and an image restoration unit, wherein the image restoration unit is equipped with a trained image restoration network. The image acquisition unit is used to acquire the degraded image to be restored, and the image restoration unit is used to restore the degraded image to be restored using the image restoration network, outputting the restored image. Specifically, the corresponding hardware deployment has been described in the method description and will not be repeated here.

[0115] In summary, the image restoration method and device based on discrete wavelet transform provided by this invention have designed an image restoration network called Multi-Wavelet Attention Memory Neural Network (MWA-MNN), which consists of three sub-networks. This invention introduces a multi-patch method for fine-to-coarse image restoration. For each fine-level, a hardware-software co-processing image restoration method is proposed, contributing its residual image to the coarse level, allowing each level to focus on a different restoration scale. This invention designs a Discrete Wavelet Attention Block (DWAB) and a Coordinate Channel Attention Block (CCAB) to extract spatial and channel feature information, thereby effectively separating degraded image layers such as rain streaks from the image to obtain a clean image. The invention utilizes the expressive power of wavelet transform for high-frequency information, thereby improving image restoration performance. This invention also designs a Supervised Selective Kernel Block (SSKB) to filter the effectively extracted features in each sub-network. This invention also proposes a scheme based on a memristor cross-array MWA-MNN circuit. Memristors are a novel type of non-volatile memory, offering faster read speeds and lower power consumption. In summary:

[0116] The MWA-MNN proposed in this invention realizes multi-scale image restoration tasks, extracts global and local feature information, and has high effectiveness and strong versatility.

[0117] The DWAB and CCAB proposed in this invention are beneficial for enhancing the ability to extract contextual features and capture details, while the proposed SSKB is beneficial for reducing redundant and invalid features, thereby achieving the goal of selecting effective features.

[0118] This invention designs a hardware and software co-processing method based on memristors and a circuit implementation scheme, which can restore images faster and more effectively on edge devices.

[0119] The effects of the present invention will now be verified.

[0120] This invention conducted extensive experiments on various publicly available benchmark datasets to evaluate the performance of the proposed method. Specifically, the proposed method for image deduplication was evaluated on three datasets: Rain200L, Rain200H, and Rain1200. For image dehazing, the model was trained on the RESIDE dataset and evaluated on the SOTS (Indoor) dataset. For the Rain200L and Rain200H datasets, the input image resolutions were 481×321 pixels and 321×481 pixels, respectively. To verify the effectiveness of the proposed method in low-light image enhancement, experiments were conducted on the LOL dataset, and the trained model was directly tested on the DICM, LIME, MEF, and NPE datasets. For the Rain1200, LOL, and RESIDE datasets, the input image resolutions were 512×512 pixels, 600×400 pixels, and 620×460 pixels, respectively. In the DICM, LIME, MEF, and NPE datasets, the image size was not fixed. Detailed information on the datasets is shown in Table 1. This invention trains and tests each dataset separately for the corresponding task.

[0121] Table 1. Dataset information for image deraining, dehazing, and low-light image enhancement.

[0122]

[0123] To evaluate the quality of the restored images in image deraining, dehazing, and low-light image enhancement using paired datasets, this invention uses Peak Signal-to-Noise Ratio (PSNR) and SSIM as quantitative metrics for comparison. For no-reference datasets, this invention uses the Natural Image Quality Evaluator (NIQE) for quantitative comparison. Higher PSNR and SSIM values ​​indicate better image restoration quality, while lower NIQE values ​​indicate better image restoration quality.

[0124] The proposed method is end-to-end and requires no pre-training. The setup is as follows: In subnetworks 1 and 2, the number of CCABs for each encoder and decoder is set to 5; in subnetwork 3, each part contains 12 CCABs, for a total of three parts. The number of channels for image deraining and dehazing is set to 40, and the number of channels for low-light image enhancement is set to 48. The entire network is trained on the PyTorch framework using an NVIDIA GeForce RTX 3090 GPU and the Adam optimizer. The training batch size is 32, and the patch size is 64×64. The initial learning rate is set to 1×10⁻⁶. -3 Multiply by 0.3 every 10 epochs, for a total of 60 epochs of training.

[0125] This invention also compares the proposed method with model-based GMMs and several data-driven methods (including RESCAN, PReNet, JORDER-E, ReHEN, MSPFN, MPRNet, RCDNet, RLNe, and CDPLG). The comparison results are shown in Table 2, demonstrating that the method of this invention achieves the highest PSNR and SSIM values, proving its superior effectiveness. Compared to the second-highest rated method, the method of this invention shows an average improvement of 0.65 dB in PSNR and an average improvement of 0.011 in SSIM.

[0126] Table 2 Quantitative comparison of image denoising on three benchmark datasets

[0127]

[0128] Note: Best results are indicated in bold, and second-best results are indicated in underline.

[0129] To further illustrate the performance of the method of the present invention, a visual comparison was performed. For example... Figure 9 As shown, compared to the actual ground conditions, the model-based method GMM does not completely eliminate rain streaks, which still significantly affect visibility. Other methods perform well in eliminating rain streaks, but inevitably encounter problems such as artifacts, color distortion, and loss of detail. In contrast, the method proposed in this invention effectively avoids these problems and achieves the best performance in rain removal.

[0130] In low-light image enhancement experiments, this invention compared quantitative metrics with state-of-the-art (SOTA) methods (including RetinexNet, MBLLEN, EnlightenGAN, DBP, HDRNet, DRBN, DeepUPE, and NDATnet), and the results are shown in Table 3. For the LOL dataset, this invention used the same metrics PSNR and SSIM as in the image deraining experiments, while for other datasets, this invention used NIQE as the quantitative comparison metric because they are no-reference datasets. Table 3 shows that the method of this invention achieved the best results in each aspect. On the LOL dataset, compared with the second-best result, the method of this invention improved PSNR and SSIM by 0.2 dB and 0.021, respectively. For the four real-world datasets, the method of this invention reduced the NIQE value by an average of 0.3636, demonstrating the significant effectiveness of the proposed method on real images.

[0131] Table 3. Quantitative comparison of low-light image enhancement on five benchmark datasets.

[0132]

[0133] Note: Best results are indicated in bold, and second-best results are indicated in underline.

[0134] Similarly, this invention conducted a visual comparison of low-light image enhancement experiments, and the comparison results are as follows: Figure 10 As shown. Figure 10 The results show that RetinexNet and MBLLEN suffer from insufficient brightness and noise, while DBP and DeepUPE improve upon the brightness issue but still retain some noise. The method of this invention effectively solves the overall low brightness problem and better handles texture details, which is closer to reality.

[0135] Furthermore, this invention also conducted image dehazing experiments, and the results are shown in Table 4. Several state-of-the-art (SOTA) methods were selected for quantitative comparison, including DCP, DehazeNet, AOD-Net, GMAN, EPDN, AUEND, and D4. The results in Table 4 show that, compared to the second-highest performing method, the PSNR and SSIM performance of this invention are 3.79 dB and 0.021 higher, respectively, demonstrating its effectiveness in image dehazing tasks.

[0136] Table 4 Quantitative comparison of image dehazing on the SOTS (Indoor) dataset

[0137]

[0138] Note: Best results are shown in bold, and second-best results are shown in underline.

[0139] Visual comparison of image dehazing, such as Figure 11 As shown. Compared to the real image, the four methods in the first row all have some residual haze; EPDN, AUEDN, and D4 all lose detail and produce color distortion. In contrast, the method of this invention eliminates the haze and preserves detail to the maximum extent, which is visually similar to reality.

[0140] To verify the effectiveness of the blocks designed in this invention for image restoration tasks, ablation experiments were conducted. This invention divides the network into blocks and designs four control groups, each trained and tested separately. The entire experiment was performed on the Rain200L dataset. The four control groups are as follows:

[0141] V1: CCAB is used as the baseline, excluding DWAB and SSKB.

[0142] V2: DWAB baseline.

[0143] V3: Based on SSKB.

[0144] •V4: The DWAB and SSKB baselines proposed in this paper.

[0145] The results of the ablation experiment are shown in Table 5.

[0146] Table 5 Results of the ablation experiment

[0147] piece <![CDATA[V1]]> <![CDATA[V2]]> <![CDATA[V3]]> <![CDATA[V4]]> CCAB √ √ √ √ DWAB √ √ SSKB √ √ PSNR 38.98 39.12 39.05 39.22 SSIM 0.980 0.986 0.983 0.987

[0148] Table 5 shows that on Rain200L, method V4 of the present invention significantly improves both PSNR and SSIM compared to the other three control groups. Specifically, compared to V1, V2, and V3, V4 improves PSNR by 0.24 dB, 0.10 dB, and 0.17 dB, respectively, and SSIM by 0.007, 0.001, and 0.004, respectively. The results in Table 5 demonstrate that the block proposed in this invention can effectively improve the effectiveness of image restoration. Furthermore, the MWA-MNN designed in this invention exhibits excellent performance and can better handle the complexity and diversity of image restoration tasks.

[0149] Furthermore, to verify the impact of the number of CCABs on the performance of the proposed model, this invention conducted experiments on the effects of using different numbers of CCABs in the network. This invention sets the number of CCABs to N, where N∈3,4,5,6. Experiments were conducted using the Rain200L dataset under various CCAB configurations, and quantitative comparisons were performed using PSNR and SSIM metrics. The experimental results are shown in Table 6.

[0150] Table 6. Effects of different CCAB quantities on experiments.

[0151] Number of CCABs N＝3 N＝4 N = 5 (default value) N＝6 PSNR↑ 39.11 39.17 39.22 39.19 SSIM↑ 0.9858 0.9865 0.9872 0.9868

[0152] As can be seen from the experimental results in Table 6, the model achieved the best performance on the Rain200L dataset when the number of CCABs was 5. Therefore, the number of CCABs is set to 5 by default in this invention.

[0153] Due to the complexity of outdoor environments, input images often suffer from problems such as blurriness and low resolution. Therefore, this invention tested its model under different input conditions and compared its performance. This invention selected several images, introduced Gaussian blur, and reduced the image resolution, such as... Figure 12 As shown, the top row displays the rainy image, while the bottom row displays the image restored by the model of this invention. The results clearly demonstrate that the model of this invention can effectively eliminate rain streaks in the image and produce minimal artifacts under different input conditions. This proves the powerful performance and excellent generalization ability of the model of this invention under various input conditions.

[0154] Image restoration tasks serve as preprocessing for higher-level vision tasks, such as object detection and image classification. The goal of this invention is to validate the effectiveness of the proposed method in practical applications. This invention uses YOLO v5 to test object detection on images processed by the method of this invention, such as... Figure 13 As shown. Figure 13 The top row of images showed image quality degradation due to rain streaks, making it impossible to identify any objects in the images. However, after processing by the model of this invention, these images successfully identified objects such as chairs, vases, and televisions. Figure 13 The middle row shows the dehazed image, which correctly detects more objects and has a significantly increased confidence score for object detection. Figure 13 The bottom row shows the enhanced image under low-light conditions, displaying results similar to the middle row. These experiments demonstrate that the method proposed in this invention can significantly improve the accuracy of images captured in adverse weather conditions when applied to advanced vision tasks. Therefore, the method of this invention has considerable practical value.

[0155] In summary, this invention proposes a hardware-software co-processing image restoration method based on memristors, named MWA-MNN. It consists of three sub-networks, mainly including DWAB, CCAB, and SSKB blocks, used to extract and select rich contextual information. This invention employs a multi-block strategy to extract detailed and texture information from multi-scale images. Furthermore, this invention proposes a circuit implementation scheme for MWA-MNN based on a memristor cross array, providing a feasible method for deploying image restoration algorithms on edge devices, achieving hardware acceleration and real-time image restoration. Finally, this invention conducts quantitative experiments on multiple datasets and compares MWA-MNN with more than 20 state-of-the-art (SOTA) methods. Experimental results show that MWA-MNN exhibits excellent performance and good versatility in various image restoration tasks.

[0156] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An image restoration method based on discrete wavelet transform, characterized in that, Including the following steps: S1. Construct an image restoration network; The image restoration network includes a first sub-network, a second sub-network, and a third sub-network. The first sub-network includes a first discrete wavelet attention block, a first coordinate channel attention block, and a first supervised selective kernel block connected in sequence. The second sub-network includes a second discrete wavelet attention block, a second coordinate channel attention block, and a second supervised selective kernel block connected in sequence. The third sub-network includes a third discrete wavelet attention block, a third coordinate channel attention block, a fourth coordinate channel attention block, and a fifth coordinate channel attention block connected in sequence. The first degraded image, obtained by performing 2×2 segmentation patching on the original degraded image, is input to the first discrete wavelet attention block and the first supervised selective kernel block. The first discrete wavelet attention block performs frequency decomposition on the first degraded image to obtain multiple first frequency sub-bands, which are then input to the first coordinate channel attention block. The first coordinate channel attention block extracts positional spatial information and channel information from the multiple first frequency sub-bands to obtain a first spatial channel feature map, which is then input to the first supervised selective kernel block. The first supervised selective kernel block filters the first spatial channel feature map and combines it with the first degraded image to obtain a first output feature map. The original degraded image is padded left and right to obtain a second degraded image, which is then input into the second discrete wavelet attention block and the second supervised selective kernel block. The second discrete wavelet attention block performs frequency decomposition on the second degraded image to obtain multiple second frequency sub-bands, which are then concatenated with the first output feature map and input into the second coordinate channel attention block. The second coordinate channel attention block extracts the spatial and channel information of the input image to obtain a second spatial channel feature map, which is then fed into the second supervised selective kernel block. The second supervised selective kernel block filters the second spatial channel feature map and combines it with the second degraded image to obtain a second output feature map. The original degraded image is subjected to spatial feature extraction by the third discrete wavelet attention block, and then concatenated with the second output feature map before being input into the third coordinate channel attention block. After sequential channel feature extraction by the third coordinate channel attention block, the fourth coordinate channel attention block, and the fifth coordinate channel attention block, the third spatial channel feature map is obtained and added to the original degraded image to obtain the restored image. S2. Train the image restoration network; S3. Input the degraded image that needs to be repaired into the trained image restoration network, and output the restored image.

2. The image restoration method based on discrete wavelet transform according to claim 1, characterized in that, During the training process in step S2, the loss function of the image restoration network is designed as follows: ， in, Indicates mixed loss, This represents the mean squared error loss. The structural similarity index measures the loss. It is used for balance and hyperparameters; The calculation formula is as follows: ， in, This represents the number of channels, height, and width of the restored or original image. Indicates the tag value. Indicates the predicted value; The calculation formula is as follows: ， in, , Let represent the image mean of the restored image and the original image, respectively. It is the covariance between the restored image and the real image. , These represent the standard deviations of the restored image and the original image, respectively. , These are different constants.

3. The image restoration method based on discrete wavelet transform according to claim 2, characterized in that: The first discrete wavelet attention block, the second discrete wavelet attention block, and the third discrete wavelet attention block all use the same discrete wavelet attention block structure. The operation process of the discrete wavelet attention block is represented by the following function: ， in, These represent the input and output images of the discrete wavelet attention block, respectively. , Indicates will conduct Feature map obtained after convolution The two parts obtained by separation This represents the discrete wavelet transform operation. This indicates a splicing operation. Represents spatial and channel attention operations. This indicates the inverse wavelet transform operation. express Convolution operation, express Convolution operation.

4. The image restoration method based on discrete wavelet transform according to claim 3, characterized in that: The third, fourth, and fifth coordinate channel attention blocks have the same structure, each including 12 sequentially connected coordinate channel attention blocks and 1... A convolutional layer and a stitching layer, the stitching layer being used to... The output of the convolutional layer is concatenated with the input of the first coordinate channel attention block; the first coordinate channel attention block and the second coordinate channel attention block each comprise five coordinate channel attention blocks connected in sequence. The operation process of the coordinate channel attention block is represented by the following function: ， , These represent the input and output images of the coordinate channel attention block, respectively. Presentation layer normalization operation, Indicates depth separability Convolution operation, This indicates the activation operation of the GELU activation function. Pay attention to the operation when representing coordinates. This indicates that you should pay attention to the operation of the channel.

5. The image restoration method based on discrete wavelet transform according to claim 3, characterized in that: The first supervisory selective core block and the second supervisory selective core block adopt the same structure of supervisory selective core block, and the supervisory selective core block includes a supervisory block and a selective core block; In the supervision module, input feature map After passing through a 1×1 convolutional layer, it is then added to the degraded image to produce a restored image. Then, a supervised attention feature map is generated through a 1×1 convolutional layer and a sigmoid function; In the selective kernel module, the input feature map is first... Feature extraction was performed using four different kernel sizes to obtain... Then Adding them together yields a new feature map. Then Perform global average pooling to obtain Channel feature map , This represents the number of channels in S; then... A feature map is generated by passing through a fully connected layer. The value of d is calculated as follows: ， in, This represents the lower bound of dimensionality reduction. Indicates the ratio of dimensionality reduction; Then the feature map Softmax is performed along the channel dimension to select features from multiple scales; Then multiply the selected features element-wise by the minimum value, and sum the results element-wise. Finally, the summed features are multiplied by the supervised attention feature map generated by the supervised module to obtain the output feature map. .

6. The image restoration method based on discrete wavelet transform according to claim 5, characterized in that: In channel convolution, the three-channel image signal is converted into a voltage signal and input row by row into the first memristor cross array, with bit lines controlling the sequential input of each signal. Image signal; In depthwise separable convolution, the following is employed: The second memristor cross array implements depthwise convolution, using samples at the same location as input from multiple channels, and utilizing each row of the second memristor cross array for... Pointwise convolution.

7. The image restoration method based on discrete wavelet transform according to claim 6, characterized in that, Execution using a third memristor cross array The global average pooling calculation in [the context of the problem] is expressed by the formula: ， This represents the result of global average pooling. This represents the k-th value on the c-th channel. Indicates the memristor value. This indicates the resistance value.

8. An image restoration device based on discrete wavelet transform, applied to step S3 as described in any one of claims 1 to 7, characterized in that: It includes an image acquisition unit and an image restoration unit, wherein the image restoration unit is equipped with the trained image restoration network; The image acquisition unit is used to acquire the degraded image that needs to be repaired, and the image restoration unit is used to restore the degraded image that needs to be repaired using the image restoration network and output the restored image.

Images