An image restoration method based on brain-like visual Transformer and ADAS edge device

Abstract

The present invention relates to the field of image restoration technology, and specifically discloses an image restoration method based on brain-like visual Transformer and an ADAS edge device, and creates a U-shaped image restoration network (LSFormer). In the network, a linear sparse Transformer block (LSTB) is proposed, and a focused linear sparse attention block (FLSA) is designed in the LSTB to reduce the computational complexity, which helps to alleviate the outlier problem associated with Softmax; a multi-scale feedforward network (MSFF) is also designed in the FLSA to enhance the local representation by using different depth convolution kernel sizes; a wavelet attention block (WAB) is also proposed in the LSFormer to improve the output image. In addition, the present invention provides a hardware implementation scheme based on a memristor-based brain-like computing system for the application of ADAS in intelligent vehicles. The present invention also evaluates the performance of the proposed network on multiple data sets, including low-light enhancement, image deraining, and image defogging tasks, and the results show that LSFormer outperforms the most advanced networks in all three tasks.

Description

Technical Field

[0001] The present invention relates to the field of image restoration technology, and in particular to an image restoration method based on brain-like vision Transformer and an ADAS edge device. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, existing advanced driver assistance systems (ADAS) can significantly improve driving safety, convenience and efficiency. ADAS relies on environmental perception modules to capture real-time images or other types of surrounding vehicles and road traffic information. However, outdoor driving is often exposed to various extreme weather conditions, such as rain, snow, and low-light night driving. These factors limit the camera's field of view and reduce the system's visual perception capabilities, resulting in inaccurate ADAS judgments and possible traffic accidents.

[0003] Image inpainting networks have the potential to restore degraded images captured by cameras into clear images, enabling the system to obtain accurate traffic environment information. Image inpainting networks have received widespread attention and have achieved significant improvements in restoration results. Most advanced image inpainting networks are mainly based on the Transformer architecture. However, these image inpainting networks often have difficulty in striking a balance between computational efficiency and restoration performance. At the same time, the outlier problem caused by Softmax makes it challenging to quantify the effectiveness of deploying image inpainting networks on edge devices. In standard Vision Transformers (ViTs), the small weights in self-attention also lead to feature aggregation, generate unnecessary features as noise, and interfere with the image restoration process. In addition, existing image inpainting networks are mainly designed for a single task and cannot adapt to different weather conditions.

[0004] Traditional ADAS is deployed on devices based on CMOS technology, and the communication between various modules (i.e. memory, processor) is achieved through the traditional von Neumann architecture. However, with the obsolescence of Moore's Law and the emergence of the memory wall problem, a computing system with large memory capacity, fast processing power, and complex data computing capabilities is essential for ADAS. But since most Transformer networks have a large number of parameters, it is difficult to apply them to devices such as automotive cameras. Summary of the invention

[0005] The present invention provides an image restoration method based on brain-like vision Transformer and an ADAS edge device, and solves the technical problem of how to provide an image restoration network with excellent performance that can adapt to different weather conditions, and how to deploy the image restoration network in a lightweight manner in an ADAS system.

[0006] In order to solve the above technical problems, the present invention provides an image restoration method based on brain-like visual Transformer, comprising the steps of:

[0007] S1. Build an image restoration network based on brain-like visual Transformer;

[0008] The image restoration network is a U-shaped network structure, which is sequentially provided with a first 3×3 convolutional layer, a first wavelet attention block, Q linear sparse Transformer blocks, a second wavelet attention block, and a second 3×3 convolutional layer from input to output, where Q is an odd number not less than 2;

[0009] The degraded image is input into the first 3×3 convolutional layer for convolution and then input into the first wavelet attention block for attention extraction. The extraction result is input into Q linear sparse Transformer blocks for downsampling and upsampling. The sampling result is input into the second wavelet attention block for attention extraction. The extraction result is input into the second 3×3 convolutional layer for convolution. The convolution result is added to the degraded image to obtain a restored image.

[0010] The entire process of the linear sparse Transformer block is expressed by the following equation:

[0011] F′ i =FLSA(LN(F i-1 ))+F i-1

[0012] F i = MsFF(LN(F′ i ))+F′ i

[0013] Among them, F′ i 、F i-1 denote the output and input of the i-1th linear sparse Transformer block, respectively, and F′ i represents the intermediate feature map, LN() represents the layer normalization operation, FLSA() represents the focused linear sparse attention extraction operation, and MsFF() represents the multi-scale feedforward operation;

[0014] S2, training the image restoration network;

[0015] S3. Input the degraded image to be restored into the trained image restoration network, and output the restored image.

[0016] Furthermore, FLSA() is expressed as:

[0017] FLSAtt(Q,K,V)=Focuse(Q)Mask(S,k)+DWConv(V)

[0018] Where Q, K, V represent the query, key, and value in Transformer, FLSAtt(Q, K, V) represents focused linear sparse attention extraction, Focus() represents the focusing function, S = Focus(K T )V represents the score, Mask() represents the mask function, k represents the adjustable parameter in the top-k operation, and DWConv() represents the depth convolution;

[0019] The focus function Focus() is expressed as:

[0020]

[0021] Among them, x refers to the independent variable, ReLU() represents the ReLU function, and x **p Represents x raised element-wise to the power of p.

[0022] Furthermore, MsFF() is expressed as:

[0023] X out =Conv 1×1 [X3,X5,X7]

[0024] Among them, X3, X5, and X7 represent the corresponding outputs of convolution of the input feature Xin using convolution kernels of sizes 3×3, 5×5, and 7×7, respectively. [] represents the connection operation. Conv 1×1 Represents 1×1 convolution;

[0025] X3, X5, and X7 are represented by:

[0026] X i =ReLU(DWConv i×i (Conv 1×1 (X in )))

[0027] Where i = 3, 5, 7, DWConv i×i Indicates the use of depthwise convolution with a convolution kernel size of i×i.

[0028] Furthermore, the first wavelet attention block and the second wavelet attention block use the same wavelet attention block, and the process of the wavelet attention block is expressed by the formula:

[0029] I out =Conv 3×3 (IDWT([I s ,I c ]+I d ))+I in

[0030] Among them, I in represents the input features of the wavelet attention block, Conv 3×3 represents 3×3 convolution, IDWT() represents inverse discrete wavelet transform, I d ,I s ,I c All are intermediate features and have:

[0031] I d =DWT(Conv 3×3 (I in ))

[0032] I s =Sigmoid(Conv 1×1 ([GAP(I d ),GMP(I d )]))

[0033] I c =Conv 1×1 (GAP(I d ))

[0034] Among them, DWT() represents discrete wavelet transform, GMP() represents global maximum pooling, GAP() represents global average pooling, and Sigmoid() represents Sigmoid function.

[0035] Furthermore, during the training process of step S2, the loss function of the image restoration network is designed as:

[0036] L mixture =αL MSE +(1-α)L SSIM

[0037] Among them, L mixture represents the mixing loss, L MSE represents the mean square error loss, L SSIM represents the structural similarity index metric loss, and α is used to balance L MSE and L SSIM Hyperparameters of

[0038] L MSE The calculation formula is as follows:

[0039]

[0040] Among them, C, H, and W represent the height, width, and number of channels of the restored image or the real image, respectively. c,h,w Represents the label value, represents the predicted value;

[0041] L SSIM The calculation formula is as follows:

[0042]

[0043] Among them, μ x , μ y Represent the pixel averages of the restored image and the real image, σ xy represents the covariance between the restored image and the real image, σ x , σ y They represent the pixel standard deviations of the restored image and the real image respectively, and C1 and C2 are different constants.

[0044] Furthermore, Q=7, the 7 linear sparse Transformer blocks form a U-shaped structure, and the first 4 linear sparse Transformer blocks perform downsampling to transform the image size into H×W×C, The last three linear sparse Transformer blocks perform upsampling, transforming the image size into H×W×C.

[0045] The present invention also provides an image restoration ADAS edge device, the key of which is: deploying the trained image restoration network described in the above method, wherein the weights of the attention layer and the linear layer of the image restoration network are mapped to a first memristor cross array; converting the captured degraded image into parallel analog voltages, and inputting the parallel analog voltages into each row of the first memristor cross array to perform attention and forward propagation calculations.

[0046] Specifically, the depth convolution of the image inpainting network is implemented by a second memristor cross array, and in the depth direction, voltage rows from different channels are simultaneously input to corresponding rows of the second memristor cross array;

[0047] The 1×1 convolution of the image restoration network is implemented through a third memristor cross array, and voltages at the same position of different channels are input to the same row of the third memristor cross array.

[0048] The present invention provides an image restoration method based on brain-like visual Transformer and an ADAD edge device, which creates a U-shaped image restoration network (referred to as linear sparse Transformer, LSFormer for short) in the present invention. In the network, a linear sparse Transformer block (LSTB) is proposed, and a focused linear sparse attention block (FLSA) is designed in the linear sparse Transformer block to reduce computational complexity, which helps to alleviate the outlier problem associated with Softmax, and uses content sparsity to select the most relevant feature information, thereby improving the quality of the restored image through these effective features, and promoting the deployment of image restoration networks in intelligent assisted driving vehicles. The present invention also designs a multi-scale feedforward network (MSFF) in the linear sparse Transformer block to enhance local representation by using different depth convolution kernel sizes. The present invention also proposes a wavelet attention block (WAB) in LSFormer to extract shallow features of the input image and improve the output image based on discrete wavelet transform (DWT). In addition, the present invention provides an efficient and promising hardware implementation scheme based on a memristor-like brain computing system for the application of ADAS in intelligent vehicles. The paper also evaluates the performance of the proposed network on multiple datasets, including low-light enhancement, image deraining, and image dehazing tasks. The results show that LSFormer outperforms the state-of-the-art networks in all three tasks. In addition, the paper also conducts multiple metric tests on the proposed implementation, and the results show that the proposed solution is more efficient and stable compared to the existing Transformer circuit. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is an architecture diagram of an image restoration network provided by an embodiment of the present invention;

[0050] Figure 2 It is a brain-like implementation diagram of LSFormer provided by an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of a circuit for reshaping and Hadamard product provided by an embodiment of the present invention;

[0052] Figure 4 1 is a visual qualitative comparison diagram of the LSFormer provided by an embodiment of the present invention and other networks, wherein (a), (b), and (c) are comparison results of low illumination enhancement, image deraining, and image defogging, respectively. DETAILED DESCRIPTION

[0053] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0054] An image restoration method based on a brain-like visual Transformer provided by an embodiment of the present invention comprises the following steps:

[0055] S1. Build an image restoration network based on brain-like visual Transformer;

[0056] S2, training the image restoration network;

[0057] S3. Input the degraded image to be restored into the trained image restoration network and output the restored image.

[0058] The goal of the image restoration network is to restore damaged images under extreme weather conditions and restore them to clear images. This process ensures that ADAS can accurately obtain road traffic information, thereby ensuring the normal operation of the system. Image restoration mainly covers tasks such as low-light enhancement, rain removal, and fog removal.

[0059] The architecture of the image restoration network constructed by the present invention is as follows Figure 1 As shown, it is a U-shaped layered linear sparse Transformer (LSFormer), including a first 3×3 convolutional layer, a first wavelet attention block, Q linear sparse Transformer blocks, a second wavelet attention block, and a second 3×3 convolutional layer, where Q is an odd number not less than 2. In this embodiment, Q=7 is set, and 7 linear sparse Transformer blocks form a U-shaped structure. The first 4 linear sparse Transformer blocks perform downsampling, and the image size is transformed into H×W×C, The last three linear sparse Transformer blocks perform upsampling. The downsampled image at each level is concatenated with the upsampled image at the previous level and then input into the linear sparse Transformer block of this level to transform the image size into H×W×C.

[0060] The degraded image is input into the first 3×3 convolutional layer for convolution and then into the first wavelet attention block for attention extraction. The extraction result is input into Q linear sparse Transformer blocks for downsampling and upsampling. The sampling result is input into the second wavelet attention block for attention extraction. The extraction result is input into the second 3×3 convolutional layer for convolution. The convolution result is added to the degraded image to obtain the restored image.

[0061] LSFormer for degraded images 3×3 convolutions are performed to perform overlapping image patch embedding. In the backbone of the network, multi-scale LSTB is used to extract global and local information from the feature map. Specifically, inside LSTB, the focused linear sparse attention block (FLSA) is used to replace the Softmax attention. This replacement reduces the computational complexity of the Q, K, and V matrices while achieving superior performance than Softmax attention. Inside LSTB, a multi-scale feedforward network (MSFF) is designed to extract multi-scale local information from the feature map using convolution kernels of different sizes. Before and after the backbone network, a wavelet attention block (WAB) is designed to perform shallow feature extraction on the input feature map and refine the feature map before generating the final clear image. Finally, the 3×3 convolution generates a residual image Add it to the input corrupted image to get the final clean image I clean =I r +I degraded .

[0062] ①Linear Sparse Transformer Block (LSTB)

[0063] In ADAS, the image restoration network needs to process high-resolution images captured by the camera in real time, which contain rich road-related information. However, the standard ViT (Vision Transformers) is not suitable for image restoration due to its high computational complexity associated with Softmax self-attention. In addition, when the image restoration network is deployed to ADAS hardware devices, the limitations of Softmax attention may lead to outliers in the quantization deployment. This is because Softmax has a small dynamic range and tends to produce large outliers during the back propagation of the gradient signal. As the training process iterates, these outliers become more significant, and this phenomenon is more obvious in low-precision networks. Therefore, it is not appropriate to use a Transformer network based on Softmax attention. In order to address these challenges, the present invention designs LSTB, which simultaneously solves the computational complexity and outlier problems associated with Softmax attention. In addition, a sparse operation is introduced to focus the network's attention on more effective feature information related to image restoration. The structure of the entire LSTB is as follows Figure 1 Specifically, the input feature map of the i-1th LSTB is represented by F i-1 , i ≥ 2, firstly perform layer normalization and then input it into FLSA. The features obtained from FLSA are added to F i-1, followed by another layer normalization step. This feature is then fed into MsFF to extract multi-scale features. The whole process can be expressed by the following equation:

[0064] F′ i =FLSA(LN(F i-1 ))+F i-1

[0065] F i = MsFF(LN(F′ i ))+F′ i

[0066] Among them, F′ i 、F i-1 denote the output and input of the i-1th linear sparse Transformer block, respectively, and F′ i represents the intermediate feature map, LN() represents the layer normalization operation, FLSA() represents the focused linear sparse attention extraction operation, and MsFF() represents the multi-scale feedforward operation.

[0067] ② Focused Linear Sparse Attention Block (FLSA)

[0068] In the standard Transformer, by involving the query Q, key K and value (L represents the number of attention heads, d represents the dimension of each attention head) matrix operation and the application of Softmax to realize self-attention. The specific attention calculation can be described as follows:

[0069]

[0070] Where Q, K, and V represent query, key, and value matrices respectively, and softmax() represents the softmax function.

[0071] Subsequently, the output of each attention head is concatenated and input into a feed-forward neural network. In Softmax attention, Q and K are calculated. T The dot product between them requires a computational complexity of O(n 2 d), n represents the number of Softmax circuit modules. In DeiT, d is 64 and n is 196, while in Swin Transformer, d and n are set to 32 and 49 respectively. The present invention designs a focused linear sparse attention, such as Figure 1 As shown, first calculate K T V, and then multiply it by Q, thus reducing the computational complexity to O(nd 2). In order to enhance the performance of linear attention, a focusing function is applied to process the Q and K matrices respectively, making them more focused on features related to image restoration. In addition, the present invention also adds the results passed through V through deep convolution to improve the diversity of features. In terms of sparsification, the top-k method effectively balances the need to retain key information and remove the noise layer. Therefore, the present invention incorporates top-k to select more useful eigenvalues ​​and enhance the feature extraction capability of the module. Specifically, the present invention first inputs the feature map into three identical 1×1 convolutions and 3×3 deep convolutions to obtain Q, K, and V matrices. Then, the focusing function is used to process the reshaped Q and K matrices to calculate the pairwise similarity between the key and the query. The representation of the focusing function is shown in the following formula:

[0072]

[0073] Among them, x refers to the independent variable, ReLU() represents the ReLU function, and x **p Represents x raised element-wise to the power of p.

[0074] The obtained features are processed by top-k, the score S is calculated, and the k maximum values ​​are selected from each row of S, and their positions are recorded. The present invention uses a mask function to filter out the attention weights with low relevance to image restoration, retaining only meaningful feature values. The sparseness process is defined by the following formula:

[0075] S=Focus(K T )V

[0076]

[0077] Here Mask represents the mask function, k represents the adjustable parameter in the top-k operation, and t i represents the kth largest value in the i-th row, and i and j represent the position of the value in the matrix S.

[0078] The present invention multiplies the result of the sparsification with the Q matrix and adds it to the V matrix after the depth convolution. The entire FLSA process can be described by the following formula:

[0079] FLSAtt(Q,K,V)=Focuse(Q)Mask(S,k)+DWConv(V)

[0080] Finally, the results from multiple attention heads are concatenated and linearly projected through a 1×1 convolution to obtain the final output.

[0081] ③Multi-scale Feedforward Network (MSFF)

[0082] In the field of image restoration, the introduction of multi-scale feature information has been proven to be very effective in improving network performance. At the same time, CNN has excellent ability in extracting local features. Therefore, in order to enhance the representation of local detail features within the network, the present invention designs a multi-scale feedforward network (MsFF). The specific implementation details are as follows: Figure 1 As shown in Figure 2. The entire feedforward network consists of three branches, each of which uses convolution kernels of sizes 3×3, 5×5, and 7×7. Through these three branches with different convolution kernel sizes, MsFF can extract local feature information at multiple scales. Specifically, the feature map is first input into a 1×1 convolution, and then input into three depthwise convolution operations of different sizes. The obtained output can be expressed as follows:

[0083] X i =ReLU(DWConv i×i (Conv 1×1 (X in )))

[0084] Among them, X in represents the input feature, i×i represents the convolution kernel size of the three branches, i=3,5,7, X3, X5, X7 represent the input feature X in The corresponding outputs of the convolutions are respectively convolved using convolution kernels of sizes 3×3, 5×5, and 7×7. 1×1 Represents a 1×1 convolution.

[0085] Then, the outputs of these three branches are connected, and multi-scale features are converged through 1×1 convolution to obtain the final output:

[0086] X out =Conv 1×1 [X3,X5,X7]

[0087] Among them, [] represents a connection operation.

[0088] ④ Wavelet Attention Block (WAB)

[0089] The present invention designs a wavelet attention block (WAB), which uses discrete wavelet transform (DWT) combined with attention mechanism to extract shallow features of the image and perform fine processing of the image. The schematic diagram of WAB is shown in Figure 1As shown. DWT decomposes the image into four different frequency subbands, including low-frequency subband (LL) and high-frequency subband (LH, HL, and HH). Each subband reflects a different level of detail in the image. By inputting the subband information into the spatial and channel attention layers, WAB separates the clean background layer of the degraded image from the noise layer such as rain strips and fog, thereby retaining the effective spatial and channel feature information. Finally, the feature maps undergo an inverse discrete wavelet transform to reconstruct them into the original image. The process of WAB can be expressed by the following formula:

[0090] I out =Conv 3×3 (IDWT([I s ,I c ]+I d ))+I in

[0091] Among them, I in represents the input features of the wavelet attention block, Conv 3×3 represents 3×3 convolution, IDWT() represents inverse discrete wavelet transform, I d ,I s ,I c All are intermediate features and have:

[0092] I d =DWT(Conv 3×3 (I in ))

[0093] I s =Sigmoid(Conv 1×1 ([GAP(I d ),GMP(I d )]))

[0094] I c =Conv 1×1 (GAP(I d ))

[0095] Where DWT() represents discrete wavelet transform, GMP() represents global maximum pooling, GAP() represents global average pooling, and Sigmoid() represents Sigmoid function. s Represents the spatial characteristics, I c Represents channel characteristics.

[0096] ⑤Loss function

[0097] The present invention combines the mean square error (MSE) with the structural similarity (SSIM) loss and proposes a hybrid loss function for training the model. This loss function not only evaluates the difference between the restored image and the real image, but also evaluates their structural similarity. The loss function of the image restoration network is specifically designed as:

[0098] L mixture =αL MSE +(1-α)L SSIM

[0099] Among them, L mixture represents the mixing loss, L MSE represents the mean square error loss, L SSIM represents the structural similarity index metric loss, and α is used to balance L MSE and L SSIM Hyperparameters of

[0100] L MSE The calculation formula is as follows:

[0101]

[0102] Among them, C, H, and W represent the height, width, and number of channels of the restored image or the real image, respectively. c,h,w Represents the label value, represents the predicted value;

[0103] L SSIM The calculation formula is as follows:

[0104]

[0105] Among them, μ x , μ y Represent the pixel averages of the restored image and the real image, σ xy represents the covariance between the restored image and the real image, σ x , σ y They represent the pixel standard deviations of the restored image and the real image respectively, and C1 and C2 are different constants.

[0106] The present invention also provides an image restoration ADAS edge device, which is deployed with a trained image restoration network (LSFormer). In order to deploy advanced networks in ADAS, the present invention provides a memristor-friendly implementation of LSFormer, which is suitable for edge artificial intelligence applications. Figure 2The memristor-based implementation of the LSFormer proposed in this paper is demonstrated. The goal of image restoration is to restore the degraded images captured by ADAS to clear images, which are subsequently used for downstream tasks such as object detection in autonomous driving. Providing high-quality images helps ADAS make more accurate decisions. After the training of LSFormer is completed, it is deployed on edge devices (i.e., cameras in ADAS). Specifically, the weights of the attention layer and linear layer in LSFormer are mapped to a memristor crossbar array. Subsequently, the captured degraded images are converted into parallel analog voltages and input into each row of the crossbar array to perform attention and forward propagation calculations. The entire process can be fully simulated based on the 2T2R crossbar array structure.

[0107] In addition, depthwise convolution and 1×1 convolution can also be implemented via a memristor crossbar array. In the depth direction, voltage rows from different channels are simultaneously input to the corresponding rows of the crossbar array. As for 1×1 convolution, voltages at the same positions of different channels are input to the same row of the crossbar array. In addition, to facilitate communication between different crossbar arrays, peripheral circuits are an essential component of the implementation of LSFormer on edge devices. These circuits include layer normalization (LN) circuits, ReLU circuits, Hadamard product circuits, and reshaping circuits. Among these circuits, the ReLU circuit can be implemented using a diode, which blocks when the input row voltage corresponding to the degraded image is lower than the threshold voltage. The LN circuit can be implemented using two op amps and a memristor, while the residual circuit can be implemented using two op amps and multiple resistors. In addition, Figure 3 (a) and (b) show the specific circuits implementing the Hadamard product and reshaping operations, respectively.

[0108] The present invention replaces Softmax attention with linear attention, so the Softmax activation function circuit can be directly replaced with a memristor cross array. These circuits only require cross arrays and op amps, without multipliers, dividers, transistors, and additional NMOS transistors in the Softmax circuit. This operation significantly improves the integration and calculation speed of the circuit. Due to the large area and high power consumption of the Softmax circuit, previous work has achieved voltage activation of multiple circuit branches by extensively reusing this circuit module. However, this process is unnecessary in the memristor implementation corresponding to LSFormer. The linear attention operation can be implemented in a single parallel forward pass, reducing the overhead of clock cycles and additional circuit support. Table 1 presents and compares the performance of existing methods.

[0109] Table 1 Performance comparison with existing methods

[0110]

[0111] The evaluation in Table 1 is based on simulations using a 180nm process, and the memristor model used for comparison is implemented based on the AIST memristor. The power consumption evaluation is based on the estimated maximum energy consumption within the threshold voltage range [-0.1V, 0.1V]. Within this range, the conductance of the memristor remains stable, suitable for neural network inference on edge devices. In Table 1, n represents the number of Softmax circuit modules, so Z / n represents the number of times the Softmax circuit runs inference with a fixed circuit configuration. In contrast, the linear attention circuit can complete one forward inference in a single parallel computation. In addition, since the interference of factors such as temperature is excluded, the linear attention circuit performs well in speed and stability compared to the previously designed Softmax activation function circuit. The LSFormer based on the linear attention implementation not only shows excellent performance in the test benchmarks, but also demonstrates compatibility with brain-like computing systems. Its unique network design allows for high integration into chip architectures designed around memristor components. In addition, the integration and scheduling of the proposed peripheral circuits enable the full circuit implementation of LSFormer on edge devices, providing an efficient and robust edge computing solution. Compared with traditional von Neumann devices, fully integrated neural networks based on memristor-based brain-like computing systems provide a promising system-on-chip approach that can be applied and deployed on practical edge devices.

[0112] The performance of LSFormer is evaluated below.

[0113] A. Evaluation Settings

[0114] 1) Datasets: To evaluate the performance and generalization ability of LSFormer, we conducted comparative experiments on various datasets, including the datasets LOL, DICM, LIME, MEF, and NP for low-light enhancement tasks, the datasets Rain200L, Rain200H, and Rain1200 for image deraining tasks, and the dataset RESIDE for image defogging tasks. For each image restoration task, we performed separate training and testing on the corresponding dataset.

[0115] 2) Comparison networks: For low-light enhancement, the networks used for comparison include RetinexNet, MBLLEN, EnlightenGAN, DBP, HDRNet, DRBN, DeepUPE, and NDATnet. For image deraining, the networks used for comparison include GMM, RESCAN, PReNet, JORDER-E, ReHEN, MSPFN, MPRNet, RCDNet, RLNet, and CDPLG. For image dehazing, the networks used for comparison include DCP, DehazeNet, AOD-Net, GMAN, EPDN, AUEND, and D4.

[0116] 3) Evaluation indicators: In order to quantitatively evaluate the experimental results, the present invention uses multiple evaluation indicators. For paired reference datasets, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) between the restored image and the real image are calculated. In the case of unpaired reference datasets, the natural image quality evaluator (NIQE) is used to measure the quality of the restored image.

[0117] 4) Training details: In the present invention's research, the present invention configures the number of LSTB blocks at each level of the encoder and decoder to be {4, 6, 6, 8}, respectively. The number of attention heads is set to {1, 2, 4, 8}, the initial number of channels is 48, and the expansion ratio is set to 2. Regarding the top-k value of FLSA, it is set to 0.75C, where C represents the number of channels in the feature map. Training is performed using the Adam optimizer for a total of 50 iterations, with a batch size of 16 and a patch size of 64. The initial learning rate is set to 1×10^(-4) with a decay rate of 0.3 every 10 iterations. The training platform uses PyTorch and NVIDIA GeForce RTX 3090GPU to achieve end-to-end model training.

[0118] B. Low illumination enhancement results

[0119] In the low-light enhancement task, the present invention uses the PSNR and SSIM of the LOL dataset as evaluation indicators. For the non-reference datasets such as DICM, LIME, MEF and NPE, the present invention uses NIQE as the evaluation indicator. The quantitative results are shown in Table 2.

[0120] Table 2 Quantitative comparison of low-light enhancement on five benchmark datasets

[0121]

[0122] Note: The best result is bold, and the second best result is underlined

[0123] From the results in Table 2, it can be seen that the network proposed in the present invention performs well on all datasets, improving PSNR and SSIM by at least 3.83dB and 0.086 respectively. On the no-reference dataset, the network of the present invention reduces the NIQE value by an average of 0.3610 relative to the second best result. In order to intuitively demonstrate the performance of the proposed network, the present invention makes a visual qualitative comparison, such as Figure 4 (a) is shown in Figure 2. RetinexNet and MBLLEN produce underexposed restored images, while DeepUPE and DBP have better restoration results than previous networks, but the network of the present invention performs well in capturing details. Quantitative and qualitative results show that the network proposed in the present invention has excellent performance in the low-light enhancement task.

[0124] C. Image deraining results

[0125] For the image deraining task, the present invention conducts experiments on three datasets, using PSNR and SSIM as quantitative evaluation indicators. The quantitative results are shown in Table 3.

[0126] Table 3 Quantitative comparison of image deraining on three benchmark datasets

[0127]

[0128] Note: The best result is bold, and the second best result is underlined

[0129] The results in Table 3 show that the network proposed in the present invention achieves the best results on all three datasets, and improves the PSNR and SSIM by 1.87 dB and 0.017 on average compared to the second best result. Figure 4 (b) shows the qualitative comparison results of the image deraining task. It can be seen that the early network is difficult to effectively remove rain marks, and the subsequent network can basically remove rain, but it will inevitably produce artifacts, color distortion and detail loss. In contrast, the results of the network of the present invention show the best restored image, which is closest to the real image.

[0130] D. Image dehazing results

[0131] On the RESIDE dataset, the present invention uses PSNR and SSIM as evaluation indicators for the image dehazing task. Table 4 presents the quantitative comparison results.

[0132] Table 4 Quantitative comparison of image dehazing on the RESIDE dataset

[0133]

[0134] Note: The best result is bold, and the second best result is underlined

[0135] It can be seen from the results in Table 4 that the network of the present invention outperforms other networks by at least 5.14 dB and 0.027 in terms of PSNR and SSIM. Figure 4 (c) shows a qualitative comparison of all SOTA networks on the RESIDE dataset. Compared with the real image, the proposed network performs well in dehazing and retains more details.

[0136] E. Ablation Experiment

[0137] 1) Selection of Top-k value in FLSA: In order to verify the effectiveness of Top-k in FLSA and the impact of k value on model performance, the present invention conducted experiments with different k values ​​and without Top-k on the LOL dataset, as shown in Table 5. From the results in Table 5, it can be seen that when the k value is too small or too large, or Top-k is not applied, the performance will degrade. Smaller k values ​​select fewer features and weaken the recovery ability of the model, while larger k values ​​introduce redundant features that interfere with image recovery. When the k value is set to 34C, the model performs best, indicating that under appropriate k values, Top-k can effectively improve model performance.

[0138] Table 5 Ablation test results with different k values

[0139]

[0140] 2) Effectiveness of WAB: To verify the effectiveness of the proposed WAB, the present invention uses LSTB as the baseline and designs multiple control groups: V1, V2, V3 and V4. The present invention conducts experiments on the Rain200L dataset, and the results are shown in Table 6. The comparison results in Table 6 show that V4 uses WAB 1 for shallow feature extraction, then inputs the image into LSTB, and uses WAB 2 to optimize the features extracted by LSTB, achieving the best performance. This proves the effectiveness of WAB in enhancing image restoration performance.

[0141] Table 6 Ablation experimental results of WAB

[0142]

[0143] In summary, the present invention provides an image restoration method based on brain-like visual Transformer and an ADAD edge device, which creates a U-shaped image restoration network (referred to as linear sparse Transformer, LSFormer for short) in which a linear sparse Transformer block (LSTB) is proposed, and a focused linear sparse attention block (FLSA) is designed in the linear sparse Transformer block to reduce computational complexity, which helps to alleviate the outlier problem associated with Softmax, and uses content sparsity to select the most relevant feature information, thereby improving the quality of the restored image through these effective features, and promoting the deployment of image restoration networks in intelligent assisted driving vehicles. The present invention also designs a multi-scale feedforward network (MSFF) in the linear sparse Transformer block to enhance local representation by using different depth convolution kernel sizes. The present invention also proposes a wavelet attention block (WAB) in LSFormer to extract shallow features of the input image and improve the output image based on discrete wavelet transform (DWT). In addition, the present invention provides an efficient and promising hardware implementation scheme based on a memristor-like brain computing system for the application of ADAS in intelligent vehicles. The paper also evaluates the performance of the proposed network on multiple datasets, including low-light enhancement, image deraining, and image dehazing tasks. The results show that LSFormer outperforms the state-of-the-art networks in all three tasks. In addition, the paper also conducts multiple metric tests on the proposed implementation, and the results show that the proposed solution is more efficient and stable compared to the existing Transformer circuit.

[0144] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An image restoration method based on brain-like visual Transformer, characterized in that: Includes steps: S1. Build an image restoration network based on brain-like visual Transformer; The image restoration network is a U-shaped network structure, which is sequentially provided with a first 3×3 convolutional layer, a first wavelet attention block, Q linear sparse Transformer blocks, a second wavelet attention block, and a second 3×3 convolutional layer from input to output, where Q is an odd number not less than 2; The degraded image is input into the first 3×3 convolutional layer for convolution and then input into the first wavelet attention block for attention extraction. The extraction result is input into Q linear sparse Transformer blocks for downsampling and upsampling. The sampling result is input into the second wavelet attention block for attention extraction. The extraction result is input into the second 3×3 convolutional layer for convolution. The convolution result is added to the degraded image to obtain a restored image. The entire process of the linear sparse Transformer block is expressed by the following equation: F′ i =FLSA(LN(F i-1 ))+F i-1 F i =MsFF(LN(F′ i ))+F′ i Among them, F′ i 、F i-1 denote the output and input of the i-1th linear sparse Transformer block, respectively, and F′ i represents the intermediate feature map, LN() represents the layer normalization operation, FLSA() represents the focused linear sparse attention extraction operation, and MsFF() represents the multi-scale feedforward operation; FLSA() is expressed as: FLSAtt(Q,K,V)=Focuse(Q)Mask(S,k)+DWConv(V) Where Q, K, V represent the query, key, and value in Transformer, FLSAtt(Q, K, V) represents focused linear sparse attention extraction, Focus() represents the focusing function, S = Focus(K T )V represents the score, Mask() represents the mask function, k represents the adjustable parameter in the top-k operation, and DWConv() represents the depth convolution; The focus function Focus() is expressed as: Among them, x refers to the independent variable, ReLU() represents the ReLU function, and x **p represents the pth power of x element by element; S2, training the image restoration network; S3. Input the degraded image to be restored into the trained image restoration network, and output the restored image.

2. According to claim 1, the image restoration method based on brain-like visual Transformer is characterized in that: MsFF() is expressed as: X out =Conv 1×1 [X3,X5,X7] Among them, X3, X5, and X7 represent the input feature X in The corresponding outputs of the convolutions are respectively performed using convolution kernels of sizes 3×3, 5×5, and 7×7. [] indicates a connection operation. Conv 1×1 Represents 1×1 convolution; X3, X5, and X7 are represented by: X i =ReLU(DWConv i×i (Conv. 1×1 (X in ))) Where i = 3, 5, 7, DWConv i×i Indicates the use of depthwise convolution with a convolution kernel size of i×i.

3. The image restoration method based on brain-like visual Transformer according to claim 2 is characterized in that: The first wavelet attention block and the second wavelet attention block use the same wavelet attention block, and the process of the wavelet attention block is expressed by the formula: I out =Conv 3×3 (IDWT([I s ,I c ]+I d ))+I in Among them, I in represents the input features of the wavelet attention block, Conv 3×3 represents 3×3 convolution, IDWT() represents inverse discrete wavelet transform, I d ,I s ,I c All are intermediate features and have: I d =DWT(Conv 3×3 (I in )) I s =Sigmoid(Conv 1×1 ([GAP(I d ),GMP(I d )])) I c =Conv 1×1 (GAP(I d )) Among them, DWT() represents discrete wavelet transform, GMP() represents global maximum pooling, GAP() represents global average pooling, and Sigmoid() represents Sigmoid function.

4. The image restoration method based on brain-like visual Transformer according to claim 1 is characterized in that: During the training process of step S2, the loss function of the image restoration network is designed as: L mixture =αL MSE +(1-α)L SSIM Among them, L mixture represents the mixing loss, L MSE represents the mean square error loss, L SSIM represents the structural similarity index metric loss, and α is used to balance L MSE and L SSIM Hyperparameters of L MSE The calculation formula is as follows: Among them, C, H, and W represent the height, width, and number of channels of the restored image or the real image, respectively. c,h,w Represents the label value, represents the predicted value; L SSIM The calculation formula is as follows: Among them, μ x , μ y Represent the pixel averages of the restored image and the real image, σ xy represents the covariance between the restored image and the real image, σ x , σ y They represent the pixel standard deviations of the restored image and the real image respectively, and C1 and C2 are different constants.

5. The image restoration method based on brain-like visual Transformer according to claim 4 is characterized in that: Q=7, the 7 linear sparse Transformer blocks form a U-shaped structure, the first 4 linear sparse Transformer blocks perform downsampling, and the image size is transformed into H×W×C, The last three linear sparse Transformer blocks perform upsampling, transforming the image size into H×W×C.

6. An image restoration ADAS edge device, characterized by: A trained image restoration network according to any one of claims 1 to 5 is deployed, wherein the weights of the attention layer and the linear layer of the image restoration network are mapped to a first memristor crossbar array; The captured degraded image is converted into parallel analog voltages, and the parallel analog voltages are input into each row of the first memristor crossbar array to perform attention and forward propagation calculations.

7. The image restoration ADAS edge device according to claim 6, characterized in that: The depth convolution of the image inpainting network is implemented by a second memristor cross array, and in the depth direction, voltage rows from different channels are simultaneously input to corresponding rows of the second memristor cross array; The 1×1 convolution of the image restoration network is implemented through a third memristor cross array, and voltages at the same position of different channels are input to the same row of the third memristor cross array.

Images