A method for binocular image restoration in rainy and snowy weather based on brain-like sparse attention mechanisms

Abstract

This invention relates to a general method for binocular image restoration in rainy and snowy weather based on a brain-like sparse attention mechanism, belonging to the field of computer vision. The method includes the following steps: S1: Constructing a binocular image dataset for rainy and snowy weather, using a binocular camera to acquire paired binocular rain / snow images and corresponding clear binocular images; S2: Designing a binocular image restoration network for rainy and snowy weather based on a brain-like sparse attention mechanism; S3: Training the brain-like sparse attention mechanism-based binocular image restoration network designed in S2 using the rainy and snowy weather binocular image dataset constructed in S1; calculating the loss value for the restored binocular images and the predicted rain / snow images, and optimizing the entire network using the Adam optimizer, finally training a binocular image restoration model for rainy and snowy weather based on a brain-like sparse attention mechanism. This invention reduces computational load and memory usage, overcomes the limitation of weak anti-occlusion ability in single-view images, and improves the quality of rainy and snowy image restoration.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a general method for binocular image restoration in rainy and snowy weather based on a brain-like sparse attention mechanism. Background Technology

[0002] Images captured in rain, snow, and fog often suffer from poor quality, severely impacting human visual perception and significantly reducing the performance of subsequent visual perception tasks. Image de-raining or de-snowing aims to restore degraded images captured in rainy or snowy environments, improving the perception capabilities of the human eye and intelligent perception systems in such conditions. Particularly in autonomous driving, image quality is easily affected by severe weather conditions due to the dynamic and ever-changing nature of real-world scenarios, resulting in a significant decline in imaging quality. Therefore, researching image restoration methods for rainy and snowy conditions has significant research and application value.

[0003] The pixels corresponding to the same rain line, raindrop (or snowflake) are located at different positions in the left and right views. When rain lines, raindrops (or snowflakes) degrade the image quality of a certain area in the left image, the corresponding pixels in the same area in the right image may not be affected. Based on this characteristic, binocular image rain removal methods have been extensively studied. In traditional binocular rain removal methods, Yamashita et al. first used the parallax of binocular images to detect the position of raindrops, and then replaced the raindrops with clear images to achieve image restoration. Recently, data-driven rain removal methods have become mainstream. Zhang et al. used Photoshop to synthesize the binocular rainy day datasets RainKITTI2012 and RainKITTI2015, and proposed binocular image rain line removal networks PRRNet and EPRRNet. By fusing elements one by one to mine multi-view information, introducing additional semantic segmentation labels for semantic segmentation, and utilizing the multi-view and semantic information of binocular images, they removed rain lines from the scene. Wei et al. proposed cross-view interaction based on an attention mechanism. Shi et al. collected a real-world binocular raindrop image dataset, the StereoWaterdrop dataset, and proposed a method for removing raindrops from binocular images. This method utilizes a row-extended attention module to expand the receptive field of attention, enabling effective information propagation between binocular images. However, the row-extended attention module explores interactive information by constructing an association matrix, which is time-consuming. The aforementioned binocular rain removal method has shortcomings: insufficient multi-view feature interaction and high complexity, resulting in low modeling efficiency. Furthermore, there is a lack of research on general binocular image restoration methods for severe weather conditions such as rain and snow. To address these issues, inspired by the selective attention of the brain, this invention proposes a general binocular image restoration method for rain and snow weather based on a brain-like sparse attention mechanism. This method efficiently mines and models the hierarchical dynamic interaction relationships of binocular images, achieving more efficient and higher-quality image restoration for rain and snow weather. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a general binocular image restoration method for rainy and snowy days based on a brain-like sparse attention mechanism. By utilizing the complementary information between binocular images, the method dynamically and efficiently completes the information obscured by rain lines, raindrops, or snowflakes, fully and efficiently models interactive multi-view features, improves efficiency, and ultimately forms a binocular image restoration method for rainy and snowy days based on a brain-like sparse attention mechanism.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for binocular image restoration in rainy and snowy weather based on a brain-like sparse attention mechanism includes the following steps:

[0007] S1: Construct a binocular image dataset for rain and snow days, and use a binocular camera to collect pairs of binocular rain and snow images and corresponding clear binocular images;

[0008] S2: Design a binocular image de-raining and snow removal network based on a brain-like sparse attention mechanism;

[0009] The binocular image desnowing network is divided into two parts: a rain and snow probability map prediction branch and a desnowing branch based on a brain-like sparse attention mechanism;

[0010] The rain and snow probability map prediction branch is a lightweight rain and snow probability prediction network that predicts the presence and probability of rain lines, raindrops, and snowflakes to obtain a rain and snow probability map.

[0011] The rain and snow removal branch based on the brain-like sparse attention mechanism includes a basic feature extraction model and a brain-like sparse attention model; wherein, the basic feature extraction model is a parameter-shared weight-shared network that extracts left view features and right view features.

[0012] The extracted left and right feature maps and the predicted rain and snow probability map are input into the brain-like sparse attention model to calculate the sparse row correlation matrix of the left and right feature maps. The sparse row correlation matrix is ​​multiplied with the right feature map to extract the features of the occluded area of ​​the left image from the right feature map and added to the left feature map to obtain the dynamically completed feature map. Finally, the dynamically completed feature map is used to predict the restoration of the left image.

[0013] Swap the feature positions of the left and right images and input them into the brain-like sparse attention model to complete the restoration of the right image;

[0014] S3: The brain-like binocular image rain and snow removal network designed in S2 is trained using the binocular image dataset of rain and snow days constructed in S1; the loss function of L1 loss is used to calculate the loss value of the restored binocular image and the predicted rain and snow map, and the Adam optimizer is combined to optimize the entire network. Finally, a binocular image restoration model of rain and snow days based on the brain-like sparse attention mechanism is trained.

[0015] Optionally, S1 specifically includes:

[0016] We collected paired binocular rain and snow images and their corresponding clear binocular images to construct a binocular rain and snow dataset. We set up a binocular camera and first captured clear binocular images. Then, we used a snow machine to create snow and captured binocular images of snowy days. Simultaneously, we placed a glass pane in front of the lens and simulated rainy days by splashing water or using a sprayer to capture binocular images of rainy days. Finally, we obtained realistic paired binocular rain and snow images and their corresponding clear images.

[0017] Optionally, S2 specifically includes:

[0018] The extracted left and right feature maps and the predicted rain / snow probability map are input into a brain-like sparse attention model. For the restored left feature map, the right feature map F is calculated first. r and left feature map F l The row correlation matrix M, based on the rain / snow probability map P rs Guided by this principle, the values ​​for positions where raindrops exist in the row correlation matrix remain unchanged, while the values ​​for positions where no raindrops obstruct the view are set to 0, resulting in a sparse row correlation matrix M. s ; through the sparse row correlation matrix and the right feature map value vector V r Perform matrix multiplication to extract features of the occluded region in the left image from the right feature map, and add them to the left feature map to obtain the dynamically completed left feature map F′. l Finally, the left feature map obtained through dynamic completion is used to predict the restored image, resulting in the left image after removing rain or snow. The calculation process is as follows:

[0019] Q l =Conv 1×1 (LN(F l )),

[0020] K r =Conv 1×1 (LN(F r )),

[0021] V r =Conv 1×1 (LN(F r )),

[0022] M = Softmax(Q) l *K r ),

[0023]

[0024] F′ l =F l +M s *V r ,

[0025] Among them, Conv 1×1 This is a convolutional layer with a kernel size of 1×1, LN is the layer normalization operation, * represents matrix multiplication, · represents pointwise multiplication, and Softmax is the activation function. This is a binarization operation;

[0026] The restoration of the right feature map involves swapping the positions of the input features in the brain-like sparse attention model, and then exchanging the features in the left and right images to obtain the dynamically completed right feature map F′. r Finally, the right feature map with dynamic completion is used to predict the restored image, resulting in the right image with rain or snow removed.

[0027] The beneficial effects of this invention are as follows: Based on a brain-like sparse attention mechanism, this invention dynamically interacts with areas containing rain lines, raindrops, or snowflakes. Compared with existing binocular image restoration methods that use row-extended attention modules, this invention reduces computational load and memory usage, while overcoming the limitation of weak anti-occlusion capability of a single view and improving the quality of image restoration in rainy or snowy weather.

[0028] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0030] Figure 1 This is a flowchart of the present invention;

[0031] Figure 2 It is a brain-like sparse attention model. Detailed Implementation

[0032] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0034] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0035] The embodiments of the present invention include the following steps:

[0036] S1: Collect paired binocular rain / snow images and their corresponding clear binocular images to construct a binocular image dataset for rain / snow days. Prepare a binocular camera and first capture clear binocular images. Then, use a snow machine to create snow and capture binocular images of snowy days. Simultaneously, place a glass pane in front of the lens and simulate rainy days by splashing water or using a sprayer to capture binocular images of rainy days. This method yields realistic paired binocular rain / snow images and their corresponding clear images.

[0037] S2: Design a binocular image deraining network based on a brain-like sparse attention mechanism. The entire network consists of two parts: a rain / snow probability map prediction branch and a deraining branch based on a brain-like sparse attention mechanism. For example... Figure 1 As shown, the rain and snow probability map prediction branch is a lightweight rain and snow probability prediction network. The network structure is a U-shaped network with relatively few model parameters, containing an encoder and a decoder. The encoder extracts robust and discriminative features from the input image using convolution and downsampling operations. The decoder upsamples the features and recovers detailed feature information through convolution. Finally, it predicts the presence and probability of rain lines, raindrops, or snowflakes to obtain the rain and snow probability map. Figure 1 As shown, the deraining and snow removal branch based on the brain-like sparse attention mechanism includes a basic feature extraction model and a brain-like sparse attention model. Among them, the weight-sharing network is the basic feature extraction model with shared parameters, extracting features from the left view and the right view.

[0038] The extracted left and right feature maps and the predicted rain / snow probability map are input into the brain-like sparse attention model: For example... Figure 2As shown, taking the restoration of the left image as an example, the right feature map F is first calculated. r and left feature map F l The row correlation matrix M, based on the rain / snow probability map P rs Guided by this principle, the values ​​of positions where rain or snow exists in the row correlation matrix remain unchanged, while the values ​​of positions without rain or snow occlusion are set to 0, resulting in a sparse row correlation matrix M. s Next, the sparse row correlation matrix and the right feature map value vector V are used to... r Perform matrix multiplication to extract features of the occluded region in the left image from the right feature map, and add them to the left feature map to obtain the dynamically completed left feature map F′. l Finally, the dynamically completed left feature map is used to predict the restored image, resulting in the left image after removing rain or snow. The specific calculation process can be represented as follows:

[0039] Q l =Conv 1×1 (LN(F l )),

[0040] K r =Conv 1×1 (LN(F r )),

[0041] V r =Conv 1×1 (LN(F r )),

[0042] M = Softmax(Q) l *K r ),

[0043]

[0044] F′ l =F l +M s *V r ,

[0045] Among them, Conv 1×1 This is a convolutional layer with a kernel size of 1×1, LN is the layer normalization operation, * represents matrix multiplication, · represents pointwise multiplication, and Softmax is the activation function. This is a binarization operation.

[0046] Similarly, restoring the right image only requires swapping the positions of the input features in the brain-like sparse attention model, exchanging the features from the left and right images, and finally obtaining the dynamically completed right feature map F′. r Finally, the right feature map with dynamic completion is used to predict the restored image, resulting in the right image with rain or snow removed.

[0047] S3: Training the Network. The L1 loss function is used to calculate the loss value between the reconstructed stereo image and the ground truth stereo image. Simultaneously, the L1 loss function is used to calculate the loss value between the rain / snow image predicted by the rain / snow probability map prediction branch and the ground truth rain / snow image. These two loss values ​​are summed and used to optimize the entire network. The entire network is then optimized using an Adaptive Moment Estimation (Adam) optimizer. Finally, a stereo image reconstruction model for rain / snow days based on a brain-like sparse attention mechanism is trained.

[0048] S4: Test the binocular image restoration system for rainy / snowy days based on the brain-like sparse attention mechanism of this invention. Input a binocular image of a rainy / snowy day, process it through the binocular image restoration system of this invention, and obtain clear pairs of binocular images.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for binocular image restoration in rainy and snowy weather based on a brain-like sparse attention mechanism, characterized by: The method includes the following steps: S1: Construct a binocular image dataset for rain and snow days, and use a binocular camera to collect pairs of binocular rain and snow images and corresponding clear binocular images; S2: Design a binocular image rain and snow removal network based on a brain-like sparse attention mechanism; The binocular image desnowing network is divided into two parts: a rain and snow probability map prediction branch and a desnowing branch based on a brain-like sparse attention mechanism; The rain and snow probability map prediction branch is a lightweight rain and snow probability prediction network that predicts the presence and probability of rain lines, raindrops, or snow to obtain a probability map of rain or snow. The rain and snow removal branch based on the brain-like sparse attention mechanism includes a basic feature extraction model and a brain-like sparse attention model; wherein, the basic feature extraction model is a parameter-shared weight-shared network that extracts left view features and right view features. The extracted left and right feature maps and the predicted rain or snow probability map are input into the brain-like sparse attention model to calculate the sparse row correlation matrix of the left and right feature maps. The sparse row correlation matrix is ​​multiplied with the right feature map to extract the features of the occluded area of ​​the left image from the right feature map, and then added to the left feature map to obtain the dynamically completed left feature map. Finally, the dynamically completed left feature map is used to predict the restoration of the left image. The feature positions of the left and right images are swapped and input into the brain-like sparse attention model to obtain the dynamically completed right feature map, which is then used to complete the restoration of the right image. S3: The brain-like binocular image rain and snow removal network designed in S2 is trained using the binocular image dataset of rain and snow days constructed in S1; the loss function of L1 loss is used to calculate the loss value of the restored binocular image and the predicted rain and snow map, and the Adam optimizer is combined to optimize the entire network. Finally, a binocular image restoration model of rain and snow days based on the brain-like sparse attention mechanism is trained.

2. The binocular image restoration method for rainy and snowy days based on a brain-like sparse attention mechanism according to claim 1, characterized in that: Specifically, S1 is: Collect paired binocular rain and snow images and corresponding clear binocular images to construct a binocular image dataset for rain and snow days; set up a binocular camera and first capture clear binocular images; then use a snowmaking machine to create snow and capture binocular images of snowy days; at the same time, place a glass in front of the lens and simulate rainy days by splashing water or using a sprayer to capture binocular images of rainy days; finally, obtain real paired binocular rain and snow images and corresponding clear images.

3. The binocular image restoration method for rainy and snowy days based on a brain-like sparse attention mechanism according to claim 2, characterized in that: Specifically, S2 is: The extracted left and right feature maps and the predicted rain / snow probability map are input into a brain-like sparse attention model. For the restored left feature map, the right feature map F is calculated first. r and left feature map F l The horizontal correlation matrix, i.e., the row correlation matrix M, is based on the rain / snow probability map P. rs Guided by this principle, the values ​​of locations in the correlation matrix where rain or snow exists remain unchanged, while the values ​​of locations without rain or snow are set to 0, resulting in a sparse row correlation matrix M. s ; through the sparse row correlation matrix and the right feature map value vector V r Perform matrix multiplication to extract features of the occluded region in the left image from the right feature map, and add them to the left feature map to obtain the dynamically completed left feature map F′. l Finally, the left feature map obtained through dynamic completion is used to predict the restored image, resulting in the left image after removing rain and snow. The calculation process is as follows: Q l =Conv 1×1 (LN(F l )), K r =Conv 1×1 (LN(F r )), V r =Conv 1×1 (LN(F r )), M=Softmax(Q l *K r ), F′ l =F l +M s *V r , Among them, Conv 1×1 This is a convolutional layer with a kernel size of 1×1, LN is the layer normalization operation, * represents matrix multiplication, · represents pointwise multiplication, and Softmax is the activation function. For binarization operation; Q l For the query vector, K r V is the key vector. r Let M and M be value vectors. s These are the row correlation matrix and the sparse row correlation matrix, respectively, P rs This is a probability graph for rain and snow. The restoration of the right feature map involves swapping the positions of the input features in the brain-like sparse attention model, and then exchanging the features in the left and right images to obtain the dynamically completed right feature map F′. r Finally, the right feature map with dynamic completion is used to predict the restored image, resulting in the right image with rain or snow removed.

Images