Traffic image rain removal method based on reference image

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a traffic image rain removal method based on a reference image, which comprises the following steps of: selecting and establishing a data set; preprocessing the data set; a rain removal model is constructed, and a backbone network extracts rich features by stacking a plurality of encoder modules, a multi-scale feature matching module and a decoder module; training the rain removal model by using the training set in the target data set, and obtaining a trained rain removal model after the trained rain removal model is verified to be qualified through the test set; and inputting a traffic rain map to be processed into the trained rain removal model for background image reconstruction, and outputting a reconstructed traffic road image by the rain removal model. The invention constructs a novel network model for natural image rain removal research, the network model focuses on feature extraction, and aims to accurately capture potential features in a rain map and a reference image corresponding to the rain map and ensure that more long-distance dependence information can be captured so as to enhance the restoration precision of a traffic background image.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a traffic image deraining method based on a reference image. Background Art

[0002] With the rapid development of the transportation industry, intelligent transportation systems have gradually become an important means to improve traffic efficiency and ensure traffic safety. However, under severe weather conditions, such as rainy days, the quality of traffic images will be seriously affected, and rain line noise will interfere with the clarity of the image, thereby affecting the performance of the intelligent transportation system. Therefore, how to effectively remove rain line noise from traffic images and improve image clarity has become an urgent problem to be solved in the transportation field. Clear traffic images not only help intelligent transportation systems to more accurately identify traffic elements such as vehicles and pedestrians, improve the efficiency of traffic management, but also provide stronger guarantees for traffic safety. However, due to the complexity and diversity of traffic scenes, as well as the randomness and uncertainty of rain line noise, traffic image deraining faces many challenges. How to maintain the details and edge features of the image while ensuring the deraining effect has become the key to traffic image deraining technology.

[0003] At present, image deraining methods mainly include methods based on physical models, image processing knowledge, sparse coding dictionary learning and classifiers, and deep convolutional neural networks. These methods can remove rain line noise in traffic images to a certain extent and improve the clarity of the image. However, in the face of complex traffic scenes and diverse rain line noise, existing methods still have some limitations. For example, methods based on physical models require accurate rain line models, while methods based on deep learning require a large amount of training data and computing resources. In view of the limitations of existing methods, it is particularly important to propose a new image deraining method. Summary of the invention

[0004] In view of the technical problems that the physical model-based methods in the above-mentioned existing methods require an accurate rain line model, while the deep learning-based methods require a large amount of training data and computing resources, the present invention provides a traffic image deraining method based on a reference image.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A traffic image deraining method based on a reference image comprises the following steps:

[0007] S1. Select and establish data set;

[0008] S2, preprocessing the data set;

[0009] S3. Build a rain removal model. The backbone network extracts rich features by stacking multiple encoder modules, multi-scale feature matching modules, and decoder modules. These features can capture spatially varying rainfall distribution. Use the training set in the target dataset to train the rain removal model. After verifying that the trained rain removal model is qualified through the test set, the trained rain removal model is obtained.

[0010] S4. Input the traffic rain image to be processed into the trained rain removal model to reconstruct the background image, and the rain removal model outputs the reconstructed traffic road image.

[0011] The method for selecting and establishing a data set in S1 is: selecting and collecting a variety of traffic images to construct an initial data set for a rain removal model, and dividing the initial data set into a training set and a test set, wherein the initial data set includes a plurality of rain scene images of traffic roads.

[0012] The method for preprocessing the data set in S2 is: using the PyTorch framework, loading the pretrained VGG16 model, and intercepting the first 29 layers of the model as a feature extractor; these layers contain most of the convolution operations of the model and can effectively extract high-level features of the image; for each traffic road rain image and traffic road image in normal weather, after preprocessing, input them into the VGG16 feature extractor to obtain the corresponding feature vector; the preprocessing steps include adjusting the image size, center cropping, converting the image into a tensor, and normalizing it; these feature vectors reflect the high-level visual information of the image; the Pearson correlation coefficient is used to measure the similarity between the rain image features and the normal image features; for each input rain image, traverse all normal images in the training set except the ground truth, calculate the correlation coefficient between their feature vectors and the input rain image feature vector, and select the target image with the highest correlation coefficient as the reference image.

[0013] The training set of the target data set in S3 includes multiple traffic road rain streak images and traffic road images under a clean background corresponding to each traffic road rain streak image; the test set of the target data set is the same as the test set of the initial data set, both including multiple traffic road rain streak images.

[0014] The method for constructing the rain removal model in S3 is:

[0015] S3.1. Image encoding: The encoder module is composed of multiple transformer blocks connected in series. Each transformer module integrates a multi-head self-attention mechanism and a feed-forward neural network to effectively capture the global dependencies in the image. In the initial stage, the image is subjected to preliminary feature extraction through L1 transformer blocks, followed by downsampling, which effectively reduces the data dimension and extracts key features. These downsampled feature maps are further subjected to L2 transformer blocks for deep feature extraction and more refined downsampling. This process significantly enhances the model's ability to understand the image content.

[0016] S3.2, Image decoding and reconstruction: The decoder module extracts high-level features through L2 Transformer blocks, then goes through an upsampling process and enters L1 Transformer blocks for further refinement. In the downsampling stage of the entire model, an innovative feature alignment and matching module is added between the encoder module and the decoder module. Through residual connections, the output of the encoder module is spliced ​​with the output of the decoder module to fuse local features and aligned features. The recursive numbers of Transformer blocks L1, L2, and L3 for the best deraining performance are set to 2, 3, and 3 respectively.

[0017] S3.3, Feature alignment and matching module: A two-stage matching strategy is adopted, combining the reference image-based method with the feature alignment and acceleration matching strategy, aiming to overcome the problem of image detail loss caused by the traditional rain removal algorithm that only considers single image rain removal; the feature alignment and matching module can locate and extract feature blocks that play a key role in rain removal, thereby optimizing the rain removal effect; the rain image features are expanded into multiple non-overlapping blocks, and the most similar reference block is found for each block in the rain image; in order to improve the accuracy, local block matching is further performed in each pair of rain blocks and reference blocks; finally, useful reference features are extracted based on the corresponding information obtained.

[0018] In S3.3, the method of expanding the rain image features into multiple non-overlapping blocks and finding the most similar reference block for each block in the rain image is:

[0019] S3.3.1. In the first stage, the rain image features are first expanded into N non-overlapping blocks, each of which consists of multiple w×h local blocks; for each rain block, the most relevant reference block is found; first, the central local block of the rain block is taken out, and the cosine similarity is calculated with each local block in the reference block; according to the similarity score, the local block in the reference image features that is most similar to the central local block of the rain block is found; a block of size w×h is cropped from the surrounding area as the most relevant reference block; after the first stage matching, each rain block obtains the most relevant reference block;

[0020] S3.3.2. In the second stage, local block matching is performed on each rain block and the reference block to obtain their index map and similarity map; the similarity score between each local block in the rain block and each local block in the reference block is calculated; the relevant blocks are extracted according to the index map, and these blocks are multiplied with the corresponding similarity map to obtain weighted features; finally, these N features are merged to obtain the reference feature.

[0021] The S3 also includes using the peak signal-to-noise ratio PSNR and the structural similarity index SSIM to evaluate the quality of the reconstruction result; the larger the PSNR, the better the image quality; SSIM is a number between 0 and 1, and the larger the SSIM, the smaller the difference between the output image and the undistorted image, that is, the better the image quality.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention constructs a new network model for natural image deraining research, which focuses on feature extraction and aims to accurately capture the potential features in the rain image and its corresponding reference image, ensuring that more long-distance dependent information can be captured to enhance the restoration accuracy of the traffic background image. In addition, the present invention also integrates a feature alignment and matching strategy, which effectively integrates multiple traffic road images in normal weather as visual information of reference images into the traffic rain streak image, further refines the potential features of the image, and realizes efficient style transfer. This strategy enables the traffic rain image to fully absorb the valuable guidance information of the rain-free reference image, and then reconstruct a derained image with richer details and better effects. After being processed by the deraining model, the quality of the reconstructed image is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0025] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0026] Figure 1A flow chart of the traffic image deraining method based on a reference image provided by the present invention.

[0027] Figure 2 It is a schematic diagram of the composition structure of the rain removal model in the present invention.

[0028] Figure 3 It is a schematic diagram of the composition structure of the encoder in the present invention.

[0029] Figure 4 It is a schematic diagram of the composition structure of the decoder in the present invention.

[0030] Figure 5 Schematic diagram of the composition structure of the Transformer block in the present invention.

[0031] Figure 6 Schematic diagram of the execution process of the reference image selection strategy in the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0034] An embodiment of the present invention provides a method for removing rain from a traffic image based on a reference image, which comprises the following steps:

[0035] Step 1: Dataset selection and establishment: Select and collect a variety of traffic images to construct an initial data set for the rain removal model, and divide the initial data set into a training set and a test set. The initial data set includes multiple rainy images of traffic roads.

[0036] Step 2: Dataset preprocessing:

[0037] Using the PyTorch framework, the pre-trained VGG16 model is loaded, and the first 29 layers of the model are intercepted as feature extractors. These layers contain most of the convolution operations of the model and can effectively extract high-level features of the image. For each traffic road rain image and traffic road image in normal weather, after preprocessing, they are input into the VGG16 feature extractor to obtain the corresponding feature vector. The preprocessing steps include resizing the image, center cropping, converting the image to a tensor, and normalizing it. These feature vectors reflect the high-level visual information of the image. The Pearson correlation coefficient is used to measure the similarity between the features of the rain image and the features of the normal image. For each input rain image, all normal images in the training set except the ground truth are traversed, and the correlation coefficient between their feature vectors and the input rain image feature vector is calculated, and the target image with the highest correlation coefficient is selected as the reference image.

[0038] This strategy ensures that the selected reference image is highly consistent with the image block to be processed at the feature level, providing valuable reference information for subsequent rain removal processing. This strategy shows significant advantages in terms of efficiency and accuracy. By integrating the pre-trained deep network for feature extraction and the Pearson correlation coefficient for accurate similarity calculation, we can quickly and accurately lock the reference image that is most similar to the image block to be processed. This innovative measure significantly improves the efficiency of rain removal and is expected to achieve a significant improvement in the quality of the image after rain removal, especially in restoring image details that are severely obscured by raindrops. In addition, this strategy shows wide versatility and can be flexibly applied to rain removal tasks for various types of traffic images, showing strong practical value.

[0039] The training set of the target dataset includes multiple traffic road rain streak images and traffic road images under clean backgrounds corresponding to each traffic road rain streak image. The test set of the target dataset is the same as the test set of the initial dataset, both of which include multiple traffic road rain streak images.

[0040] Step 3: Model construction: Build a rain removal model, such as Figure 2 As shown in the figure, the backbone network extracts rich features by stacking multiple encoder modules, multi-scale feature matching modules and decoder modules, which can capture the spatially varying rainfall distribution. The deraining model is trained using the training set in the target dataset, and the trained deraining model is verified to be qualified by the test set to obtain the trained deraining model.

[0041] Step 3.1: Image encoding: Figure 3As shown in the figure, the encoder module is composed of multiple Transformer blocks connected in series. Each Transformer module integrates a multi-head self-attention mechanism and a feedforward neural network to effectively capture the global dependencies in the image. In the initial stage, the image is subjected to preliminary feature extraction through L1 Transformer blocks, followed by downsampling, which effectively reduces the data dimension and extracts key features. These downsampled feature maps are further subjected to L2 Transformer blocks for deep feature extraction and more refined downsampling. This process significantly enhances the model's ability to understand the image content.

[0042] Step 3.2: Image decoding and reconstruction: Figure 4 , Figure 5 As shown in the figure, the decoder module extracts high-level features through L2 Transformer blocks, then goes through an upsampling process and enters L1 Transformer blocks for further refinement. In the downsampling stage of the entire model, a feature alignment matching module is innovatively introduced between the encoder module and the decoder module. Through residual connections, the output of the encoder module is spliced ​​with the output of the decoder module to fuse local features and aligned features. Experiments have shown that the recursive numbers of Transformer blocks L1, L2, and L3 that achieve the best deraining performance are set to 2, 3, and 3, respectively.

[0043] Step 3.3: Feature alignment matching module: The present invention adopts a two-stage matching strategy, such as Figure 6 As shown in the figure, the reference image-based method is innovatively combined with feature alignment and accelerated matching strategies, aiming to overcome the problem of image detail loss caused by traditional rain removal algorithms that only consider single image rain removal. The feature alignment matching module can locate and extract feature blocks that are critical to rain removal, thereby optimizing the rain removal effect. Specifically, the rain image features are expanded into multiple non-overlapping blocks, and the most similar reference block is found for each block in the rain image. This significantly reduces the matching computational cost compared to traditional methods. In order to improve accuracy, local block matching is further performed in each pair of rain blocks and reference blocks. Finally, useful reference features are extracted based on the corresponding information obtained.

[0044] Specifically, the rain image features are expanded into multiple non-overlapping blocks, and the most similar reference block is found for each block in the rain image. This significantly reduces the matching computational cost compared to traditional methods. In order to improve accuracy, local block matching is further performed in each pair of rain blocks and reference blocks. Finally, relevant reference features are extracted based on the matching information.

[0045] Step 3.3.1: In the first stage, the rain image features are first expanded into N non-overlapping blocks, each of which consists of multiple w×h local blocks. For each rain block, find its most relevant reference block. First, take out the central local block of the rain block and calculate the cosine similarity with each local block in the reference block. According to the similarity score, find the local block in the reference image feature that is most similar to the central local block of the rain block. A block of size w×h is cropped from the surrounding area as the most relevant reference block. After the first stage matching, each rain block obtains the most relevant reference block.

[0046] Step 3.3.2: In the second stage, local block matching is performed on each rain block and the reference block to obtain their index map and similarity map. The similarity score between each local block in the rain block and each local block in the reference block is calculated. The relevant blocks are extracted according to the index map, and these blocks are multiplied with the corresponding similarity map to obtain weighted features. Finally, these N features are merged to obtain the reference feature.

[0047] After training the deraining model, the embodiment of the present invention also uses two indicators, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), to evaluate the quality of the reconstruction result. The larger the PSNR, the better the image quality. SSIM is a number between 0 and 1. The larger the SSIM, the smaller the difference between the output image and the undistorted image, that is, the better the image quality.

[0048] Step 4: Input the traffic rain image to be processed into the trained rain removal model to reconstruct the background image, and the rain removal model outputs the reconstructed traffic road image.

[0049] In summary, the present invention provides an innovative rain image reconstruction method, which significantly improves the processing effect of traffic images under rainy conditions by combining reference images with feature alignment and matching strategies. The present invention uses the technical means of deep learning and feature engineering to construct a rain removal model that focuses on feature extraction and efficient matching. By introducing a reference image, the model can accurately capture the potential correlation between the rain image and the clean background image, effectively overcoming the shortcomings of the traditional single image rain removal algorithm in preserving image details. At the same time, the introduction of the feature alignment and matching module further improves the rain removal accuracy and efficiency of the model, and realizes the accurate migration and efficient use of visual information. This method can not only remove rain line noise, but also enhance the detailed features of the image during the reconstruction process, generate clearer and more realistic rain removal images, and provide solid technical support for the stable operation of the intelligent transportation system.

[0050] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. A traffic image deraining method based on a reference image, characterized in that: The following steps are involved: S1. Select and establish data set; S2, preprocessing the data set; S3. Build a rain removal model. The backbone network extracts rich features by stacking multiple encoder modules, multi-scale feature matching modules, and decoder modules. These features can capture spatially varying rainfall distribution. Use the training set in the target dataset to train the rain removal model. After verifying that the trained rain removal model is qualified through the test set, the trained rain removal model is obtained. S4. Input the traffic rain image to be processed into the trained rain removal model to reconstruct the background image, and the rain removal model outputs the reconstructed traffic road image.

2. The method for removing rain from traffic images based on reference images according to claim 1, characterized in that: The method for selecting and establishing a data set in S1 is: selecting and collecting a variety of traffic images to construct an initial data set for a rain removal model, and dividing the initial data set into a training set and a test set, wherein the initial data set includes a plurality of rain scene images of traffic roads.

3. The method for removing rain from traffic images based on reference images according to claim 1, characterized in that: The method for preprocessing the data set in S2 is: using the PyTorch framework, loading the pre-trained VGG16 model, and intercepting the first 29 layers of the model as a feature extractor; these layers contain most of the convolution operations of the model, which can effectively extract high-level features of the image; for each traffic road rain image and traffic road image under normal weather, after preprocessing, input it into the VGG16 feature extractor to obtain the corresponding feature vector; the preprocessing steps include adjusting the image size, center cropping, converting the image into a tensor, and performing normalization processing; These feature vectors reflect the high-level visual information of the image. The Pearson correlation coefficient is used to measure the similarity between the features of the rain image and the features of the normal image. For each input rain image, all normal images in the training set except the ground truth are traversed, the correlation coefficient between their feature vectors and the feature vector of the input rain image is calculated, and the target image with the highest correlation coefficient is selected as the reference image.

4. The method for removing rain from traffic images based on reference images according to claim 1, characterized in that: The training set of the target data set in S3 includes multiple traffic road rain streak images and traffic road images under a clean background corresponding to each traffic road rain streak image; the test set of the target data set is the same as the test set of the initial data set, both including multiple traffic road rain streak images.

5. The method for removing rain from traffic images based on reference images according to claim 1, characterized in that: The method for constructing the rain removal model in S3 is: S3.

1. Image encoding: The encoder module is composed of multiple Transformer blocks connected in series. Each Transformer module integrates a multi-head self-attention mechanism and a feedforward neural network to effectively capture the global dependencies in the image. In the initial stage, the image is subjected to preliminary feature extraction through L1 Transformer blocks, followed by downsampling, which effectively reduces the data dimension while extracting key features. These downsampled feature maps are further subjected to L2 Transformer blocks for deeper feature extraction and more refined downsampling, which significantly enhances the model’s ability to understand the image content. S3.2, Image decoding and reconstruction: The decoder module performs high-level feature extraction through L2 Transformer blocks, then goes through an upsampling process, and then enters L1 Transformer blocks for further refinement. In the downsampling stage of the entire model, a feature alignment and matching module is innovatively added between the encoder module and the decoder module. Through residual connection, the output of the encoder module is concatenated with the output of the decoder module to fuse local features and aligned features; the recursive numbers of Transformer blocks L1, L2, and L3 for the best deraining performance are set to 2, 3, and 3 respectively; S3.3, Feature alignment and matching module: A two-stage matching strategy is adopted, combining the reference image-based method with the feature alignment and acceleration matching strategy, aiming to overcome the problem of image detail loss caused by the traditional rain removal algorithm that only considers single image rain removal; the feature alignment and matching module can locate and extract feature blocks that play a key role in rain removal, thereby optimizing the rain removal effect; the rain image features are expanded into multiple non-overlapping blocks, and the most similar reference block is found for each block in the rain image; in order to improve the accuracy, local block matching is further performed in each pair of rain blocks and reference blocks; finally, useful reference features are extracted based on the corresponding information obtained.

6. The method for removing rain from traffic images based on reference images according to claim 5, characterized in that: In S3.3, the method of expanding the rain image features into multiple non-overlapping blocks and finding the most similar reference block for each block in the rain image is: S3.3.

1. In the first stage, the rain image features are first expanded into N non-overlapping blocks, each of which consists of multiple w×h local blocks; for each rain block, the most relevant reference block is found; first, the central local block of the rain block is taken out, and the cosine similarity is calculated with each local block in the reference block; according to the similarity score, the local block in the reference image feature that is most similar to the central local block of the rain block is found; Crop a block of size w×h from the surrounding area as the most relevant reference block; After the first stage of matching, each rain block obtains the most relevant reference block; S3.3.2, in the second stage, local block matching is performed on each rain block and the reference block to obtain their index map and similarity map; similarity scores are calculated between each local block in the rain block and each local block in the reference block; relevant blocks are extracted according to the index map, and these blocks are multiplied with the corresponding similarity map to obtain weighted features; Finally, these N features are merged to obtain the reference features.

7. The method for removing rain from traffic images based on reference images according to claim 1, characterized in that: The S3 also includes using the peak signal-to-noise ratio PSNR and the structural similarity index SSIM to evaluate the quality of the reconstruction result; the larger the PSNR, the better the image quality; SSIM is a number between 0 and 1, and the larger the SSIM, the smaller the difference between the output image and the undistorted image, that is, the better the image quality.

Images