A method for adverse weather target detection based on contrast learning

Abstract

The application discloses a kind of adverse weather target detection methods based on contrast learning, belong to computer vision technical field, which comprises the following steps: S1, scene adaptation;S2, target detection;S3, domain generalization target detection;S4, model training;S5, inference and result output.The application uses the above-mentioned adverse weather target detection method based on contrast learning, with the help of multi-module cooperation and double-mode design, both reduces the detail loss caused by traditional image enhancement, significantly improves the target detection accuracy under known adverse weather;Break through single domain limitation, realize the adaptation to a variety of scenes, while meeting real-time requirements.

Description

A Target Detection Method Based on Contrastive Learning in Severe Weather Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for target detection in severe weather based on contrastive learning. Background Technology

[0002] As technologies such as autonomous driving and intelligent monitoring accelerate their commercialization, target detection systems have become a core component supporting their safe operation. In autonomous driving scenarios, the system needs to identify targets such as vehicles, pedestrians, and traffic signs in real time to provide decision-making basis for path planning and emergency braking; in the field of intelligent monitoring, target detection is a key technology for realizing abnormal behavior warnings and security in key areas.

[0003] However, existing technologies have the following shortcomings: they are prone to losing key detection details such as target edges and textures; when faced with various severe weather conditions such as heavy rain, heavy snow, and dense fog, the cross-domain generalization ability is poor due to the complex differences in the distribution between the source domain and the multi-target domain; auxiliary domains such as text and infrared images are used to improve generalization ability, but the auxiliary domains will change the feature space of the source domain, resulting in a decrease in detection performance in source domain scenes such as sunny days, and the generalization ability and source domain accuracy are not balanced.

[0004] Therefore, there is an urgent need in this field to develop a target detection method based on contrastive learning in severe weather that can effectively solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a target detection method based on contrastive learning in severe weather. This method, through multi-module collaboration and dual-mode design, not only reduces the loss of details caused by traditional image enhancement and significantly improves the target detection accuracy under known severe weather conditions, but also breaks through the limitations of single domains, achieves adaptation to multiple scenarios, and meets real-time requirements.

[0006] To achieve the above objectives, this invention provides a target detection method based on contrastive learning in severe weather, comprising the following steps:

[0007] S1. Collect target image data containing various severe weather conditions, and label the targets in the images to obtain known severe weather images;

[0008] S2. Input the known severe weather images from S1 into the multi-scale weather degradation encoder and the similarity attention feature suppression module respectively to obtain weather degradation features. and enhanced target features ;

[0009] The multi-scale weather degradation encoder is built based on Darknet-53 and FPN. It uses a 3D attention mechanism to weight and fuse the output features of each scale branch through the output features of the three scale branches. The calculation formula is as follows:

[0010] ;

[0011] ;

[0012] in, Characteristics of weather degradation; Attention weights; For the first Weather degradation feature maps extracted from each scale branch; For the first Feature maps of each scale branch Perform an upsampling operation; The height of the input image; The width of the input image; Index for scale branches;

[0013] The similarity attention feature suppression module calculates the similarity between target-related features and weather degradation features based on cosine similarity, generates inverse similarity weights to weight and fuse target-related features; the enhanced target features are input into the YOLOv8s detection head to complete target classification and bounding box regression, and outputs the target category and bounding box coordinates;

[0014] S3. Construct a dual-branch alignment model based on causal reasoning, wherein the model includes a causal attention feature alignment module and a dual-detection branch consistency alignment module;

[0015] The known severe weather images and preset severe weather description text in S1 are input into the causal attention feature alignment module, and the severe weather description text is encoded using the CLIP text encoder to generate text features. , and These correspond to the source domain, target domain, and enhanced text features, respectively; the CLIP image encoder is used to extract image features from images of unknown severe weather. Based on L2 loss and , and Feature alignment is performed to obtain the aligned image features.

[0016] Main branch reception of dual-detection branch consistency alignment module And the aligned image features, detected by the head Output the first prediction box Compared with the first category probability ; Detection head Output the second prediction box With the second category probability Auxiliary branch reception as well as The fusion generates fusion features; KL divergence constraints are used. and The class probability distribution is used with L1 loss constraint. and The location of the bounding box;

[0017] S4. Construct training data using synthetic and real datasets, and divide the data into training, validation, and test sets in a 7:1.5:1.5 ratio; construct the total loss function by fusing contrastive learning loss, CAAM loss, DCAM loss, classification loss, and regression loss; the classification loss is cross-entropy loss, and the regression loss is GIoU loss; train the model using the Adam optimizer to obtain the trained model;

[0018] S5. Resize the acquired images to 640×640 for preprocessing. Based on the model trained in S4, execute the output of S2 in general mode. Input detection head to get and In domain generalization mode, S3 generation is performed first. Input two branches through Constrain output consistency, select and Predicted bounding boxes with a confidence level > 0.5 are output after non-maximum suppression. Simultaneously output detection results and real-time performance indicators.

[0019] Preferably, the output features in S2 through three scale branches are as follows:

[0020] The first branch extracts fine-grained interference features through a 3×3 convolution with a stride of 1, and outputs a feature map. ;in The height of the input image. The width of the input image;

[0021] The second branch extracts medium-scale interference features through a 3×3 convolution with a stride of 2 combined with batch normalization, and outputs... ;

[0022] The third branch extracts large-scale weather occlusion features using a 3×3 convolution with a stride of 4 combined with ReLU activation, and outputs... .

[0023] Preferably, the formula for calculating the cosine similarity in S2 is:

[0024] ;

[0025] in, For the first At each scale, the coordinates are Cosine similarity between weather degradation features and target features; For the first In each scale branch, coordinates Weather degradation feature vector at the location; For the first In each scale branch, coordinates Target-related feature vectors at the location; These are the pixel coordinates of the feature map; It is a minute value;

[0026] The formula for weighted fusion of target-related features using inverse similarity weighting is:

[0027] ;

[0028] in, Enhanced target features; This is the inverse similarity weight; This is the original target feature map; Let be the cosine similarity.

[0029] Preferably, the loss function of the causal attention feature alignment module in S3 is expressed as:

[0030] ;

[0031] in, For causal attention feature alignment loss; This is a function that projects a 1×1 convolution onto the image feature dimension. For the source domain; For the target domain; To enhance text features; These are image features.

[0032] Preferably, the formula for fusing text features and enhanced target features in the auxiliary branch of S3 is as follows:

[0033] ;

[0034] in, The fused feature map; The size of the feature map in the enhanced image.

[0035] Preferably, the formula for the total loss function in S4 is:

[0036] ;

[0037] ;

[0038] in, Temperature coefficient; These are positive sample weather characteristics; Features of negative samples; This is the total loss function; To compare learning loss; For classification loss; For regression loss; The weather degradation characteristics are positive samples; For the first Weather degradation characteristics of a negative sample; This represents the number of negative samples.

[0039] Preferably, in S4, the real-time performance metrics include the average accuracy and the detection frame rate; the synthetic dataset contains 5000 labeled images in rainy and foggy weather, FoggyDriving contains 2000 foggy weather images, Urban-Scene contains 1500 snowy weather images, and the training set, validation set, and test set contain 5950, 1275, and 1275 images, respectively.

[0040] Therefore, the present invention employs the aforementioned target detection method based on contrastive learning in severe weather. Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0041] (1) This invention adopts the technical means of “MWDE (Multi-scale Weather Degradation Encoder) + SimAM (Similarity Attention Feature Suppression Module)”, first extracts multi-scale weather degradation features, and then calculates the similarity between them and the target features to dynamically suppress weather interference and retain key detection details. It abandons the traditional path of “enhancing first and then detecting”, thereby overcoming the technical problem of “image enhancement methods losing details due to inconsistency between the enhanced target and the detection target”, and thus achieving the technical effect of avoiding the loss of key detection information and improving the target detection accuracy under known severe weather conditions;

[0042] (2) The technical means of “constructing a domain generalized target detection model + CAAM (causal attention feature alignment module)” in this invention aligns the features of the source domain (clear image) and the auxiliary domain (text description) through the CAAM module, and mines the stable causal relationship between weather and target, breaking through the limitation of “single source domain - single target domain”, thereby overcoming the technical problem of “poor cross-domain generalization ability of transfer learning methods”, and thus achieving the technical effect of adapting to known and unknown severe weather scenarios and improving the generalization ability of the model;

[0043] (3) The technical means of “designing DCAM (dual detection branch consistency alignment module)” in this invention balances the utilization of the auxiliary domain and the performance of the source domain by constraining the output difference between the main branch (processing suppressed features) and the auxiliary branch (fusing text features), thereby overcoming the technical problem that “the introduction of the auxiliary domain by the auxiliary domain-type method leads to a decrease in the performance of the source domain”, and thus achieving the technical effect of improving the generalization ability while ensuring that the detection accuracy of the source domain (such as sunny day) is not lost.

[0044] (4) The present invention adopts the technical means of “dividing general target detection mode and domain generalized target detection mode”, and selects and adapts to different scenarios as needed, thereby overcoming the technical problem that “the existing single detection mode cannot simultaneously meet the requirements of 'high accuracy of known weather' and 'generalization of unknown weather'”, and thus achieving the technical effect of adapting to more practical application scenarios (such as using the general mode for fixed monitoring and the domain generalized mode for complex weather in mountainous areas) and improving the practicality of the technology.

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0046] Figure 1 is a flowchart of an embodiment of the adverse weather target detection method based on contrastive learning according to the present invention;

[0047] Figure 2 is a flowchart of a dual-branch aligned target detection method for severe weather based on contrastive learning, according to an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used in the present invention should have the ordinary meaning understood by those skilled in the art.

[0049] Example 1

[0050] As shown in Figures 1 and 2, the present invention provides a target detection method for severe weather based on contrastive learning, comprising the following steps:

[0051] S1. For scenarios such as real-time detection of vehicles and pedestrians required for autonomous driving and stable recognition of traffic signs required for intelligent monitoring, two core requirements are defined: high-precision detection in known adverse weather conditions and generalized adaptation to unknown adverse weather conditions.

[0052] The input for known severe weather detection is defined as weather type label and corresponding image, and the output is target category and bounding box coordinates. The core metric is the mean average accuracy (mAP) under known weather scenarios. The input for unknown severe weather detection is unknown severe weather image without weather type label, and the output is target category and bounding box coordinates. The core metrics are mAP under unknown weather scenarios and the mAP decay rate of the source region under clear weather. The known severe weather image is obtained.

[0053] Fixed monitoring systems, such as those at urban intersections, can adapt to general modes because weather conditions can be predicted; mobile platforms, such as drones and autonomous vehicles, adapt to domain-generalized modes because weather conditions are variable.

[0054] S2. Input the known severe weather images from S1 into the multi-scale weather degradation encoder (MWDE). The multi-scale weather degradation encoder is built based on Darknet-53+FPN (feature pyramid network) and is divided into 3 scale branches to extract weather degradation features under 1×1, 3×3 and 5×5 receptive fields respectively.

[0055] The first branch extracts fine-grained interference features such as raindrops / snowflakes through a 3×3 convolution with a stride of 1, and outputs a feature map. ,in, Input image height / width;

[0056] The second branch extracts medium-scale interference features such as fog by using a 3×3 convolution with a stride of 2 combined with batch normalization, and outputs... ;

[0057] The third branch extracts large-scale weather occlusion features through a 3×3 convolution with a stride of 4 combined with ReLU activation, and outputs... ;

[0058] Using 3D attention mechanism , , Weighted fusion generates the final weather degradation characteristics. The formula is expressed as:

[0059] ;

[0060] ;

[0061] in, Characteristics of weather degradation; Attention weights; For the first Weather degradation feature maps extracted from each scale branch; For the first Feature maps of each scale branch Perform an upsampling operation; The height of the input image; The width of the input image; Index for scale branches;

[0062] The Similarity Attention Feature Suppression (SimAM) module uses a YOLOv8s backbone network to extract target-related features from the input image. ;

[0063] Based on cosine similarity, the similarity between weather features and target features at each location is calculated, expressed by the formula:

[0064] ;

[0065] in, For the first At each scale, the coordinates are Cosine similarity between weather degradation features and target features; For the first In each scale branch, coordinates Weather degradation feature vector at the location; For the first In each scale branch, coordinates Target-related feature vectors at the location; These are the pixel coordinates of the feature map; To prevent tiny values ​​with a denominator of 0;

[0066] Generate inverse similarity weights The target features are weighted and fused, retaining target details with low similarity. The calculation formula is as follows:

[0067] ;

[0068] in, Enhanced target features; This is the inverse similarity weight; This is the original target feature map; Cosine similarity;

[0069] Enhanced target features Input the YOLOv8s detector head to complete target classification and bounding box regression, and output the target category and bounding box coordinates;

[0070] S3. For the domain-generalized object detection mode, a two-branch alignment model based on causal reasoning is constructed. The core of this model includes the Causal Attention Feature Alignment Module (CAAM) and the Dual Detection Branch Consistency Alignment Module (DCAM).

[0071] Input the known severe weather images and preset severe weather description text from S1 into CAAM, and use the CLIP text encoder to encode the severe weather description text to generate text features. , and These correspond to the source domain, target domain, and enhanced text features, respectively, with each dimension being [missing information]. ;

[0072] CLIP image encoder Extracting image features Established through a "semantic bridge generator" and , and The connection;

[0073] Attention alignment is performed on image features and text features to uncover stable causal relationships between "weather and target," achieving alignment between source domain and auxiliary domain (text) features. The loss function formula is as follows:

[0074] ;

[0075] in, For causal attention feature alignment loss; This is a function that projects a 1×1 convolution onto the image feature dimension. For the source domain; For the target domain; To enhance text features; Image features;

[0076] Main branch reception of dual-detection branch consistency alignment module And the aligned image features, detected by the head Output the first prediction box Compared with the first category probability ; Detection head Output the second prediction box With the second category probability Text features are fused through auxiliary branches. and The formula is:

[0077] ;

[0078] in, The fused feature map; To increase the size of the feature map in the enhanced image;

[0079] Will Input the second detection head Output the second prediction box With the second category probability ;

[0080] Using KL divergence constraints and The category probability distributions are consistent, as shown in the following formula:

[0081] ;

[0082] in, This is the loss for class consistency. The probability distribution of the main branch's category prediction; Predict the probability distribution of the categories for auxiliary branches;

[0083] Using L1 loss constraints and The bounding box positions are consistent, as shown in the following formula:

[0084] ;

[0085] in, For regression consistency loss; This represents the number of predicted boxes; The main branch is the first The bounding box prediction results for each target; For auxiliary branches to the first The bounding box prediction results for each target;

[0086] The total consistency loss is calculated as follows:

[0087] ;

[0088] in, This represents the consistency alignment loss for dual-detection branches; , These are the weighting coefficients. ;

[0089] S4. Data sources include the synthetic datasets VOC-Fog and VOC-Rain, and the real datasets FoggyDriving and Urban-Scene. The synthetic datasets contain 5000 annotated images taken in rainy or foggy weather, FoggyDriving contains 2000 images taken in foggy weather, and Urban-Scene contains 1500 images taken in snowy weather. The annotation standard uses the PASCALVOC format, and the target categories (motor vehicles, non-motor vehicles, pedestrians, traffic signs) are annotated along with their bounding box coordinates. , , , After merging all datasets (a total of 8500 images), the data was randomly divided into a training set of 5950 images, a validation set of 1275 images, and a test set of 1275 images in a ratio of 7:1.5:1.5. The training set was augmented with random flipping and a brightness perturbation of ±15%.

[0090] The formula for fusing contrastive learning loss (optimized MWDE), consistency alignment loss (optimized DCAM), classification loss, and regression loss is as follows:

[0091] ;

[0092] ;

[0093] in, For temperature coefficient, ; These are positive sample weather characteristics; Features of negative samples; This is the total loss function; To compare learning loss; For causal attention feature alignment loss; For classification loss; For regression loss; The weather degradation characteristics are positive samples; For the first Weather degradation characteristics of a negative sample; This represents the number of negative samples.

[0094] The classification loss is expressed as:

[0095] ;

[0096] in, This represents the true label distribution; This represents the predicted probability distribution for the target category;

[0097] The regression loss is expressed as:

[0098] ;

[0099] in, The true bounding box; The predicted bounding box for the target;

[0100] The model is trained using the Adam optimizer, with an initial learning rate of... The learning rate decay strategy is "decay to the first 1 / 10 every 10 epochs", and the batch size is 16; thus, the trained model is obtained.

[0101] S5. Based on the model trained in S4, when the input scene information is "fixed intersection monitoring + historical weather data only rain and fog", the general mode is triggered. When the input scene information is "mountain autonomous driving + no historical weather data", the domain generalization mode is triggered.

[0102] During inference, images captured by vehicle-mounted cameras or surveillance cameras are uniformly resized to 640×640 before being input; in general mode, S2 output is executed. Input detection head to get and Domain generalization mode first executes S3 generation. Input two branches through Constrain output consistency, select and Predicted bounding boxes with a confidence level > 0.5 are output after non-maximum suppression (NMS, IoU threshold 0.45). It synchronously outputs detection results (target category, bounding box) and real-time performance metrics.

[0103] Therefore, the present invention adopts the above-mentioned target detection method based on contrastive learning in severe weather. This method, with the help of multi-module collaboration and dual-mode design, not only reduces the loss of details caused by traditional image enhancement and significantly improves the target detection accuracy under known severe weather conditions, but also breaks through the limitations of single domain, achieves adaptation to multiple scenarios, and meets real-time requirements.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A target detection method based on contrastive learning in severe weather, characterized in that, Includes the following steps: S1. Acquire target image data containing various severe weather conditions and label the targets in the images to obtain known severe weather images; S2. Input the known severe weather images from S1 into a multi-scale weather degradation encoder and a similarity attention feature suppression module to obtain weather degradation features. and enhanced target features The multi-scale weather degradation encoder is built based on Darknet-53 and FPN. It uses a 3D attention mechanism to weight and fuse the output features of each of the three scale branches. The calculation formula is as follows: ； ;in, Characteristics of weather degradation; Attention weights; For the first Weather degradation feature maps extracted from each scale branch; For the first Feature maps of each scale branch Perform an upsampling operation; The height of the input image; The width of the input image; The index is used for the scale branch; the similarity attention feature suppression module calculates the similarity between target-related features and weather degradation features based on cosine similarity, and generates inverse similarity weights to weight and fuse target-related features; the enhanced target features are input into the YOLOv8s detection head to complete target classification and bounding box regression, and output the target category and bounding box coordinates; S3, construct a two-branch alignment model based on causal reasoning, which includes a causal attention feature alignment module and a two-detection-branch consistency alignment module; the known severe weather images and preset severe weather description text from S1 are input into the causal attention feature alignment module, and the severe weather description text is encoded using the CLIP text encoder to generate text features. 、 and These correspond to the source domain, target domain, and enhanced text features, respectively; the CLIP image encoder is used to extract image features from images of unknown severe weather. Based on L2 loss and 、 and Feature alignment is performed to obtain aligned image features; the main branch of the dual-detection branch consistency alignment module receives... And the aligned image features, detected by the head Output the first prediction box Compared with the first category probability ; Detection head Output the second prediction box With the second category probability Auxiliary branch reception as well as The fusion generates fusion features; KL divergence constraints are used. and The class probability distribution is used with L1 loss constraint. and S4. Construct training data using synthetic and real datasets, dividing the data into training, validation, and test sets in a 7:1.5:1.5 ratio; construct a total loss function by fusing contrastive learning loss, CAAM loss, DCAM loss, classification loss, and regression loss; the classification loss is cross-entropy loss, and the regression loss is GIoU loss; train the model using the Adam optimizer to obtain the trained model; S5. Resize the acquired images to 640×640 for preprocessing; based on the model trained in S4, in general mode, execute S2 to output the enhanced target features. The input detector head is used to obtain the final predicted bounding box. With the final category probability In domain generalization mode, S3 is executed first to generate the fused feature map. Input two branches through Constrain output consistency, select and Predicted bounding boxes with a confidence level > 0.5 are output after non-maximum suppression. Simultaneously output detection results and real-time performance indicators.

2. The method for target detection in severe weather based on contrastive learning according to claim 1, characterized in that, S2 outputs features through three scale branches: the first branch extracts fine-grained interference features through a 3×3 convolution with a stride of 1, and outputs a feature map. ； The second branch extracts medium-scale interference features through a 3×3 convolution with a stride of 2 combined with batch normalization, and outputs... The third branch extracts large-scale weather occlusion features using a 3×3 convolution with a stride of 4 combined with ReLU activation, and outputs... 。 3. The method for target detection in severe weather based on contrastive learning according to claim 1, characterized in that, The formula for calculating cosine similarity in S2 is: ;in, For the first At each scale, the coordinates are Cosine similarity between weather degradation features and target features; For the first In each scale branch, coordinates Weather degradation feature vector at the location; For the first In each scale branch, coordinates Target-related feature vectors at the location; These are the pixel coordinates of the feature map; For minute values; the formula for weighted fusion of target-related features by inverse similarity weight is: ;in, Enhanced target features; This is the inverse similarity weight; This is the original target feature map; Let be the cosine similarity.

4. The method for target detection in severe weather based on contrastive learning according to claim 1, characterized in that, The loss function of the causal attention feature alignment module in S3 is expressed as: ;in, For causal attention feature alignment loss; This is a function that projects a 1×1 convolution onto the image feature dimension. For the source domain; For the target domain; To enhance text features; These are image features.

5. The severe weather target detection method based on contrastive learning according to claim 4, characterized in that, The formula for fusing text features and enhanced target features in the auxiliary branch of S3 is as follows: ;in, The fused feature map; To increase the size of the feature map in the enhanced image; To enhance the target feature map The dimensions; 6. The method for target detection in severe weather based on contrastive learning according to claim 5, characterized in that, The formula for the total loss function in S4 is: ； ;in, Temperature coefficient; These are positive sample weather characteristics; Features of negative samples; This is the total loss function; To compare learning loss; For classification loss; For regression loss; This represents the number of negative samples. This represents the consistency alignment loss for dual-detection branches.

7. The method for target detection in severe weather based on contrastive learning according to claim 1, characterized in that, In S4, real-time performance metrics include average accuracy and detection frame rate; the synthetic dataset contains 5000 labeled images in rainy and foggy weather, FoggyDriving contains 2000 foggy weather images, and Urban-Scene contains 1500 snowy weather images. The training set, validation set, and test set contain 5950, 1275, and 1275 images, respectively.

8. A computer device, characterized in that, include: A processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1-7.

9. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1-7.

Images