A complex weather scene target detection model and method based on a large visual language model
By introducing a visual language model and an adaptive feature restoration module into the object detection model, combined with the combined image feature consistency loss, the problem of reduced target detection accuracy in complex weather scenarios is solved, and the detection performance and generalization ability are improved.
Patent Information
- Application Number
- CN202411669573.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-04-04
AI Technical Summary
The existing target detection methods perform poorly in complex weather scenarios, and have poor generalization ability and practicality. Especially in rainy, foggy, snowy and other conditions, the image deteriorates seriously, resulting in blurred target edges and reduced detection accuracy.
The complex weather scene object detection model based on the visual language model is adopted. Through the image prior and adaptive feature restoration module provided by the visual language model, the conversion from text prior to restoration features is realized, and the detection performance is improved. At the same time, joint image feature consistency loss is introduced to enhance the feature quality and attention level of the target area.
It improves the performance of target detection in complex weather scenarios, enhances the generalization ability and practicality of the detector, and can more effectively deal with complex weather scenarios of different types and degrees.
Smart Images
Figure CN119762840A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to target detection problems in the field of deep learning, and in particular to a target detection model and method for complex weather scenes based on a large visual language model. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.
[0003] Object detection plays an extremely important role in the field of computer vision. It is not only the basis of many visual tasks, but also a key bridge between academic research and industrial applications. It is widely used in various fields including manufacturing, agriculture, healthcare, monitoring and security, traffic control, and self-driving cars. In recent years, the emergence of deep convolutional neural networks has promoted significant progress in object detection technology, and a series of algorithms with good performance have emerged, such as YOLO, Faster RCNN, and SSD Retina-Net. However, due to the interference of weather conditions (such as rain, fog, snow, low light, etc.), the captured images are severely degraded and the edges of the targets are blurred, resulting in reduced accuracy of general object detection methods and unsatisfactory application results.
[0004] To solve this problem, many scholars have conducted research on it. Object detection methods in complex weather environments can be divided into three categories according to the way they process degraded images:
[0005] The first type of method directly uses degraded images to train target detection models, such as . However, the blur and occlusion in degraded images pose challenges to the detector in learning key features, which inevitably leads to performance degradation;
[0006] The second type of method first restores the clarity and texture information of the image through existing image restoration algorithms (AODNet, GrideHazeNet, MIRNet), and then performs target detection on the restored image. Although the restored images have better visual quality, they may not bring the same benefits to the target detection task. Many studies have shown that there is no obvious correlation between the visual quality of the image and the accuracy of target detection. Since the ultimate goals of the image restoration task and the detection task are different, there may be conflicts between the two. In addition, the images generated by the image restoration task contain noise and may lose key information, which may reduce the performance of the subsequent detection model;
[0007] The third method is to integrate image restoration into the target detection network model, and use the optimization goal of the detection task to guide the image restoration module to restore key features that are beneficial to the detection task, such as IA-YOLO, DH-YOLO, DSNet, etc. This type of method often requires separate modeling for different weather scenes, and the model generalization ability and practicality are poor. Summary of the invention
[0008] The purpose of the present invention is to: address the problems of poor generalization ability and practicality of existing detection methods, and provide a complex weather scene target detection model and method based on a large visual language model, which utilizes the rich priors of the large visual language model to identify the degradation priors of the image to be detected, adaptively restores image features to guide target detection, and improves detector performance; at the same time, introduces a joint image feature consistency loss, so that the restoration module pays more attention to the target area, further improving the performance of the detection method.
[0009] The technical solution of the present invention is as follows:
[0010] A complex weather scene target detection model based on a large visual language model. The front end of the target detector YOLOv5 introduces the large visual language model to provide image priors. The back end of the backbone network head of the target detector YOLOv5 introduces an adaptive feature restoration module to realize the conversion from text priors to restored features. The restored features are then merged into the target detection backbone network.
[0011] Furthermore, the visual language large model outputs a text description of the degradation of the image to be detected, that is, a text prior, based on the image to be detected and the Prompt instruction; the text description includes: the content, type and severity of the degradation of the image to be detected.
[0012] Furthermore, the backbone network head extracts shallow image features of the image to be detected.
[0013] Furthermore, the adaptive feature restoration module comprises:
[0014] Image-text alignment module, 2D degradation quantization module, feature restoration module and feature fusion module;
[0015] The image-text alignment module completes the alignment of the text prior output by the visual language model with the image features, and converts the text prior into a one-dimensional degraded prior in the image space;
[0016] The 2D degradation quantization module realizes the conversion of the one-dimensional degradation prior to the two-dimensional degradation map, and obtains the degradation of the image at each pixel position;
[0017] The feature restoration module uses a two-dimensional degradation map to restore the shallow features of the image extracted by the backbone network head, and the restored features bring benefits to the subsequent target detection task;
[0018] The feature fusion module performs feature fusion on the restored features and the shallow features of the image, and after fusion, the features are fed into the backbone network to jointly participate in subsequent target detection work.
[0019] Furthermore, during the training process, the adaptive feature restoration module uses detector target loss and joint image feature consistency loss back propagation to update parameters, thereby restoring features that can bring gains to target detection.
[0020] Furthermore, the feature restoration module includes:
[0021] A convolutional layer for fusing channel information, a residual layer consisting of two consecutive residual blocks, a fully connected layer, and a cross-attention operator.
[0022] Furthermore, the joint image feature consistency loss is used to constrain the similarity between the restored features and the original features. The target detection loss and the joint image feature consistency loss are used to constrain the adaptive feature restoration module, thereby enhancing the feature quality and attention level of the target area.
[0023] Furthermore, the residual layer increases a certain network depth; the cross-attention mechanism introduces additional cross-weights to measure the interactions between different features, and adjusts the weight distribution of the input features according to these weights.
[0024] Furthermore, the image-text alignment module is implemented using a multi-layer perceptron.
[0025] The present invention also proposes a complex weather scene target detection method based on a visual language large model, based on the above-mentioned complex weather scene target detection model based on a visual language large model, comprising:
[0026] Step S1: The image to be detected I d The Prompt command is sent to the visual language model through the interface call, and the visual language model outputs the image to be detected I d Degenerate text description P txt ;
[0027] Step S2: Describe the text P txt Use the image-text alignment module to transform it from text space to image space and obtain the degraded prior P in the image feature space. dp ;
[0028] Step S3: The image to be detected I dPass it into the backbone network head to extract the shallow features X of the image;
[0029] Step S4: Combine the shallow features X of the image with the degradation prior P dp Input to the 2D degradation quantification module to obtain the two-dimensional degradation graph I DM ;
[0030] Step S5: Using the two-dimensional degradation graph I DM Perform feature restoration on the shallow feature X of the image with degraded characteristics to obtain the restored feature;
[0031] Step S6: Fusing the shallow features X of the image with the restored features;
[0032] Step S7: Input the fused features into the subsequent network of the detection model to finally obtain the detection result.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] 1. Aiming at target detection in complex weather scenes, the present invention proposes to introduce a large visual language model prior into the current target detection network to achieve target detection in weather scenes of different types and degrees of complexity.
[0035] 2. In order to solve the problem of feature differences between language priors and images, the present invention proposes an adaptive feature restoration module, which fully considers the correlation between text priors and image features, adaptively restores features of the image to be detected according to detection needs, and improves detection performance.
[0036] 3. The present invention introduces the joint image feature consistency loss, which enhances the feature quality and attention level of the target area and further improves the detection performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a framework diagram of the target detection model for complex weather scenes based on a large visual language model;
[0038] Figure 2 This is the network structure diagram of the adaptive feature restoration module. DETAILED DESCRIPTION
[0039] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0040] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.
[0041] Embodiment 1
[0042] See also Figure 1 , a complex weather scene target detection model based on a large visual language model, which can handle target detection in different types and degrees of complex weather environments, including:
[0043] The front end of the target detector YOLOv5 introduces a large visual language model to provide image priors. The back end of the backbone network head of the target detector YOLOv5 introduces an adaptive feature restoration module to realize the conversion from text priors to restored features. The restored features are then merged into the target detection backbone network.
[0044] In this embodiment, specifically, the visual language large model outputs a text description of the degradation of the image to be detected, that is, a text prior, based on the image to be detected and the Prompt instruction; the text description includes: the content, type and severity of the degradation of the image to be detected.
[0045] In this embodiment, specifically, the backbone network head extracts shallow image features of the image to be detected.
[0046] In this embodiment, specifically, Figure 2 As shown, the adaptive feature restoration module includes:
[0047] Image-text alignment module, 2D degradation quantization module, feature restoration module and feature fusion module;
[0048] The image-text alignment module completes the alignment of the text prior output by the visual language model with the image features, and converts the text prior into a one-dimensional degraded prior in the image space;
[0049] The 2D degradation quantization module realizes the conversion of the one-dimensional degradation prior to the two-dimensional degradation map, and obtains the degradation of the image at each pixel position;
[0050] The feature restoration module uses a two-dimensional degradation map to restore the shallow features of the image extracted by the backbone network head, and the restored features bring benefits to the subsequent target detection task;
[0051] The feature fusion module performs feature fusion on the restored features and the shallow features of the image, and after fusion, the features are fed into the backbone network to jointly participate in subsequent target detection work.
[0052] In this embodiment, specifically, during the training process, the adaptive feature restoration module uses the detector target loss and the joint image feature consistency (image consistency, IC) loss back propagation to update parameters, thereby restoring the restored features that can bring gains to target detection and improve the performance of the detector; that is, the joint image feature consistency loss is introduced to constrain the similarity between the restored features and the original features, and the target detection loss and the joint image feature consistency loss are used to constrain the adaptive feature restoration module, thereby enhancing the feature quality and attention level of the target area, thereby improving the performance of the detector;
[0053] In this embodiment, it should be noted that the joint image feature consistency loss L IC The formula is shown in formula 1, where represents the restored eigenvalue of the i-th channel with spatial position (j, k), F i，j，k represents the original eigenvalue of the spatial position (j, k) of the i-th channel;
[0054] The total loss of the target detection network model is shown in Formula 2, where L cls , L box , L obj They are classification loss, positioning loss and confidence loss, respectively, cls , box , obj , IC L cls , L box , L obj , L IC The weight of
[0055]
[0056] L total =λ cls L cls +λ box L box +λ obj L obj +λIC *L IC (2)
[0057] In this embodiment, specifically, the feature restoration module is based on a cross attention mechanism, and specifically includes:
[0058] A convolutional layer for fusing channel information, a residual layer consisting of two consecutive residual blocks, a fully connected layer, and a cross-attention operator.
[0059] In this embodiment, specifically, the joint image feature consistency loss is used to constrain the similarity between the restored features and the original features, and the target detection loss and the joint image feature consistency loss are used to constrain the adaptive feature restoration module, thereby enhancing the feature quality and attention level of the target area.
[0060] In this embodiment, specifically, the residual layer increases a certain network depth to improve network performance, while preventing the gradient from disappearing due to the excessive depth of the YOLOv5 network layer; the cross-attention mechanism introduces additional cross-weights to measure the interaction between different features, and adjusts the weight distribution of the input features according to these weights, thereby improving the expressiveness and performance of the model.
[0061] In this embodiment, specifically, the image text alignment module is implemented using a multilayer perceptron (MLP).
[0062] This embodiment also proposes a complex weather scene target detection method based on a visual language large model, based on the above-mentioned complex weather scene target detection model based on a visual language large model, specifically including the following steps:
[0063] Step S1: The image to be detected I d The Prompt command is sent to the visual language model CLIP through the interface call, and the visual language model outputs the image to be detected I d Degenerate text description P txt , this strategy lays a solid foundation for the model to handle images in different weather conditions and degradation situations; as shown in Formula 3:
[0064]
[0065] Step S2: Describe the text P txt Use the image-text alignment module to transform it from text space to image space and obtain the degraded prior p in the image feature space. dp , see formula 4 for details;
[0066]
[0067] Step S3: The image to be detected I d Pass it into the backbone network head to extract the shallow features X of the image;
[0068] Step S4: Combine the shallow features X of the image with the degradation prior P dp Input to the 2D degradation quantification module to obtain the two-dimensional degradation graph I DM ; The two-dimensional degenerate graph I DM It provides a representation of the degradation content and degree of degradation of the degraded image; in order to obtain the degradation of the image at each pixel position, the cross-attention mechanism is used to convert the degradation prior P dp Convert to 2D degradation graph I DM , where W q , W v , W k They are all linear transformation matrices. For specific operations, see Formula 5 and Formula 6.
[0069] Q=XW q K=P dp W k V=P dp W v (5)
[0070] I DM =Softmax(QK T )V (6)
[0071] Step S5: Using the two-dimensional degradation graph I DM Perform feature restoration on the shallow feature X of the image with degraded characteristics to obtain the restored feature;
[0072] Step S6: Fusing the shallow features X of the image with the restored features;
[0073] Step S7: Input the fused features into the subsequent network of the detection model to finally obtain the detection result.
[0074] In this embodiment, it should also be noted that the data set preparation in the training process of a complex weather scene target detection model based on a large visual language model proposed in this embodiment includes:
[0075] The BD100K dataset is selected as the detection image dataset under complex weather conditions. This detection image dataset is a large-scale autonomous driving scene dataset, which contains images and videos of various weather conditions such as rain, fog, and dust; the annotated categories include pedestrians, bicycles, cars, buses, trucks, motorcycles, stop signs, stop lines, and traffic lights. The BD100K data is divided into training sets, validation sets, and test sets in a ratio of 6:2:2; and data enhancement is performed using methods such as rotation, mirroring, color space transformation, Mosaic, and Mixup.
[0076] At the same time, a Prompt instruction set was constructed by arranging and combining other auxiliary words such as weather type, intensity, and blocked area. It contains 20 instructions, such as "Please describe the weather conditions in the picture and point out its intensity and invisible parts."
[0077] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.
[0078] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.
Claims
1. A complex weather scene target detection model based on a large visual language model, characterized in that: The front end of the target detector YOLOv5 introduces a large visual language model to provide image priors. The back end of the backbone network head of the target detector YOLOv5 introduces an adaptive feature restoration module to realize the conversion from text priors to restored features. The restored features are then merged into the target detection backbone network.
2. According to claim 1, a complex weather scene target detection model based on a large visual language model is characterized in that: The visual language large model outputs a text description of the degradation of the image to be detected, that is, a text prior, based on the image to be detected and the Prompt instruction; the text description includes: the content, type and severity of the degradation of the image to be detected.
3. The complex weather scene target detection model based on the visual language large model according to claim 2 is characterized in that: The backbone network head extracts shallow image features of the image to be detected.
4. The complex weather scene target detection model based on the visual language large model according to claim 3 is characterized in that: The adaptive feature restoration module comprises: Image-text alignment module, 2D degradation quantization module, feature restoration module and feature fusion module; The image-text alignment module completes the alignment of the text prior output by the visual language model with the image features, and converts the text prior into a one-dimensional degraded prior in the image space; The 2D degradation quantization module realizes the conversion of the one-dimensional degradation prior to the two-dimensional degradation map, and obtains the degradation of the image at each pixel position; The feature restoration module uses a two-dimensional degradation map to restore the shallow features of the image extracted by the backbone network head, and the restored features bring benefits to the subsequent target detection task; The feature fusion module performs feature fusion on the restored features and the shallow features of the image, and after fusion, the features are fed into the backbone network to jointly participate in subsequent target detection work.
5. The complex weather scene target detection model based on the visual language large model according to claim 4 is characterized in that: During the training process, the adaptive feature restoration module uses the detector target loss and the joint image feature consistency loss back propagation to update parameters, thereby restoring features that can bring gains to target detection.
6. The complex weather scene target detection model based on the visual language large model according to claim 5 is characterized in that: The feature restoration module comprises: A convolutional layer for fusing channel information, a residual layer consisting of two consecutive residual blocks, a fully connected layer, and a cross-attention operator.
7. The complex weather scene target detection model based on the visual language large model according to claim 6 is characterized in that: The joint image feature consistency loss is used to constrain the similarity between the restored features and the original features. The target detection loss and the joint image feature consistency loss are used to constrain the adaptive feature restoration module, thereby enhancing the feature quality and attention level of the target area.
8. The complex weather scene target detection model based on a large visual language model according to claim 6 is characterized in that: The residual layer increases the depth of the network; the cross-attention mechanism introduces additional cross-weights to weigh the interactions between different features and adjusts the weight distribution of the input features according to these weights.
9. The complex weather scene target detection model based on a large visual language model according to claim 4, characterized in that: The image-text alignment module is implemented using a multi-layer perceptron.
10. A method for target detection in complex weather scenes based on a large visual language model, characterized in that: A complex weather scene target detection model based on a large visual language model according to any one of claims 1 to 9, comprising: Step S1: The image to be detected I d The Prompt command is sent to the visual language model through the interface call, and the visual language model outputs the image to be detected I d Degenerate text description P txt ; Step S2: Describe the text P txt Use the image-text alignment module to transform it from text space to image space and obtain the degraded prior P in the image feature space. dp ; Step S3: The image to be detected I d Pass it into the backbone network head to extract the shallow features X of the image; Step S4: Combine the shallow features X of the image with the degradation prior P dp Input to the 2D degradation quantification module to obtain the two-dimensional degradation graph I DM ; Step S5: Using the two-dimensional degradation graph I DM Perform feature restoration on the shallow feature X of the image with degraded characteristics to obtain the restored feature; Step S6: Fusing the shallow features X of the image with the restored features; Step S7: Input the fused features into the subsequent network of the detection model to finally obtain the detection result.