A smart driving target detection method based on fine-tuning of a large vision-language model

Abstract

The present invention discloses an intelligent driving target detection method based on fine-tuning of a large visual-language model, and aims to solve the problem that the existing target detection method has poor detection accuracy for common classes and unknown classes in intelligent driving scenarios. The technical solution is to construct a target detection system consisting of a main feature extraction module, a dual-branch feature fine-tuning module and a post-processing prediction value fusion module. The dual-branch feature fine-tuning module consists of two dual first branch networks and second branch networks, and each branch network consists of a feature enhancement module, a multimodal decoder module and a head network. The second branch network in the dual-branch feature fine-tuning module is trained using a training set, and the weight parameters of the second branch network are updated with the training iterations during training. The trained target detection system detects the input image to obtain a target detection result. The present invention can significantly improve the detection accuracy of known classes, prevent the problem of knowledge forgetting during the fine-tuning process, and maintain the detection accuracy of the large visual-language model for unknown classes.

Description

Technical Field

[0001] The present invention relates to the field of vision-language multimodal large models and image recognition target detection, and in particular to a target detection method for optimizing the accuracy of known objects and unknown objects of a vision-language large model in intelligent driving scenarios. Background Art

[0002] Object detection for intelligent driving scenarios has always been a key task in achieving high-level autonomous driving. However, complex open-world environments and numerous object categories are key factors that restrict the implementation of object detection methods. Most existing object detection algorithms are based on predefined categories and can only detect a fixed number of categories consistent with the training data. They cannot detect unknown objects beyond the predefined categories. This results in some uncommon unknown objects such as suddenly appearing wild animals and stones in the middle of the road not being detected, which greatly affects the safety of intelligent driving and easily leads to serious traffic accidents such as car accidents.

[0003] To solve the problem of detecting arbitrary categories, the Vision-Language Detection Model was proposed and achieved powerful zero-shot recognition capabilities. The Vision-Language Detection Model uses text or word representations to map categories to text semantic space, and uses large-scale image-text data for training to obtain rich object and scene knowledge, making it a new research hotspot in the field of object detection. The paper "Liu S, Zeng Z, Ren T, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection [J]. arXiv preprint arXiv: 2303.05499, 2023." (GroundingDINO) introduces a method to construct categories in object detection datasets as text descriptions. It is based on the detector in the paper "Zhang H, Li F, Liu S, et al. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection [C] / / The Eleventh International Conference on Learning Representations. 2022." (DINO), adds multiple image-text feature fusions, and is trained on multiple large-scale datasets, achieving powerful zero-sample recognition categories, which can realize diversified object category detection through text vocabulary. For example, GroundingDINO can achieve an accuracy of 52.5mAP (mean Average Precision) without using the MS COCO dataset for training. It currently ranks first in zero-shot recognition capability on this dataset, proving that the algorithm has rich knowledge of objects and scenes and strong zero-shot detection performance.

[0004] However, GroundingDINO is a "generalist" model rather than a "specialist" model. Although it performs well on general detection datasets such as MSCOCO, it performs poorly on intelligent driving datasets and has inaccurate detection of some common types of objects (such as pedestrians, cars, etc.). For intelligent driving scenarios, the detection accuracy of common types of objects is very important. Therefore, further research on model design and training strategies is needed to apply the currently advanced GroundingDINO model to intelligent driving scenarios. How to enhance the GroundingDINO model's detection ability for common types of objects in intelligent driving scenarios, and how to retain the model's ability to perceive unknown objects while learning new knowledge, are the keys to the intelligent driving target detection method's ability to cope with complex driving environments. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the existing large visual-language detection model (such as GroundingDINO) has poor detection accuracy for common classes in intelligent driving scenarios, which leads to the problem that the existing advanced detection technology cannot be applied to intelligent driving scenarios. A method for intelligent driving target detection based on fine-tuning of a large visual-language model is proposed. The method utilizes a dual-branch feature fine-tuning structure design, visual-semantic contrast learning, and prediction result fusion technology without post-processing to improve the detection accuracy of common class objects, and minimizes the problem of knowledge forgetting during the fine-tuning process, and maintains the perception ability of the large visual-language detection model for unknown objects. The dual-branch feature fine-tuning module of the present invention adopts two dual network structures, and retains the unknown class recognition ability of the model by freezing the first network branch, and fine-tuning the second network branch to enable it to quickly learn the known class recognition knowledge; the visual-semantic contrast learning of the present invention inputs the semantic representation center with high confidence in the first network branch into the second network branch, and the second network branch outputs the visual-semantic prediction category calculation loss value to align the known class representation to the visual semantic space; the prediction result fusion module without post-processing of the present invention can fuse the prediction results of the two branches without using the post-processing operation of non-maximum suppression (NMS), reducing the computational overhead while achieving high accuracy.

[0006] The technical solution of the present invention is to construct a target detection system. The system consists of an image preprocessing module, a main feature extraction module, a dual-branch feature fine-tuning module, a post-processing prediction value fusion module and a loss function calculation module. Prepare and construct the data set required for the target detection system, and divide the data set into a training set and a test set. The training set image data is randomly flipped, randomly cropped and normalized by data enhancement technology to enhance the diversity of training data. Only size scaling and normalization are used for the test set to maintain the visual clues of the original image.

[0007] The training set is used to train the second network branch in the dual-branch feature fine-tuning module. In each training iteration, the image preprocessing module enhances the input image and generates the corresponding true value label. The main feature extraction module receives the input training set image data and label text data, uses the visual feature extraction network (SwinTransformer network in GroundingDINO) to extract visual features, and outputs the image features to the dual-branch feature fine-tuning module; uses the text feature extraction network (Bert network in GroundingDINO) to extract text features, and outputs the text features to the dual-branch feature fine-tuning module. The main feature extraction module is weight-frozen, and there is no need to calculate gradients and update parameters. The dual-branch feature fine-tuning module consists of two dual network branches (respectively denoted as the first network branch and the second network branch), each of which consists of a feature enhancement module, a multimodal decoder module, and a head network module. Among them, the weight of the first network branch is frozen, and the network weight parameters are not updated. The second network branch is learnable, and the network weight parameters are updated with training iterations. The first network branch outputs a set of semantic representation centers and sends it to the second network branch. The second network branch outputs the visual-semantic prediction category, the second network branch predicts the object category, and the second network branch predicts the object position. The loss value is calculated with the supervised truth value constructed by the loss function calculation module, and the weight parameters of the second network branch are updated using gradient back propagation. When the training step reaches the preset value, the training is stopped, the network weight parameters of the last step are retained, and the values ​​are assigned to the trainable module in the target detection system (i.e., the second network branch in the dual-branch feature fine-tuning module).

[0008] When detecting the input image to be detected, the target detection system scales and normalizes the input image through the image preprocessing module, extracts visual features through the visual feature extraction network in the main feature extraction module, and inputs the visual features into the first network branch and the second network branch. At the same time, the target detection system performs text feature extraction operations on the input category text, including known category text (categories in the training data set such as Pedestrian, Car, etc.) and unknown category text (categories outside the training data set such as Vehicle, Roadblock, Obstacle, etc.), and extracts text features through the text feature extraction network in the main feature extraction module. The visual features and text features are input into the first network branch to obtain the object category predicted by the first network branch and the object position predicted by the first network branch. The visual features and text features are input into the second network branch to obtain the object category predicted by the second network branch and the object position predicted by the second network branch. The object category predicted by the first network branch, the object position predicted by the first network branch, the object category predicted by the second network branch, and the object position predicted by the second network branch are input into the non-post-processing prediction value fusion module, and the non-post-processing prediction value fusion module aligns the results of the two branch predictions and outputs the object category and the object position.

[0009] The specific technical solution of the present invention comprises the following steps:

[0010] The first step is to build a target detection system. Figure 1 As shown in the figure, the target detection system consists of an image preprocessing module, a main feature extraction module, a dual-branch feature fine-tuning module, a non-post-processing prediction value fusion module and a loss function calculation module. Figure 1 The dotted lines in indicate that the connection relationship only exists during the training of the target detection system.

[0011] The image preprocessing module is connected to the main feature extraction module and the loss function calculation module, receives the input image, performs enhancement processing (including flipping, cropping, scaling, and normalization) during the training of the target detection system, sends the processed image and the corresponding true value label to the main feature extraction module, and sends the true value label to the loss function calculation module. When the target detection system performs target detection on the input image set to be detected, it performs scaling and normalization processing to obtain the processed image, and sends the processed image to the main feature extraction module.

[0012] The main feature extraction module is connected to the dual-branch feature fine-tuning module, and the main feature extraction module consists of a visual feature extraction module and a text feature extraction module. The visual feature extraction module receives the processed image from the image preprocessing module, extracts visual features, and sends the visual features to the dual-branch feature fine-tuning module. The visual feature extraction module is composed of the SwinTransformer neural network in GroundingDINO (see the document "Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 10012-10022." Liu Z, Lin Y et al.'s paper: SwinTransformer). The text feature extraction module extracts text features from the input category text (including known category text (categories in the training data set such as Pedestrian, Car, etc.) and unknown category text (categories outside the training data set such as Vehicle, Roadblock, etc.), and sends the text features to the dual-branch feature fine-tuning module. The text feature extraction module is composed of a Bert neural network (see the document "Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [J]. arXiv preprint arXiv: 1810.04805, 2018." Devlin J, Chang MW et al.'s paper: Bert).

[0013] The dual-branch feature fine-tuning module is connected to the visual feature extraction module, the text feature extraction module and the non-post-processing prediction value fusion module. The dual-branch feature fine-tuning module consists of two network branches with the same structure, namely the first network branch and the second network branch. The weight parameters of the first network branch are not updated, and the weight parameters of the second network branch are updated by gradient.

[0014] The first network branch is connected to the visual feature extraction module, the text feature extraction module and the no-post-processing prediction value fusion module, and is composed of a first feature enhancement module, a first multimodal decoder module and a first head network module.

[0015] The first feature enhancement module is connected to the visual feature extraction module, the text feature extraction module, and the first multimodal decoder module, and is composed of an image-text cross-attention network and a text-image cross-attention network. The text-image cross-attention network of the first feature enhancement module receives text features from the text feature extraction module, receives visual features from the visual feature extraction module, uses the text-image cross-module attention mechanism to align text features, obtains feature-enhanced text features, and sends the feature-enhanced text features to the first multimodal decoder module; the image-text cross-attention network of the first feature enhancement module uses the image-text cross-modal attention mechanism to align image features, obtains feature-enhanced visual features, and sends the feature-enhanced visual features to the first multimodal decoder module.

[0016] The first multimodal decoder module is connected to the first feature enhancement module and the first head network module, and is composed of an image query attention network and a text query attention network, and receives feature-enhanced visual features and feature-enhanced text features from the first feature enhancement module. The image query attention network of the first multimodal decoder module defines a fixed number N query (W query =300) as the query vector set, defining a size B×N query ×4 as the initial anchor box position of the query vector set, and use the image query attention mechanism to use the feature-enhanced visual features to give the query vector set visual information, so as to obtain the query vector set with visual information enhancement, and send the query vector set with visual information enhancement to the first head network module; the text query attention network of the first multimodal decoder module uses the text query attention mechanism to use the feature-enhanced text features to give the query vector set with image information enhancement text information, so as to obtain the query vector set with image-text information enhancement, and send the query vector set with image-text information enhancement to the first head network module.

[0017] The first head network module is connected to the first feature enhancement module, the first multimodal decoder module, and the non-post-processing prediction value fusion module, and is composed of a first classification head network and a first regression head network. The first classification head network receives the query vector set enhanced with image-text information from the first multimodal decoder module, receives the text features enhanced with features from the first feature enhancement module, calculates the cosine similarity between the query vector set enhanced with image-text information and the text features enhanced with features, generates the first network branch predicted object category, and sends the first network branch predicted object category to the non-post-processing prediction value fusion module; the first head network module uses the generated first network branch predicted object category as an indication to screen out a semantic representation set with high confidence, and then performs weighted summation on the semantic representation set to obtain the semantic representation center, and sends it to the second head network module of the second network branch; the first regression head network (composed of a fully connected layer) calculates the prediction box position according to the query vector set enhanced with image-text information received from the first multimodal decoder module, generates the first network branch predicted object position, and sends the first network branch predicted object position to the non-post-processing prediction value fusion module.

[0018] The second network branch is connected to the visual feature extraction module, the text feature extraction module and the no-post-processing prediction value fusion module, and is composed of a second feature enhancement module, a second multimodal decoder module, and a second head network module.

[0019] The second feature enhancement module is connected to the visual feature extraction module, the text feature extraction module, and the second multimodal decoder module, and is composed of an image-text cross-attention network and a text-image cross-attention network. The text-image cross-attention network of the second feature enhancement module receives text features from the text feature extraction module, receives visual features from the visual feature extraction module, and uses the text-image cross-module attention mechanism to align text features to obtain feature-enhanced text features, and sends the feature-enhanced text features to the second multimodal decoder module; the image-text cross-attention network of the second feature enhancement module uses the image-text cross-modal attention mechanism to align image features to obtain feature-enhanced visual features, and sends the feature-enhanced visual features to the second multimodal decoder module.

[0020] The second multimodal decoder module is connected to the second feature enhancement module and the second head network module, and is composed of an image query attention network and a text query attention network, and receives feature-enhanced visual features and feature-enhanced text features from the second feature enhancement module. The image query attention network of the second multimodal decoder module defines a fixed number N queryThe dynamic weights of the image query vector set are used as the query vector set, and the image query attention mechanism is used to use the feature-enhanced visual features to give the query vector set visual information, so as to obtain the query vector set with visual information enhancement, and the query vector set with visual information enhancement is sent to the second head network module; the text query attention network of the second multimodal decoder module uses the text query attention mechanism to use the feature-enhanced text features to give the image information enhanced query vector set text information, so as to obtain the image-text information enhanced query vector set, and the image-text information enhanced query vector set is sent to the second head network module.

[0021] The second head network module is connected to the first head network module, the second feature enhancement module, the second multimodal decoder module, and the non-post-processing prediction value fusion module, and is composed of a second classification head network and a second regression head network. The second classification head network receives the query vector set enhanced with image-text information from the second multimodal decoder module, receives the text features enhanced with features from the second feature enhancement module, calculates the cosine similarity between the query vector set enhanced with image-text information and the text features enhanced with features, generates the second network branch prediction object category, and sends the second network branch prediction object category to the loss function calculation module and the non-post-processing prediction value fusion module; the second classification head network receives the semantic representation center from the first classification head network, calculates the cosine similarity between the query vector set enhanced with image-text information and the semantic representation center, obtains the visual-semantic prediction category, and sends the visual-semantic prediction category to the loss function calculation module. The second regression head network receives the image-text information enhanced query vector set from the second multimodal decoder module, receives the feature-enhanced text features from the second feature enhancement module, calculates the predicted box position for the image-text information enhanced query vector set and the feature-enhanced text features, generates the second network branch predicted object position, and sends the second network branch predicted object position to the loss function calculation module and the post-processing free prediction value fusion module.

[0022] The post-processing-free prediction value fusion module is connected to the dual-branch feature fine-tuning module, and is composed of a geometric mean fusion module and a category-related prediction box screening module. The geometric mean fusion module receives the object category predicted by the first network branch from the first head network module, receives the object category predicted by the second network branch from the second head network module, and fuses the object category predicted by the first network branch and the object category predicted by the second network branch using the geometric mean operation to obtain the known and unknown class object categories. At the same time, the category-related prediction box screening module receives the object position predicted by the first network branch from the first head network module, receives the object position predicted by the second network branch from the second head network module, and uses the category-related prediction box screening operation to align the object position predicted by the first network branch and the object position predicted by the second network branch to obtain the known and unknown class object positions.

[0023] The loss function calculation module is connected to the image preprocessing module and the dual-branch feature fine-tuning module, and is composed of a supervised truth value construction module and a loss value calculation module. The loss function calculation module is only used during the training process of the target detection system. The supervised truth value construction module receives the true value label of the image preprocessing module and generates a supervised truth value for loss value calculation. The loss value calculation module is composed of a text-centric classification loss function, a visual-semantic classification loss function, a regression loss function, and an intersection-over-union loss function. It receives the supervised truth value of the supervised truth value module and the second network branch predicts the object category, the visual-semantic prediction category, and the second network branch predicts the object position, calculates the loss value, and uses gradient backpropagation to update the network parameters of the second network branch of the target detection system.

[0024] The second step is to use the image preprocessing module to build the training set and test set. The method is:

[0025] 2.1 Collect intelligent driving target detection scene images as target detection datasets, and annotate each target detection scene image in the target detection dataset by:

[0026] 2.1.1 Use the SODA intelligent driving scene dataset (see the document "Han J, Liang X, Xu H, et al. SODA10M: a large-scale 2D self / semi-supervised object detection dataset for autonomous driving [J]. arXiv preprint arXiv: 2106.11118, 2021." Han J, Liang X et al.'s paper: SODA10M: A large-scale two-dimensional self / semi-supervised object detection dataset for autonomous driving) as the training set. The SODA dataset has six categories: pedestrians, cyclists, cars, trucks, trams, and tricycles, containing 10,000 images. The CODA intelligent driving dataset (see the document "Li K, Chen K, Wang H, et al. Coda: A real-world roadcorner case dataset for object detection in autonomous driving [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 406-423." Li K, Chen K et al.'s paper: CODA: A real-world road corner case dataset for object detection in autonomous driving) is used as the test set. In addition to the 6 categories in the SODA dataset, the CODA dataset also has 23 rare categories such as carts and traffic cones, containing 4884 images. During testing, the 6 categories in the SODA dataset are put into the known object category set C k , put the 23 rare categories in the CODA dataset into the unknown object category set C uk Let the total number of images in the training set be M, and the total number of images in the test set be T, M is 10000, and T is 4884.

[0027] 2.2.2 The M images and corresponding labels in the training set are enhanced using the image preprocessing module to obtain the enhanced training set D t , the method is:

[0028] 2.2.2.1 Let variable m = 1 and initialize the optimized training set D t is empty; let the mth image have N m (N m ≥1) labeled boxes, let N m The i-th annotation box among the annotation boxes is the position truth label, Represents the i-th annotation box The coordinates of the upper left corner, Represents the i-th annotation box The coordinates of the lower right corner of m The i-th annotation box among the annotation boxes Category for The classification truth label, 1≤i≤N s .

[0029] 2.2.2.2 Randomly flip the training set D t Flip the mth image in the image to get the mth flipped image, and the position truth label corresponding to the mth image Flip the image to get the true value label of the flipped position; the flip probability of the random flip method is set to 0.5.

[0030] 2.2.2.3 Use the random cropping method to crop the mth flipped image to obtain the mth cropped image. The position truth label after flipping the mth image is cropped to obtain the cropped position truth label; the cropping probability of the random cropping method is set to 0.5.

[0031] 2.2.2.4 Use the random scaling method to scale the mth flipped image to obtain the mth scaled image, and scale the position truth label after cropping the mth image to obtain the optimized position truth label; the scaling probability of the random scaling method is set to 0.5.

[0032] 2.2.2.5 Use the image normalization operation to normalize the mth scaled image to obtain the mth normalized image. Put the mth normalized image, the optimized position truth label of the mth image, and the classification truth label corresponding to the position truth label into D t .

[0033] 2.2.2.6 If m≤M, go to 2.2.2.2; if m>M, obtain the enhanced training set D consisting of M normalized images, M image optimized position truth labels, and classification truth labels corresponding to the position truth labels t , go to 2.2.3.

[0034] 2.2.3 D t The optimized position truth labels of the M images and the classification truth labels corresponding to the position truth labels are sent to the loss function calculation module.

[0035] 2.2.4 The image preprocessing module uses the image scaling normalization method to optimize the T images and corresponding labels in the test set, and obtains a new test set Test consisting of the normalized T images, the optimized position truth labels of the T images, and the classification truth labels corresponding to the position truth labels. The method is:

[0036] 2.2.4.1 Let variable t = 1 and initialize the new test set Test to be empty;

[0037] 2.2.4.2 Use the scaling operation to scale the t-th image in the test set to a size of 800×1333 to obtain the t-th scaled image, and scale the position truth label of the t-th image to obtain the optimized position truth label;

[0038] 2.2.4.3 Use image normalization operation to normalize the t-th scaled image to obtain the normalized t-th image, and put the normalized t-th image, the position truth label of the t-th image after scaling, and the classification truth label corresponding to the position truth label into the new test set Test.

[0039] 2.2.4.4 If t≤T, go to 2.2.3.2; if t>T, obtain a new test set Test consisting of T normalized images, T optimized position truth labels of images, and T classification truth labels of images. The Test test set is used to evaluate the performance of the target detection system (results in Table 1).

[0040] The third step is to use the gradient back propagation method to train the dual-branch feature fine-tuning module in the target detection system built in the first step to obtain the model saving parameters. The method is:

[0041] 3.1 Initialize the network weight parameters of each module in the target detection system. Use the weight parameters of the GroundingDINO model to initialize the weight parameters of the main feature extraction module and the dual-branch feature fine-tuning module. The initialization model parameters of the first network branch and the second network branch of the dual-branch feature fine-tuning module are consistent, where the model parameters of the first network branch are frozen and the model parameters of the second network branch are learnable.

[0042] 3.2 Set the training parameters of the target detection system. Set the initial learning rate (learning_rate) to 0.0001, select AdamW as the model training optimizer, and the hyperparameter "weight decay" of the optimizer is 0.0001. The batch size (mini_batch_size) of the network training is B=8. The training step (epoch) is 12.

[0043] 3.3 Train the object detection system by calculating the difference between the predicted position and predicted category of the known object output by the current network and the true value as the loss value, and using the gradient direction propagation to update the network weight parameters until the 12th epoch is trained. The final network weight parameters are the model parameters of the 12th epoch. The specific method is as follows:

[0044] 3.3.1 Initialize batch number N b =1, initialize step number N epoch = 1, define a batch from the optimized training set D t Read B images. The visual feature extraction module in the main feature extraction module extracts B images from D t Read the Nth b Batch, a total of B images, these B images are recorded as a matrix I of size B×H×W×3 train . H and W represent the width and height of the input image respectively, and "3" represents the RGB channels of the image. The text feature extraction module in the main feature extraction module receives the category text input by the user, that is, the known category text I text , including "Pedestrian", "Cyclist", "Car", "Truck", "Bus", "Tricycle".

[0045] 3.3.2 Main feature extraction module uses feature extraction method to extract I train Visual feature extraction is performed to obtain I train The visual features F v , to I text Perform text feature extraction to obtain text feature F t , the method is: the visual feature extraction module is used to extract I train Perform visual feature extraction to obtain visual feature F v At the same time, the text feature extraction module is used to extract I text Perform text feature extraction to obtain text feature F t , F t and F v Send it to the dual-branch feature fine-tuning module. The specific method is as follows:

[0046] 3.3.2.1 Visual Feature Extraction Module Extraction I train The visual features of v , in is the first scale visual feature map, size is is the second scale visual feature map, size is is the third scale visual feature map, size is is the fourth scale visual feature map, size is E = 256 is the visual feature dimension, and F v Sent to the dual-branch feature fine-tuning module.

[0047] 3.3.2.2 Text Feature Extraction Module Extraction I text The category text feature F t , F t Sent to the dual-branch feature fine-tuning module.

[0048] 3.3.3 The second network branch in the dual-branch feature fine-tuning module receives F from the main feature extraction module v and F t , perform feature enhancement and alignment operations to obtain the second network branch to predict the object category Visual-semantic prediction categories (output only during training), location information The weight parameters of the first network branch in the dual-branch feature fine-tuning module are not updated with gradients, but the ... The intermediate feature output of the first network branch is needed, so the first network branch needs to perform forward propagation. The second network branch receives F from the visual feature extraction network of the main feature extraction module. v , receives F from the text feature extraction network t , after the second feature enhancement module, the second multimodal decoder module, and the second head network module, the second network branch predicts the object category Visual-semantic prediction categories and location information Here’s how:

[0049] 3.3.3.1 The first feature enhancement module of the first network branch adopts the first branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F′ t1 and feature-enhanced visual features F′ v1 The specific method is as follows: the text-image cross-attention network of the first feature enhancement module receives the text feature F from the text feature extraction module t , the text-image cross-module attention mechanism is used to align the text features and obtain the feature-enhanced text feature F′ t1 ; The image-text cross-attention network of the first feature enhancement module receives the visual features F from the visual feature extraction module v , using the image-text cross-modal attention mechanism to align visual features F v, and obtain the feature-enhanced visual feature F′ v1 . F′ v1 and F′ t1 The dimensions are all B×N text ×E text , where N text is the number of category texts, N text =6, E text is the dimension of text features, E text =256. v1 and F′ t1 Sent to the first multimodal decoder module.

[0050] 3.3.3.2 The first multimodal decoder module receives F′ from the first feature enhancement module v1 and F′ t1 The image query attention network of the first multimodal decoder module defines a B×N query ×E query The query vector set Q1, where N query is the number of query vector sets, N query =300, E query is the feature dimension of the query vector set, E query = 256. The image query attention network of the first multimodal decoder module defines a B×N query ×4 query vector set initial anchor box position A1. The image query attention network of the first multimodal decoder module is F′ v1 , Q1 and A1 perform image query attention operations, give the query vector set visual information, and obtain the query vector set with enhanced visual information The first multimodal decoder’s text query attention network is used to calculate F′ v1 , Perform text query attention operation with A1 to generate a query vector set Q1′ with enhanced image-text information. Send Q1′ to the first head network module of the first network branch.

[0051] 3.3.3.3 The first classification head network of the first head network module receives Q′1 from the first multimodal decoder module and receives F′ from the first feature enhancement module t1 , calculate Q′1 and F′ t1 The cosine similarity of query ×N text The first network branch predicts the object category The elements of represent the predicted probability values ​​of the query vector set in a certain category of text. Send to the prediction value fusion module without post-processing;

[0052] 3.3.3.4 The first classification head network As an indication, filter out the semantic representation set S with high confidence, and then perform weighted summation on S to obtain the semantic representation center set F s , F s The method of sending to the second head network module of the second network branch is:

[0053] 3.3.3.4.1 Initialize c = 0, indicating the cth category. Initialize F s An empty collection used to store C k The semantic representation center of each class.

[0054] 3.3.3.4.2 Set the screening threshold β to 0.3, screen the prediction values ​​of the c-th category, and obtain the semantic representation set (S) of the c-th category whose confidence is greater than β c :

[0055]

[0056] in, represents the prediction probability value of the first network branch in the cth category text of the query vector q, yes An element in , q is an element in Q1′, representing a query vector.

[0057] 3.3.3.4.3 Take the weighted average of the semantic representation and confidence of the cth category and get a size of B×E query The semantic representation center of the c-th category (F s ) c :

[0058]

[0059] in, Represents the semantic representation set of the cth category (S) c The first branch network predicts the probability value of a query vector s in (S) c The elements in is a vector, Indicates (S) c The sum of the predicted probability values ​​of the first branch network of each element in .

[0060] 3.3.3.4.4 If c<C k , let c = c + 1, and change (F s ) c Deposit F s Go to 3.3.3.4.2; if c = C k , indicating the semantic representation center set F s , F sThe second head network module sent to the second network branch (transmitted only during target detection system training, used to calculate visual-semantic contrast loss, not transmitted when detecting user input images), go to 3.3.3.5.

[0061] 3.3.3.5 The first regression head network of the first head network module calculates the predicted box position according to the query vector set Q1′ enhanced by the image-text information received from the first multimodal decoder module, and generates the first network branch to predict the object position The first network branch predicts the object position Sent to the prediction value fusion module without post-processing.

[0062] 3.3.3.6 The second feature enhancement module of the second network branch adopts the second branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F′ t2 and feature-enhanced visual features F′ v2 The method is as follows: the text-image cross-attention network of the second feature enhancement module receives F from the text feature extraction module t , the text-image cross-module attention mechanism is used to align the text features and obtain the second feature enhanced text feature F′ t2 ; The image-text cross-attention network of the second feature enhancement module receives the visual features F from the visual feature extraction module v , using the image-text cross-modal attention mechanism to align visual features F v , and obtain the visual feature F′ enhanced by the second feature v2 , F′ t2 and F′ v2 Sent to the second multimodal decoder module. F′ v2 and F′ t2 The dimensions are all B×N text ×E text .

[0063] 3.3.3.7 The second multimodal decoder module receives F′ from the second feature enhancement module v2 and F′ t2 The image query attention network of the second multimodal decoder module defines a B×N query ×F query The query vector set Q2. The image query attention network of the second multimodal decoder module defines a size of B×N query ×4 query vector set initial anchor box position A2. The image query attention network of the second multimodal decoder module is F′ v2, Q2 and A2, perform image query attention operation, give visual information to the query vector set, and obtain the query vector set with enhanced visual information The second multimodal decoder’s text query attention network is used to calculate F′ v2 , Perform text query attention operation with A2 to generate a query vector set with enhanced image-text information, denoted as Q2′. Send Q2′ to the second head network module of the second network branch.

[0064] 3.3.3.8 The second classification head network of the second head network module receives Q′2 from the second multimodal decoder module and receives F′ from the second feature enhancement module t2 , calculate Q′2 and F′ t2 The cosine similarity of query ×N text The second prediction category matrix The second classification head network receives F from the classification head network of the first branch network. s , calculate Q′2 and F s The cosine similarity of query ×N text The second visual-semantic prediction category The second regression head network of the second head network module receives Q′2 from the second multimodal decoder module and calculates the position of the predicted box as The dimension size is B×N query ×4. The loss function calculation module sent to the target detection system.

[0065] 3.3.4 The supervised truth value construction module in the loss function calculation module receives the second classification head network Receive D from the image preprocessing module t The optimized position truth labels of the M images and the classification truth labels corresponding to the position truth labels are used to construct the Nth b The supervised truth value building module converts D into the supervised truth value through the Hungarian algorithm adopted in the DETR paper (see the literature "Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers [C] / / European conference on computer vision. Cham: Springer International Publishing, 2020: 213-229." Carion N, Massa F et al.: End-to-end object detection with Transformer)i The classification truth labels in Perform one-to-one pairing for Generate the corresponding supervised truth value Size is B×N query ×N text ; D t The position truth labels and Perform one-to-one pairing for Generate the corresponding supervised truth value Size is B×N query ×4. Visual-semantic prediction categories is an auxiliary classification prediction, and its supervised true value is Will Sent to the loss value calculation module.

[0066] 3.3.5 The loss value calculation module in the loss function calculation module receives the supervision truth value construction module The loss value is calculated for the object category, visual-semantic prediction category and location information predicted by the second network branch and the supervised true value, and the network weight parameters of the second network branch are updated using gradient back propagation. Total loss function for:

[0067]

[0068] in YesP cls and The classification loss function used for calculation is Focal Loss (see the literature "Lin TY, Goyal P, Girshick R, et al. Focal loss for dense object detection [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 2980-2988." Lin TY, Goyal P et al.'s paper: Focal loss for dense object detection). It is a visual-semantic contrast loss function, using Focal Loss. It is the regression loss function, using L1 Loss (also known as mean absolute error). It is the intersection over union loss function, using GIoU loss (see the literature "Rezatofighi H, Tsoi N, Gwak JY, et al. Generalized intersection over union: A metric and a loss for bounding box regression [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 658-666." Rezatofighi H, Tsoi N et al. paper: Generalized intersection over union: metric and loss for bounding box regression). is the weight of the classification loss function and is set to 1. is the weight of the visual-semantic contrast loss function and is set to 1. is the weight of the regression loss function, set to 5. iou is the weight of the intersection-over-union loss function and is set to 2.

[0069] 3.3.6 If N b ≤M / B, let N b =N b +1, go to 3.3.1; if N b >M / B, that is, the Nth epoch Step length training, let N b =1, go to 3.3.7.

[0070] 3.3.7 If N epoch =11, let learning_rate = learning_rate × 0.1, go to 3.3.8; if N epoch ≠11, directly switch to 3.3.8.

[0071] 3.3.8 If N epoch ≤11, let N epoch =N epoch +1, go to 3.3.1; if N epoch =12, save the model parameters of the second network branch at this time, indicating that the training is completed, and go to 3.3.9.

[0072] 3.3.9 Load the network weight parameters of the second network branch obtained by training into the second network branch of the target detection system to obtain the trained target detection system. The fourth step is to use the trained target detection system to perform category and position detection on the Test test set or the image to be detected input by the user, the method is:

[0073] 4.1 The image preprocessing module uses the image scaling and normalization method described in step 2.2.4 to optimize the Test test set or the image to be detected input by the user to obtain a normalized image set to be detected. The image set to be detected is sent to the main feature extraction module;

[0074] 4.2 Main feature extraction module uses the feature extraction method described in step 3.3.2 to extract visual features of the image set to be detected, and obtains the visual feature FF of the image set to be detected v ; For predefined category texts (including known category texts I text and unknown category text) to extract text features and obtain the predefined category text feature FF t ; The present invention defines "Vehicle", "Roadblock" and "Obstacle" as unknown categories. v and FF t Sent to the dual-branch feature fine-tuning module.

[0075] 4.3 The first feature enhancement module in the dual-branch feature fine-tuning module adopts the first branch cross-attention method in step 3.3.3.1 to perform v and FF t Perform feature enhancement and alignment to obtain the visual feature FF′ of the first branch feature enhancement of the image set to be detected v1 And feature-enhanced text features FF′ t1 , FF′ v1 and FF′ t1 Send it to the first multimodal decoder module; at the same time, the second feature enhancement module in the dual-branch feature fine-tuning module adopts the second branch cross-attention method in step 3.3.3.6 to FF v and FF t Perform feature enhancement and alignment to obtain the visual feature FF′ of the second branch feature enhancement of the image set to be detected v2 and text feature FF′ t2 , FF′ v2 and FF′ t2 Sent to the second multimodal decoder module.

[0076] 4.4 The first multimodal decoder module of the first network branch receives FF′ from the first feature enhancement module v1 and FF′ t1 , using the query vector set Q1 of the first network branch and the initial anchor box position A1 of the first network branch, the image query attention network of the first multimodal decoder module is FF′ v1 , Q1 and A1 perform image query attention operations, give the query vector set visual information, and obtain the query vector set with enhanced visual information The first multimodal decoder’s text query attention network for FF′ v1 , Perform text query attention operation with A1 to generate a query vector set QQ1′ with enhanced image-text information, and send QQ1′ to the first head network module of the first network branch; at the same time, the second multimodal decoder module of the second network branch receives FF′ from the second feature enhancement module v2 and FF′ t2 , using the query vector set Q2 of the second network branch and the initial anchor box position A1 of the first network branch, the image query attention network of the second multimodal decoder module is FF′ v2 , Q2 and A1, perform image query attention operation, give visual information to the query vector set, and obtain the query vector set with enhanced visual information The second multimodal decoder’s text query attention network is used to calculate FF′ v2 , And A1 perform text query attention operation to generate a query vector set with enhanced image-text information, recorded as QQ2′. Send QQ2′ to the second head network module of the second network branch. The reason why the initial anchor box position of the second network branch adopts the same initial anchor box position A1 as the first network branch during runtime is that the reason for the mismatch in the prediction values ​​of the two network branches is that the initial anchor boxes cannot correspond one to one. Therefore, using the same initial anchor box position A1 for the multimodal decoder modules of the two branches can align the order of the prediction result sets of the first and second network branches, making it convenient for the post-processing prediction value fusion module to fuse the prediction results of the first and second network branches. 4.5 The first classification head network of the first head network module receives QQ′1 from the first multimodal decoder module and receives FF′ from the first feature enhancement module. t1 , calculate QQ′1 and FF′ t1 The cosine similarity of query ×N text The predicted category of the first network branch The elements of represent the predicted probability values ​​of the multimodal query vector set of the first network branch in a certain category of text; the regression head network module of the first network branch receives QQ′1 from the first multimodal decoder module, calculates the position of the prediction box, and obtains the predicted object position of the first network branch At the same time, the classification head network module of the second network branch receives QQ″2 from the second multimodal decoder module and receives FF′ from the second feature enhancement module. t2 , calculate QQ″2 and FF″ t2 The cosine similarity of query ×N text The predicted category of the second network branch The regression head network module of the second network branch receives QQ″2 from the second multimodal decoder module, calculates the position of the predicted box, and obtains the predicted object position of the second network branch Will Sends no post-processing predictions to the fusion module.

[0077] 4.6 Category-related prediction box screening module without post-processing prediction value fusion module receives from the first network branch Received from the second network branch Filtering operation fusion using category-related prediction boxes and Get the object position of the cth category of the image to be detected

[0078]

[0079] in, express The predicted box position of the cth class, express The predicted box position of the c-th class.

[0080] 4.7 Geometric mean fusion module without post-processing prediction value fusion module receives from the first network branch The second network branch receives Fusion using geometric mean operation and Get the object category score of the cth category of the image to be detected

[0081]

[0082] in, express The prediction score of the cth class, C k Indicates the category of known objects, C uk represents the category of the unknown object, and α is the geometric mean parameter, which is set to 0.3.

[0083] The fifth step is to obtain the object category and object position of the image set to be detected, and the target detection is completed.

[0084] The present invention can achieve the following beneficial effects:

[0085] The present invention proposes an intelligent driving target detection method based on fine-tuning of a large visual-language model, which only fine-tunes one network branch of the large visual-language model, which accounts for 19% of the total model training parameters. The present invention adopts a dual-branch feature fine-tuning structure design, visual-semantic contrast learning, and prediction result fusion technology without post-processing to improve the detection accuracy of common objects in intelligent driving without affecting the detection capability of the large visual-language model for unknown objects. The present invention is suitable for fine-tuning the existing advanced large visual-language model and applying it to intelligent driving scenarios. It can detect both known and unknown classes and has great application value. The following effects can be achieved by using the present invention:

[0086] 1. The present invention constructs a complete intelligent driving target detection system, which integrates the main feature extraction module, the dual-branch feature fine-tuning module and the post-processing-free prediction value fusion module. While ensuring the model's ability to recognize unknown objects, the weight parameters of the second network branch are fine-tuned using the intelligent driving known class data set, greatly improving the recognition ability of known class objects. The present invention has preliminarily verified its performance on the CODA data set, and can achieve accurate detection of known and unknown class objects.

[0087] 2. The present invention designs a dual-branch feature fine-tuning module, adopts two dual network structures, and retains the recognition ability of the visual-language large model for unknown objects by freezing the weight parameters of the main feature extraction module and the first network branch. By fine-tuning the weight parameters of the second network branch, the target detection system can quickly learn the knowledge of known objects. The present invention designs visual-semantic contrast learning and adds visual-semantic contrast loss. Align the visual representation of known class objects to the semantic space to improve the recognition accuracy of known class objects. The present invention designs a prediction result fusion module without post-processing. It can fuse the prediction results of the first and second network branches through geometric mean operation and category-related prediction box screening operation without using post-processing operations such as non-maximum suppression (NMS), which not only reduces the computational overhead but also achieves high detection accuracy for known and unknown class objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 The logical structure diagram of the target detection system constructed in the first step of the present invention.

[0089] Figure 2 It is the overall flow chart of the present invention.

[0090] Figure 3 This is a visualization diagram comparing the detection results of the method of the present invention with the GroundingDINO method. DETAILED DESCRIPTION

[0091] like Figure 2As shown, the present invention comprises the following steps:

[0092] The first step is to build a target detection system. Figure 1 As shown in the figure, the target detection system consists of an image preprocessing module, a main feature extraction module, a dual-branch feature fine-tuning module, a non-post-processing prediction value fusion module and a loss function calculation module. Figure 1 The dotted lines in indicate that the connection relationship only exists during the training of the target detection system.

[0093] The image preprocessing module is connected to the main feature extraction module and the loss function calculation module, receives the input image, performs enhancement processing (including flipping, cropping, scaling, and normalization) during the training of the target detection system, sends the processed image and the corresponding true value label to the main feature extraction module, and sends the true value label to the loss function calculation module. When the target detection system performs target detection on the input image set to be detected, it performs scaling and normalization processing to obtain the processed image, and sends the processed image to the main feature extraction module.

[0094] The main feature extraction module is connected to the dual-branch feature fine-tuning module, and the main feature extraction module consists of a visual feature extraction module and a text feature extraction module. The visual feature extraction module receives the processed image from the image preprocessing module, extracts visual features, and sends the visual features to the dual-branch feature fine-tuning module. The visual feature extraction module is composed of the SwinTransformer neural network in GroundingDINO (see the document "Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 10012-10022." Liu Z, Lin Y et al.'s paper: SwinTransformer). The text feature extraction module extracts text features from the input category text (including known category text (categories in the training data set such as Pedestrian, Car, etc.) and unknown category text (categories outside the training data set such as Vehicle, Roadblock, etc.), and sends the text features to the dual-branch feature fine-tuning module. The text feature extraction module is composed of a Bert neural network (see the document "Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [J]. arXiv preprint arXiv: 1810.04805, 2018." Devlin J, Chang MW et al.'s paper: Bert).

[0095] The dual-branch feature fine-tuning module is connected to the visual feature extraction module, the text feature extraction module and the non-post-processing prediction value fusion module. The dual-branch feature fine-tuning module consists of two network branches with the same structure, namely the first network branch and the second network branch. The weight parameters of the first network branch are not updated, and the weight parameters of the second network branch are updated by gradient.

[0096] The first network branch is connected to the visual feature extraction module, the text feature extraction module and the no-post-processing prediction value fusion module, and is composed of a first feature enhancement module, a first multimodal decoder module and a first head network module.

[0097] The first feature enhancement module is connected to the visual feature extraction module, the text feature extraction module, and the first multimodal decoder module, and is composed of an image-text cross-attention network and a text-image cross-attention network. The text-image cross-attention network of the first feature enhancement module receives text features from the text feature extraction module, receives visual features from the visual feature extraction module, uses the text-image cross-module attention mechanism to align text features, obtains feature-enhanced text features, and sends the feature-enhanced text features to the first multimodal decoder module; the image-text cross-attention network of the first feature enhancement module uses the image-text cross-modal attention mechanism to align image features, obtains feature-enhanced visual features, and sends the feature-enhanced visual features to the first multimodal decoder module.

[0098] The first multimodal decoder module is connected to the first feature enhancement module and the first head network module, and is composed of an image query attention network and a text query attention network, and receives feature-enhanced visual features and feature-enhanced text features from the first feature enhancement module. The image query attention network of the first multimodal decoder module defines a fixed number N query (N query =300) as the query vector set, defining a size B×N query ×4 as the initial anchor box position of the query vector set, and use the image query attention mechanism to use the feature-enhanced visual features to give the query vector set visual information, so as to obtain the query vector set with visual information enhancement, and send the query vector set with visual information enhancement to the first head network module; the text query attention network of the first multimodal decoder module uses the text query attention mechanism to use the feature-enhanced text features to give the query vector set with image information enhancement text information, so as to obtain the query vector set with image-text information enhancement, and send the query vector set with image-text information enhancement to the first head network module.

[0099] The first head network module is connected to the first feature enhancement module, the first multimodal decoder module, and the non-post-processing prediction value fusion module, and is composed of a first classification head network and a first regression head network. The first classification head network receives the query vector set enhanced with image-text information from the first multimodal decoder module, receives the text features enhanced with features from the first feature enhancement module, calculates the cosine similarity between the query vector set enhanced with image-text information and the text features enhanced with features, generates the first network branch predicted object category, and sends the first network branch predicted object category to the non-post-processing prediction value fusion module; the first head network module uses the generated first network branch predicted object category as an indication to screen out a semantic representation set with high confidence, and then performs weighted summation on the semantic representation set to obtain the semantic representation center, and sends it to the second head network module of the second network branch; the first regression head network (composed of a fully connected layer) calculates the prediction box position according to the query vector set enhanced with image-text information received from the first multimodal decoder module, generates the first network branch predicted object position, and sends the first network branch predicted object position to the non-post-processing prediction value fusion module.

[0100] The second network branch is connected to the visual feature extraction module, the text feature extraction module and the no-post-processing prediction value fusion module, and is composed of a second feature enhancement module, a second multimodal decoder module, and a second head network module.

[0101] The second feature enhancement module is connected to the visual feature extraction module, the text feature extraction module, and the second multimodal decoder module, and is composed of an image-text cross-attention network and a text-image cross-attention network. The text-image cross-attention network of the second feature enhancement module receives text features from the text feature extraction module, receives visual features from the visual feature extraction module, and uses the text-image cross-module attention mechanism to align text features to obtain feature-enhanced text features, and sends the feature-enhanced text features to the second multimodal decoder module; the image-text cross-attention network of the second feature enhancement module uses the image-text cross-modal attention mechanism to align image features to obtain feature-enhanced visual features, and sends the feature-enhanced visual features to the second multimodal decoder module.

[0102] The second multimodal decoder module is connected to the second feature enhancement module and the second head network module, and is composed of an image query attention network and a text query attention network, and receives feature-enhanced visual features and feature-enhanced text features from the second feature enhancement module. The image query attention network of the second multimodal decoder module defines a fixed number N queryThe dynamic weights of the image query vector set are used as the query vector set, and the image query attention mechanism is used to use the feature-enhanced visual features to give the query vector set visual information, so as to obtain the query vector set with visual information enhancement, and the query vector set with visual information enhancement is sent to the second head network module; the text query attention network of the second multimodal decoder module uses the text query attention mechanism to use the feature-enhanced text features to give the image information enhanced query vector set text information, so as to obtain the image-text information enhanced query vector set, and the image-text information enhanced query vector set is sent to the second head network module.

[0103] The second head network module is connected to the first head network module, the second feature enhancement module, the second multimodal decoder module, and the non-post-processing prediction value fusion module, and is composed of a second classification head network and a second regression head network. The second classification head network receives the query vector set enhanced with image-text information from the second multimodal decoder module, receives the text features enhanced with features from the second feature enhancement module, calculates the cosine similarity between the query vector set enhanced with image-text information and the text features enhanced with features, generates the second network branch prediction object category, and sends the second network branch prediction object category to the loss function calculation module and the non-post-processing prediction value fusion module; the second classification head network receives the semantic representation center from the first classification head network, calculates the cosine similarity between the query vector set enhanced with image-text information and the semantic representation center, obtains the visual-semantic prediction category, and sends the visual-semantic prediction category to the loss function calculation module. The second regression head network receives the image-text information enhanced query vector set from the second multimodal decoder module, receives the feature-enhanced text features from the second feature enhancement module, calculates the predicted box position for the image-text information enhanced query vector set and the feature-enhanced text features, generates the second network branch predicted object position, and sends the second network branch predicted object position to the loss function calculation module and the post-processing free prediction value fusion module.

[0104] The post-processing-free prediction value fusion module is connected to the dual-branch feature fine-tuning module, and is composed of a geometric mean fusion module and a category-related prediction box screening module. The geometric mean fusion module receives the object category predicted by the first network branch from the first head network module, receives the object category predicted by the second network branch from the second head network module, and fuses the object category predicted by the first network branch and the object category predicted by the second network branch using the geometric mean operation to obtain the known and unknown class object categories. At the same time, the category-related prediction box screening module receives the object position predicted by the first network branch from the first head network module, receives the object position predicted by the second network branch from the second head network module, and uses the category-related prediction box screening operation to align the object position predicted by the first network branch and the object position predicted by the second network branch to obtain the known and unknown class object positions.

[0105] The loss function calculation module is connected to the image preprocessing module and the dual-branch feature fine-tuning module, and is composed of a supervised truth value construction module and a loss value calculation module. The loss function calculation module is only used during the training process of the target detection system. The supervised truth value construction module receives the true value label of the image preprocessing module and generates a supervised truth value for loss value calculation. The loss value calculation module is composed of a text-centric classification loss function, a visual-semantic classification loss function, a regression loss function, and an intersection-over-union loss function. It receives the supervised truth value of the supervised truth value module and the second network branch predicts the object category, the visual-semantic prediction category, and the second network branch predicts the object position, calculates the loss value, and uses gradient backpropagation to update the network parameters of the second network branch of the target detection system.

[0106] The second step is to use the image preprocessing module to build the training set and test set. The method is:

[0107] 2.1 Collect intelligent driving target detection scene images as target detection datasets, and annotate each target detection scene image in the target detection dataset by:

[0108] 2.1.1 Use the SODA intelligent driving scene dataset (see the document "Han J, Liang X, Xu H, et al. SODA10M: a large-scale 2D self / semi-supervised object detection dataset for autonomous driving [J]. arXiv preprint arXiv: 2106.11118, 2021." Han J, Liang X et al.'s paper: SODA10M: A large-scale two-dimensional self / semi-supervised object detection dataset for autonomous driving) as the training set. The SODA dataset has six categories: pedestrians, cyclists, cars, trucks, trams, and tricycles, containing 10,000 images. The CODA intelligent driving dataset (see the document "Li K, Chen K, Wang H, et al. Coda: A real-world roadcorner case dataset for object detection in autonomous driving [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 406-423." Li K, Chen K et al.'s paper: CODA: A real-world road corner case dataset for object detection in autonomous driving) is used as the test set. In addition to the 6 categories in the SODA dataset, the CODA dataset also has 23 rare categories such as carts and traffic cones, containing 4884 images. During testing, the 6 categories in the SODA dataset are placed in the known object category set C k , put the 23 rare categories in the CODA dataset into the unknown object category set C uk Let the total number of images in the training set be M, and the total number of images in the test set be T, M is 10000, and T is 4884.

[0109] 2.2.2 The M images and corresponding labels in the training set are enhanced using the image preprocessing module to obtain the enhanced training set D t , the method is:

[0110] 2.2.2.1 Let variable m = 1 and initialize the optimized training set D t is empty; let the mth image have N m (N m ≥1) labeled boxes, let N m The i-th annotation box among the annotation boxes is the position truth label, Represents the i-th annotation box The coordinates of the upper left corner, Represents the i-th annotation box The coordinates of the lower right corner of m The i-th annotation box among the annotation boxes Category for The classification truth label, 1≤i≤N s .

[0111] 2.2.2.2 Randomly flip the training set D t Flip the mth image in the image to get the mth flipped image, and the position truth label corresponding to the mth image Flip the image to get the true value label of the flipped position; the flip probability of the random flip method is set to 0.5.

[0112] 2.2.2.3 Use the random cropping method to crop the mth flipped image to obtain the mth cropped image. The position truth label after flipping the mth image is cropped to obtain the cropped position truth label; the cropping probability of the random cropping method is set to 0.5.

[0113] 2.2.2.4 Use the random scaling method to scale the mth flipped image to obtain the mth scaled image, and scale the position truth label after cropping the mth image to obtain the optimized position truth label; the scaling probability of the random scaling method is set to 0.5.

[0114] 2.2.2.5 Use the image normalization operation to normalize the mth scaled image to obtain the mth normalized image. Put the mth normalized image, the optimized position truth label of the mth image, and the classification truth label corresponding to the position truth label into D t .

[0115] 2.2.2.6 If m≤M, go to 2.2.2.2; if m>M, obtain the enhanced training set D consisting of M normalized images, M image optimized position truth labels, and classification truth labels corresponding to the position truth labels t , go to 2.2.3.

[0116] 2.2.3 D t The optimized position truth labels of the M images and the classification truth labels corresponding to the position truth labels are sent to the loss function calculation module.

[0117] 2.2.4 The image preprocessing module uses the image scaling normalization method to optimize the T images and corresponding labels in the test set, and obtains a new test set Test consisting of the normalized T images, the optimized position truth labels of the T images, and the classification truth labels corresponding to the position truth labels. The method is:

[0118] 2.2.4.1 Let variable t = 1 and initialize the new test set Test to be empty;

[0119] 2.2.4.2 Use the scaling operation to scale the t-th image in the test set to a size of 800×1333 to obtain the t-th scaled image, and scale the position truth label of the t-th image to obtain the optimized position truth label;

[0120] 2.2.4.3 Use image normalization operation to normalize the t-th scaled image to obtain the normalized t-th image, and put the normalized t-th image, the position truth label of the t-th image after scaling, and the classification truth label corresponding to the position truth label into the new test set Test.

[0121] 2.2.4.4 If t≤T, go to 2.2.3.2; if t>T, obtain a new test set Test consisting of T normalized images, T optimized position truth labels of images, and T classification truth labels of images. The Test test set is used to evaluate the performance of the target detection system (results in Table 1).

[0122] The third step is to use the gradient back propagation method to train the dual-branch feature fine-tuning module in the target detection system built in the first step to obtain the model saving parameters. The method is:

[0123] 3.1 Initialize the network weight parameters of each module in the target detection system. Use the weight parameters of the GroundingDINO model to initialize the weight parameters of the main feature extraction module and the dual-branch feature fine-tuning module. The initialization model parameters of the first network branch and the second network branch of the dual-branch feature fine-tuning module are consistent, where the model parameters of the first network branch are frozen and the model parameters of the second network branch are learnable.

[0124] 3.2 Set the training parameters of the target detection system. Set the initial learning rate (learning_rate) to 0.0001, select AdamW as the model training optimizer, and the hyperparameter "weight decay" of the optimizer is 0.0001. The batch size (mini_batch_size) of the network training is B=8. The training step (epoch) is 12.

[0125] 3.3 Train the object detection system by calculating the difference between the predicted position and predicted category of the known object output by the current network and the true value as the loss value, and using the gradient direction propagation to update the network weight parameters until the 12th epoch is trained. The final network weight parameters are the model parameters of the 12th epoch. The specific method is as follows:

[0126] 3.3.1 Initialize batch number N b =1, initialize step number N epoch = 1, define a batch from the optimized training set D t Read B images. The visual feature extraction module in the main feature extraction module extracts B images from D t Read the Nth b Batch, a total of B images, these B images are recorded as a matrix I of size B×H×W×3 train . H and W represent the width and height of the input image respectively, and "3" represents the RGB channels of the image. The text feature extraction module in the main feature extraction module receives the category text input by the user, that is, the known category text I text , including "Pedestrian", "Cyclist", "Car", "Truck", "Bus", "Tricycle".

[0127] 3.3.2 Main feature extraction module uses feature extraction method to extract I train Visual feature extraction is performed to obtain I train The visual features F v , to I text Perform text feature extraction to obtain text feature F t , the method is: the visual feature extraction module is used to extract I train Perform visual feature extraction to obtain visual feature F v At the same time, the text feature extraction module is used to extract I text Perform text feature extraction to obtain text feature F t , F t and F v Send it to the dual-branch feature fine-tuning module. The specific method is as follows:

[0128] 3.3.2.1 Visual Feature Extraction Module Extraction I train The visual features of v , in is the first scale visual feature map, size is is the second scale visual feature map, size is is the third scale visual feature map, size is is the fourth scale visual feature map, size is E = 256 is the visual feature dimension, and F v Sent to the dual-branch feature fine-tuning module.

[0129] 3.3.2.2 Text Feature Extraction Module Extraction I text The category text feature F t , F t Sent to the dual-branch feature fine-tuning module.

[0130] 3.3.3 The second network branch in the dual-branch feature fine-tuning module receives F from the main feature extraction module v and F t , perform feature enhancement and alignment operations to obtain the second network branch to predict the object category Visual-semantic prediction categories (output only during training), location information The weight parameters of the first network branch in the dual-branch feature fine-tuning module are not updated with gradients, but the ... The intermediate feature output of the first network branch is needed, so the first network branch needs to perform forward propagation. The second network branch receives F from the visual feature extraction network of the main feature extraction module. v , receives F from the text feature extraction network t , after the second feature enhancement module, the second multimodal decoder module, and the second head network module, the second network branch predicts the object category Visual-semantic prediction categories and location information Here’s how:

[0131] 3.3.3.1 The first feature enhancement module of the first network branch adopts the first branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F′ t1 and feature-enhanced visual features F′ v1 The specific method is as follows: the text-image cross-attention network of the first feature enhancement module receives the text feature F from the text feature extraction module t , the text-image cross-module attention mechanism is used to align the text features and obtain the feature-enhanced text feature F′ t1 ; The image-text cross-attention network of the first feature enhancement module receives the visual features F from the visual feature extraction module v , using the image-text cross-modal attention mechanism to align visual features F v, and obtain the feature-enhanced visual feature F′ v1 . F′ v1 and F′ t1 The dimensions are all B×N text ×E text , where N text is the number of category texts, N text =6, E text is the dimension of text features, E text =256. v1 and F′ t1 Sent to the first multimodal decoder module.

[0132] 3.3.3.2 The first multimodal decoder module receives F′ from the first feature enhancement module v1 and F′ t1 The image query attention network of the first multimodal decoder module defines a B×N query ×E query The query vector set Q1, where N query is the number of query vector sets, N query =300, E query is the feature dimension of the query vector set, E query = 256. The image query attention network of the first multimodal decoder module defines a B×N query ×4 query vector set initial anchor box position A1. The image query attention network of the first multimodal decoder module is F′ v1 , Q1 and A1 perform image query attention operations, give the query vector set visual information, and obtain the query vector set with enhanced visual information The first multimodal decoder’s text query attention network is used to calculate F′ v1 , Perform text query attention operation with A1 to generate a query vector set Q1′ with enhanced image-text information. Send Q1′ to the first head network module of the first network branch.

[0133] 3.3.3.3 The first classification head network of the first head network module receives Q′1 from the first multimodal decoder module and receives F′ from the first feature enhancement module t1 , calculate Q′1 and F′ t1 The cosine similarity of query ×N text The first network branch predicts the object category The elements of represent the predicted probability values ​​of the query vector set in a certain category of text. Send to the prediction value fusion module without post-processing;

[0134] 3.3.3.4 The first classification head network As an indication, filter out the semantic representation set S with high confidence, and then perform weighted summation on S to obtain the semantic representation center set F s , F s The method of sending to the second head network module of the second network branch is:

[0135] 3.3.3.4.1 Initialize c = 0, indicating the cth category. Initialize F s An empty collection used to store C k The semantic representation center of each class.

[0136] 3.3.3.4.2 Set the screening threshold β to 0.3, screen the prediction values ​​of the c-th category, and obtain the semantic representation set (S) of the c-th category whose confidence is greater than β c :

[0137]

[0138] in, represents the prediction probability value of the first network branch in the cth category text of the query vector q, yes An element in , q is an element in Q1′, representing a query vector.

[0139] 3.3.3.4.3 Take the weighted average of the semantic representation and confidence of the cth category and get a size of B×E query The semantic representation center of the c-th category (F s ) c :

[0140]

[0141] in, Represents the semantic representation set of the cth category (S) c The first branch network predicts the probability value of a query vector s in (S) c The elements in is a vector, Indicates (S) c The sum of the predicted probability values ​​of the first branch network of each element in .

[0142] 3.3.3.4.4 If c<C k , let c = c + 1, and change (F s ) c Deposit F s Go to 3.3.3.4.2; if c = C k , indicating the semantic representation center set F s , Fs The second head network module sent to the second network branch (transmitted only during target detection system training, used to calculate visual-semantic contrast loss, not transmitted when detecting user input images), go to 3.3.3.5.

[0143] 3.3.3.5 The first regression head network of the first head network module calculates the predicted box position according to the query vector set Q1′ enhanced by the image-text information received from the first multimodal decoder module, and generates the first network branch to predict the object position The first network branch predicts the object position Sent to the prediction value fusion module without post-processing.

[0144] 3.3.3.6 The second feature enhancement module of the second network branch adopts the second branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F′ t2 and feature-enhanced visual features F′ v2 The method is as follows: the text-image cross-attention network of the second feature enhancement module receives F from the text feature extraction module t , the text-image cross-module attention mechanism is used to align the text features and obtain the second feature enhanced text feature F′ t2 ; The image-text cross-attention network of the second feature enhancement module receives the visual features F from the visual feature extraction module v , using the image-text cross-modal attention mechanism to align visual features F v , and obtain the visual feature F′ enhanced by the second feature v2 , F′ t2 and F′ v2 Sent to the second multimodal decoder module. F′ v2 and F′ t2 The dimensions are all B×N text ×E text .

[0145] 3.3.3.7 The second multimodal decoder module receives F′ from the second feature enhancement module v2 and F′ t2 The image query attention network of the second multimodal decoder module defines a B×N query ×E query The query vector set Q2. The image query attention network of the second multimodal decoder module defines a size of B×N query ×4 query vector set initial anchor box position A2. The image query attention network of the second multimodal decoder module is F′ v2, Q2 and A2, perform image query attention operation, give visual information to the query vector set, and obtain the query vector set with enhanced visual information The second multimodal decoder’s text query attention network is used to calculate F′ v2 , Perform text query attention operation with A2 to generate a query vector set with enhanced image-text information, denoted as Q2′. Send Q2′ to the second head network module of the second network branch.

[0146] 3.3.3.8 The second classification head network of the second head network module receives Q′2 from the second multimodal decoder module and receives F′ from the second feature enhancement module t2 , calculate Q′2 and F′ t2 The cosine similarity of query ×N text The second prediction category matrix The second classification head network receives F from the classification head network of the first branch network. s , calculate Q′2 and F s The cosine similarity of query ×N text The second visual-semantic prediction category The second regression head network of the second head network module receives Q′2 from the second multimodal decoder module and calculates the position of the predicted box as The dimension size is B×N query ×4. The loss function calculation module sent to the target detection system.

[0147] 3.3.4 The supervised truth value construction module in the loss function calculation module receives the second classification head network Receive the optimized position truth labels of the M images in Dt and the classification truth labels corresponding to the position truth labels from the image preprocessing module to construct the Nth b The supervised truth value building module uses the Hungarian algorithm adopted in the DETR paper (see the paper "Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers [C] / / European conference on computer vision. Cham: Springer International Publishing, 2020: 213-229." Carion N, Massa F et al.: End-to-end object detection with Transformer) to compare the classification truth labels in Dt with Perform one-to-one pairing for Generate the corresponding supervised truth value Size is B×N query ×N text ; D t The position truth labels and Perform one-to-one pairing for Generate the corresponding supervised truth value Size is B×N query ×4. Visual-semantic prediction categories is an auxiliary classification prediction, and its supervised true value is Will Sent to the loss value calculation module.

[0148] 3.3.5 The loss value calculation module in the loss function calculation module receives the supervision truth value construction module The loss value is calculated for the object category, visual-semantic prediction category and location information predicted by the second network branch and the supervised true value, and the network weight parameters of the second network branch are updated using gradient back propagation. Total loss function for:

[0149]

[0150] in YesP cls and The classification loss function used for calculation is Focal Loss (see the literature "Lin TY, Goyal P, Girshick R, et al. Focal loss for dense object detection [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 2980-2988." Lin TY, Goyal P et al.'s paper: Focal loss for dense object detection). It is a visual-semantic contrast loss function, using Focal Loss. It is the regression loss function, using L1 Loss (also known as mean absolute error). is the intersection over union loss function, using GIoU loss (see the literature "Rezatofighi H, Tsoi N, Gwak JY, et al. Generalized intersection over union: A metric and a loss for bounding box regression [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 658-666." Rezatofighi H, Tsoi N et al. paper: Generalized intersection over union: metric and loss for bounding box regression). is the weight of the classification loss function and is set to 1. is the weight of the visual-semantic contrast loss function and is set to 1. is the weight of the regression loss function, set to 5. iou is the weight of the intersection-over-union loss function and is set to 2.

[0151] 3.3.6 If N b ≤M / B, let N b =N b +1, go to 3.3.1; if N b >M / B, that is, the Nth epoch Step length training, let N b =1, go to 3.3.7.

[0152] 3.3.7 If N epoch =11, let learning_rate = learning_rate × 0.1, go to 3.3.8; if N epoch ≠11, directly switch to 3.3.8.

[0153] 3.3.8 If N epoch ≤11, let N epoch =N epoch +1, go to 3.3.1; if N epoch =12, save the model parameters of the second network branch at this time, indicating that the training is completed, and go to 3.3.9.

[0154] 3.3.9 Load the network weight parameters of the second network branch obtained by training into the second network branch of the target detection system to obtain the trained target detection system. The fourth step is to use the trained target detection system to perform category and position detection on the Test test set or the image to be detected input by the user, the method is:

[0155] 4.1 The image preprocessing module uses the image scaling and normalization method described in step 2.2.4 to optimize the Test test set or the image to be detected input by the user to obtain a normalized image set to be detected. The image set to be detected is sent to the main feature extraction module;

[0156] 4.2 Main feature extraction module uses the feature extraction method described in step 3.3.2 to extract visual features of the image set to be detected, and obtains the visual feature FF of the image set to be detected v ; For predefined category texts (including known category texts I text and unknown category text) to extract text features and obtain the predefined category text feature FF t ; The present invention defines "Vehicle", "Roadblock" and "Obstacle" as unknown categories. v and FF t Sent to the dual-branch feature fine-tuning module.

[0157] 4.3 The first feature enhancement module in the dual-branch feature fine-tuning module adopts the first branch cross-attention method in step 3.3.3.1 to perform v and FF t Perform feature enhancement and alignment to obtain the visual feature FF′ of the first branch feature enhancement of the image set to be detected v1 And feature-enhanced text features FF′ t1 , FF′ v1 and FF′ t1 Send it to the first multimodal decoder module; at the same time, the second feature enhancement module in the dual-branch feature fine-tuning module adopts the second branch cross-attention method in step 3.3.3.6 to FF v and FF t Perform feature enhancement and alignment to obtain the visual feature FF′ of the second branch feature enhancement of the image set to be detected v2 and text feature FF′ t2 , FF′ v2 and FF′ t2 Sent to the second multimodal decoder module.

[0158] 4.4 The first multimodal decoder module of the first network branch receives FF′ from the first feature enhancement module v1 and FF′ t1 , using the query vector set Q1 of the first network branch and the initial anchor box position A1 of the first network branch, the image query attention network of the first multimodal decoder module is FF′ v1 , Q1 and A1 perform image query attention operations, give the query vector set visual information, and obtain the query vector set with enhanced visual information The first multimodal decoder’s text query attention network for FF′ v1 , Perform text query attention operation with A1 to generate a query vector set QQ1′ with enhanced image-text information, and send QQ1′ to the first head network module of the first network branch; at the same time, the second multimodal decoder module of the second network branch receives FF′ from the second feature enhancement module v2 and FF′ t2 , using the query vector set Q2 of the second network branch and the initial anchor box position A1 of the first network branch, the image query attention network of the second multimodal decoder module is FF′ v2 , Q2 and A1, perform image query attention operation, give visual information to the query vector set, and obtain the query vector set with enhanced visual information The second multimodal decoder’s text query attention network is used to calculate FF′ v2 , And A1 perform text query attention operation to generate a query vector set with enhanced image-text information, recorded as QQ2′. Send QQ2′ to the second head network module of the second network branch. The reason why the initial anchor box position of the second network branch adopts the same initial anchor box position A1 as the first network branch during runtime is that the reason for the mismatch in the prediction values ​​of the two network branches is that the initial anchor boxes cannot correspond one to one. Therefore, using the same initial anchor box position A1 for the multimodal decoder modules of the two branches can align the order of the prediction result sets of the first and second network branches, making it convenient for the post-processing prediction value fusion module to fuse the prediction results of the first and second network branches. 4.5 The first classification head network of the first head network module receives QQ′1 from the first multimodal decoder module and receives FF′ from the first feature enhancement module. t1 , calculate QQ′1 and FF′ t1 The cosine similarity of query ×N text The predicted category of the first network branch The elements of represent the predicted probability values ​​of the multimodal query vector set of the first network branch in a certain category of text; the regression head network module of the first network branch receives QQi from the first multimodal decoder module, calculates the position of the prediction box, and obtains the predicted object position of the first network branch At the same time, the classification head network module of the second network branch receives QQ″2 from the second multimodal decoder module and receives FF′ from the second feature enhancement module. t2 , calculate QQ″2 and FF′ t2 The cosine similarity of query ×N text The predicted category of the second network branch The regression head network module of the second network branch receives QQ″2 from the second multimodal decoder module, calculates the position of the predicted box, and obtains the predicted object position of the second network branch Will Sends no post-processing predictions to the fusion module.

[0159] 4.6 Category-related prediction box screening module without post-processing prediction value fusion module receives from the first network branch Received from the second network branch Filtering operation fusion using category-related prediction boxes and Get the object position of the cth category of the image to be detected

[0160]

[0161] in, express The predicted box position of the cth class, express The predicted box position of the c-th class.

[0162] 4.7 Geometric mean fusion module without post-processing prediction value fusion module receives from the first network branch The second network branch receives Fusion using geometric mean operation and Get the object category score of the cth category of the image to be detected

[0163]

[0164] in, express The prediction score of the cth class, C k Indicates the category of known objects, C uk represents the category of the unknown object, and α is the geometric mean parameter, which is set to 0.3.

[0165] The fifth step is to obtain the object category and object position of the image set to be detected, and the target detection is completed.

[0166] The experiment is designed to verify the performance of the target detection system of the present invention. Since the images input by the user do not have true value labels, the performance of the target detection system cannot be evaluated. Therefore, the constructed test set Test is used for performance evaluation.

[0167] First, define the unknown class text word library of the intelligent driving scenario. Although a total of 29 categories have been defined in the CODA dataset, since the unknown category information of the evaluation dataset should be unknown, using the 29 categories in the CODA dataset as the unknown class text word library will leak information, which is not conducive to evaluating the performance of the present invention in general intelligent driving scenarios. Therefore, in addition to the 6 known classes in the SODA dataset, the present invention defines "Vehicle", "Roadblock" and "Obstacle" as unknown object categories for unknown class object detection.

[0168] Then, the target detection algorithm performance evaluation index is defined. Since the present invention is aimed at the open world environment of intelligent driving, it is necessary to detect both known classes and unknown classes, so it is necessary to evaluate the detection performance of known classes and unknown classes separately. For known class objects, AP 50 The indicator indicates that the average precision (AP) of IoU greater than 0.5. AP 50 The larger the value, the higher the detection accuracy of known objects. For unknown objects, since unknown objects cannot be fully defined and labeled, the recall rate is used for evaluation. 10 、Recall 20 、Recall 30 The three indicators represent the recall rate of unknown objects under 10, 20 and 30 prediction boxes. 10 、Recall 20 、Recall 30 The larger the value, the higher the detection accuracy of unknown objects. mRecall represents Recall 10 、Recall 20 and Recall 30 The mean of the three indicators.

[0169] The methods used for evaluation include GroundingDINO (Liu S, Zeng Z, Ren T, et al. Groundingdino: Marrying dino with grounded pre-training for open-set object detection [J]. arXiv preprint arXiv: 2303.05499, 2023.), fine-tuning the weight parameters of the full model of GroundingDINO (i.e., fine-tuning all the weight parameters in GroundingDINO) and the method proposed in the present invention. The comparison method is shown in Table 1, where GroundingDINO is the basic visual-language large model fine-tuned by the present invention, fine-tuning GroundingDINO is fine-tuning the weight parameters of the full model, which is the most direct fine-tuning method, and the LORA method fine-tuning GroundingDINO is to use the LORA method (Hu EJ, Wallis P, Allen-Zhu Z, et al. LoRA: Low-Rank Adaptation of Large Language Models [C] / / International Conference on Learning Representations.2021." (LoRA: Low-rank adaptive fine-tuning method for large language models) is fine-tuned, which is the mainstream method for fine-tuning large language models. The present invention implements and evaluates it on the multimodal large model GroundingDINO.

[0170] The experimental results evaluated on the CODA dataset are analyzed. The performance comparison of the target detection algorithm based on the CODA dataset is shown in Table 1. 50Compared with GroundingDINO, the present invention improves by 24.2 points, is 1.1 points higher than the method of directly fine-tuning the full model weight parameters of GroundingDINO, and is 7.1 points higher than fine-tuning GroundingDINO using the LORA method, which shows that the present invention can effectively fine-tune GroundingDINO and has a significant performance improvement in the recognition accuracy of known objects. In terms of the unknown class detection accuracy index mRecall, compared with GroundingDINO without fine-tuning, the present invention only drops by 0.8 points (fine-tuning the model is bound to lead to the forgetting of old knowledge, and the degree of decline in unknown class detection performance represents the degree of knowledge forgetting), which shows that the present invention only leads to a small amount of knowledge forgetting. Compared with fine-tuning the full model weight parameters of GroundingDINO, the present invention is 29.2 points higher, and compared with fine-tuning GroundingDINO using the LORA method, the present invention is 6.7 points higher, which shows that the present invention has obvious advantages in preventing knowledge forgetting in unknown object recognition.

[0171] Table 1

[0172]

[0173] The present invention is visualized and analyzed. Figure 3 As shown, GroundingDINO (first column), fine-tuned GroundingDINO (second column) and the present invention (third column) are visualized and analyzed on the CODA dataset, and the arrows indicate the areas missed or misdetected by GroundingDINO and fine-tuned GroundingDINO. The present invention is more accurate than GroundingDINO and fine-tuned GroundingDINO in detecting objects of known and unknown classes. In row (a), GroundingDINO missed cars and electric vehicles, and fine-tuned GroundingDINO missed warning poles; in row (b), GroundingDINO missed a row of vehicles on the roadside, and fine-tuned GroundingDINO missed traffic cones; in row (c), GroundingDINO missed pedestrians, and fine-tuned GroundingDINO missed carts; in row (d), GroundingDINO mistakenly detected the background area as a bus, and fine-tuned GroundingDINO missed a dog on the sidewalk. It can be clearly seen from the figure that for some common objects such as pedestrians and vehicles, the detection accuracy of the present invention is greatly improved. For some uncommon categories such as roadblocks, carts, dogs, etc., the present invention can also accurately detect, indicating that the present invention can adapt to complex intelligent driving scenarios, proving the effectiveness of the present invention.

Claims

1. A smart driving target detection method based on fine-tuning of a large vision-language model, Features The following steps are involved: The first step is to build a target detection system; the target detection system consists of an image preprocessing module, a main feature extraction module, a dual-branch feature fine-tuning module, a non-post-processing prediction value fusion module, and a loss function calculation module; The image preprocessing module is connected to the main feature extraction module and the loss function calculation module, receives the input image, performs enhancement processing during the training of the target detection system, sends the processed image and the corresponding true value label to the main feature extraction module, and sends the true value label to the loss function calculation module; When the target detection system performs target detection on the input image set to be detected, scaling and normalization processing are performed to obtain a processed image, and the processed image is sent to the main feature extraction module; The main feature extraction module is connected to the dual-branch feature fine-tuning module, and the main feature extraction module is composed of a visual feature extraction module and a text feature extraction module; the visual feature extraction module receives the processed image from the image preprocessing module, extracts the visual features, and sends the visual features to the dual-branch feature fine-tuning module; The visual feature extraction module is composed of the SwinTransformer neural network in GroundingDINO; the text feature extraction module extracts text features from the input category text and sends the text features to the dual-branch feature fine-tuning module; The text feature extraction module is composed of a Bert neural network; the input category text includes known category text and unknown category text; The dual-branch feature fine-tuning module is connected to the visual feature extraction module, the text feature extraction module and the non-post-processing prediction value fusion module. The dual-branch feature fine-tuning module is composed of two network branches with the same structure, which are respectively called the first network branch and the second network branch; The weight parameters of the first network branch are not updated, and the weight parameters of the second network branch are updated by gradient; The first network branch is connected to the visual feature extraction module, the text feature extraction module and the non-post-processing prediction value fusion module, and is composed of a first feature enhancement module, a first multimodal decoder module, and a first head network module; The first feature enhancement module is connected to the visual feature extraction module, the text feature extraction module, and the first multimodal decoder module, and is composed of an image-text cross-attention network and a text-image cross-attention network. The text-image cross-attention network of the first feature enhancement module receives text features from the text feature extraction module, receives visual features from the visual feature extraction module, aligns text features using a text-image cross-module attention mechanism, obtains feature-enhanced text features, and sends the feature-enhanced text features to the first multimodal decoder module; The image-text cross-attention network of the first feature enhancement module uses the image-text cross-modal attention mechanism to align image features, obtain feature-enhanced visual features, and send the feature-enhanced visual features to the first multimodal decoder module; The first multimodal decoder module is connected to the first feature enhancement module and the first head network module, and is composed of an image query attention network and a text query attention network, and receives feature-enhanced visual features and feature-enhanced text features from the first feature enhancement module; the image query attention network of the first multimodal decoder module defines a fixed number N query The dynamic weights of query is a positive integer, defining a size B×N query ×4 as the initial anchor box position of the query vector set, use the image query attention mechanism to use the feature-enhanced visual features to give the query vector set visual information, obtain the visual information enhanced query vector set, and send the visual information enhanced query vector set to the first head network module; the text query attention network of the first multimodal decoder module uses the text query attention mechanism to use the feature-enhanced text features to give the image information enhanced query vector set text information, obtain the image-text information enhanced query vector set, and send the image-text information enhanced query vector set to the first head network module; The first head network module is connected to the first feature enhancement module, the first multimodal decoder module, and the non-post-processing prediction value fusion module, and is composed of a first classification head network and a first regression head network; the first classification head network receives the query vector set enhanced with image-text information from the first multimodal decoder module, receives the feature-enhanced text features from the first feature enhancement module, calculates the cosine similarity of the query vector set enhanced with image-text information and the feature-enhanced text features, generates a first network branch prediction object category, and sends the first network branch prediction object category to the non-post-processing prediction value fusion module; The first head network module uses the generated first network branch predicted object category as an indication to screen out a high-confidence semantic representation set, then performs weighted summation on the semantic representation set to obtain a semantic representation center, and sends it to the second head network module of the second network branch; The first regression head network (composed of a fully connected layer) calculates the predicted box position according to the query vector set enhanced by the image-text information received from the first multimodal decoder module, generates the first network branch predicted object position, and sends the first network branch predicted object position to the non-post-processing prediction value fusion module; The second network branch is connected to the visual feature extraction module, the text feature extraction module and the non-post-processing prediction value fusion module, and is composed of a second feature enhancement module, a second multimodal decoder module, and a second head network module; The second feature enhancement module is connected to the visual feature extraction module, the text feature extraction module, and the second multimodal decoder module, and is composed of an image-text cross-attention network and a text-image cross-attention network. The text-image cross-attention network of the second feature enhancement module receives text features from the text feature extraction module, receives visual features from the visual feature extraction module, aligns text features using a text-image cross-module attention mechanism, obtains feature-enhanced text features, and sends the feature-enhanced text features to the second multimodal decoder module; The image-text cross-attention network of the second feature enhancement module uses the image-text cross-modal attention mechanism to align image features, obtain feature-enhanced visual features, and send the feature-enhanced visual features to the second multimodal decoder module; The second multimodal decoder module is connected to the second feature enhancement module and the second head network module, is composed of an image query attention network and a text query attention network, and receives feature-enhanced visual features and feature-enhanced text features from the second feature enhancement module; The image query attention network of the second multimodal decoder module defines a fixed number N query The dynamic weights of the image query vector set are used as the query vector set, and the image query attention mechanism is used to use the feature-enhanced visual features to give the query vector set visual information, so as to obtain the query vector set with enhanced visual information, and the query vector set with enhanced visual information is sent to the second head network module; the text query attention network of the second multimodal decoder module uses the text query attention mechanism to use the feature-enhanced text features to give the image information enhanced query vector set text information, so as to obtain the image-text information enhanced query vector set, and the image-text information enhanced query vector set is sent to the second head network module; The second head network module is connected to the first head network module, the second feature enhancement module, the second multimodal decoder module, and the no-post-processing prediction value fusion module, and is composed of a second classification head network and a second regression head network; the second classification head network receives the query vector set enhanced with image-text information from the second multimodal decoder module, receives the feature-enhanced text features from the second feature enhancement module, calculates the cosine similarity between the query vector set enhanced with image-text information and the feature-enhanced text features, generates a second network branch to predict the object category, and sends the second network branch prediction object category to the loss function calculation module and the no-post-processing prediction value fusion module; The second classification head network receives the semantic representation center from the first classification head network, calculates the cosine similarity between the query vector set enhanced with image-text information and the semantic representation center, obtains the visual-semantic prediction category, and sends the visual-semantic prediction category to the loss function calculation module; The second regression head network receives the query vector set enhanced with image-text information from the second multimodal decoder module, receives the text features enhanced with features from the second feature enhancement module, calculates the predicted box position for the query vector set enhanced with image-text information and the text features enhanced with features, generates the predicted object position of the second network branch, and sends the predicted object position of the second network branch to the loss function calculation module and the prediction value fusion module without post-processing; The post-processing-free prediction value fusion module is connected to the dual-branch feature fine-tuning module, and is composed of a geometric mean fusion module and a category-related prediction box screening module; the geometric mean fusion module receives the object category predicted by the first network branch from the first head network module, receives the object category predicted by the second network branch from the second head network module, and fuses the object category predicted by the first network branch and the object category predicted by the second network branch using a geometric mean operation to obtain known and unknown object categories; at the same time, the category-related prediction box screening module receives the object position predicted by the first network branch from the first head network module, receives the object position predicted by the second network branch from the second head network module, and uses the category-related prediction box screening operation to align the object position predicted by the first network branch and the object position predicted by the second network branch to obtain known and unknown object positions; The loss function calculation module is connected to the image preprocessing module and the dual-branch feature fine-tuning module, and is composed of a supervised truth value construction module and a loss value calculation module; the loss function calculation module is only used during the training process of the target detection system; the supervised truth value construction module receives the truth value label of the image preprocessing module and generates a supervised truth value for loss value calculation; The loss value calculation module is composed of a text center classification loss function, a visual-semantic classification loss function, a regression loss function, and an intersection-over-union loss function. It receives the supervised truth value of the supervised truth value module and the second network branch predicts the object category, the visual-semantic prediction category, and the second network branch predicts the object position, calculates the loss value, and uses gradient back propagation to update the network parameters of the second network branch of the target detection system. The second step is to use the image preprocessing module to build the training set and test set. The method is: 2.1 Collect intelligent driving target detection scene images as target detection datasets, and annotate each target detection scene image in the target detection dataset by: 2.1.1 Use the SODA intelligent driving scene dataset as the training set; the SODA dataset has 6 categories: pedestrians, cyclists, cars, trucks, trams, and tricycles, containing 10,000 images; use the CODA intelligent driving dataset as the test set; in addition to the 6 categories in the SODA dataset, the CODA dataset also has 23 rare categories, containing 4,884 images; during testing, put the 6 categories in the SODA dataset into the known object category set C k , put the 23 rare categories in the CODA dataset into the unknown object category set C uk ; Let the total number of images in the training set be M, let the total number of images in the test set be T, M is 10000, T is 4884; 2.2.2 The M images and corresponding labels in the training set are enhanced using the image preprocessing module to obtain the enhanced training set D t , we get an enhanced training set D consisting of M normalized images, M optimized position truth labels, and classification truth labels corresponding to the position truth labels. t ; 2.2.3 D t The optimized position truth labels of the M images and the classification truth labels corresponding to the position truth labels are sent to the loss function calculation module; 2.2.4 The image preprocessing module uses the image scaling normalization method to optimize the T images and corresponding labels in the test set, and obtains a new test set Test consisting of the normalized T images, the optimized position truth labels of the T images, and the classification truth labels corresponding to the position truth labels; The third step is to use the gradient back propagation method to train the dual-branch feature fine-tuning module in the target detection system built in the first step to obtain the model preservation parameters; the method is: 3.1 Initialize the network weight parameters of each module in the target detection system; use the weight parameters of the GroundingDINO model to initialize the weight parameters of the main feature extraction module and the dual-branch feature fine-tuning module; the initialization model parameters of the first network branch and the second network branch of the dual-branch feature fine-tuning module are consistent, where the model parameters of the first network branch are frozen, and the model parameters of the second network branch are learnable; 3.2 Set the training parameters of the target detection system, including setting the initial learning rate learning_rate to 0.0001, selecting AdamW as the model training optimizer, and the hyperparameter weight decay of the optimizer is 0.0001; the batch size of network training mini_batch_size is B=8; the training step epoch is 12; 3.3 Train the target detection system by calculating the difference between the predicted position and predicted category of the known object output by the current network and the true value as the loss value, and using the gradient direction propagation to update the network weight parameters until the 12th epoch is trained; the final network weight parameters are the model parameters of the 12th epoch; the specific method is as follows: 3.3.1 Initialize batch number N b =1, initialize step number N epoch = 1, define a batch from the optimized training set D t Read B images; the visual feature extraction module in the main feature extraction module extracts B images from D t Read the Nth b Batch, a total of B images, these B images are recorded as a matrix I of size B×H×W×3 train ; Where H and W represent the width and height of the input image respectively, and 3 represents the RGB channels of the image; the text feature extraction module in the main feature extraction module receives the category text input by the user, that is, the known category text I text , including Pedestrian, Cyclist, Car, Truck, Bus, Tricycle; 3.3.2 Main feature extraction module uses feature extraction method to extract I train Visual feature extraction is performed to obtain I train The visual features F v , to I text Perform text feature extraction to obtain text feature F t , the method is: the visual feature extraction module is used to extract I train Perform visual feature extraction to obtain visual feature F v At the same time, the text feature extraction module is used to extract I text Perform text feature extraction to obtain text feature F t , F t and F v Send to the dual-branch feature fine-tuning module; the specific method is as follows: 3.3.2.1 Visual Feature Extraction Module Extraction I train The visual features of v , in is the first scale visual feature map, size is is the second scale visual feature map, size is is the third scale visual feature map, size is is the fourth scale visual feature map, size is E = 256 is the visual feature dimension, and F v Send to the dual-branch feature fine-tuning module; 3.3.2.2 Text Feature Extraction Module Extraction I text The category text feature F t , F t Send to the dual-branch feature fine-tuning module; 3.3.3 The second network branch in the dual-branch feature fine-tuning module receives F from the main feature extraction module v and F t , perform feature enhancement and alignment operations to obtain the second network branch to predict the object category Visual-semantic prediction categories Location Information The second network branch receives F from the visual feature extraction network of the main feature extraction module. v , receives F from the text feature extraction network t , after the second feature enhancement module, the second multimodal decoder module, and the second head network module, the second network branch predicts the object category Visual-semantic prediction categories and location information Here’s how: 3.3.3.1 The first feature enhancement module of the first network branch adopts the first branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F t ' 1 and feature-enhanced visual features F′ v1 , F′ v1 and F t ' 1 Sent to the first multimodal decoder module; F′ v1 and F t ' 1 The dimensions are all B×N text ×E text , where N text is the number of category texts, N text is a positive integer, E text is the dimension of text features, E text is a positive integer; 3.3.3.2 The first multimodal decoder module receives F′ from the first feature enhancement module v1 and F t ' 1 ; The image query attention network of the first multimodal decoder module defines a B×N query ×E query The query vector set Q 1 , where N query is the number of query vector sets, E query is the feature dimension of the query vector set, E query is a positive integer; the image query attention network of the first multimodal decoder module defines a size of B×N query ×4 query vector set initial anchor box position A 1 ; The image query attention network of the first multimodal decoder module is F′ v1 , Q 1 and A 1 Perform image query attention operation, give visual information to the query vector set, and obtain a query vector set with enhanced visual information The first multimodal decoder’s text query attention network is used to calculate F′ v1 , and A 1 Perform text query attention operation to generate a query vector set Q with enhanced image-text information 1 ′; Q 1 'Sent to the first head network module of the first network branch; 3.3.3.3 The first classification head network of the first head network module receives Q′ from the first multimodal decoder module 1 , receives F from the first feature enhancement module t ' 1 , calculate Q′ 1 and F t ' 1 The cosine similarity of query ×N text The first network branch predicts the object category The elements of represent the predicted probability values ​​of the query vector set in a certain category of text. Send to the prediction value fusion module without post-processing; 3.3.3.4 The first classification head network As an indication, filter out the semantic representation set S with high confidence, and then perform weighted summation on S to obtain the semantic representation center set F s , F s The method of sending to the second head network module of the second network branch is: 3.3.3.4.1 Initialize c = 0, indicating the cth category; initialize F s An empty collection used to store C k The semantic representation center of each class; 3.3.3.4.2 Filter the prediction values ​​of the c-th category and obtain the semantic representation set (S) of the c-th category whose confidence is greater than β c : in, represents the prediction probability value of the first network branch in the cth category text of the query vector q, yes An element of Q 1 The elements in ′ represent a query vector; β is the screening threshold; 3.3.3.4.3 Take the weighted average of the semantic representation and confidence of the cth category and get a size of B×E query The semantic representation center of the c-th category (F s ) c : in, Represents the semantic representation set of the cth category (S) c The first branch network predicts the probability value of a query vector s in (S) c The elements in is a vector, Indicates (S) c The sum of the predicted probability values ​​of the first branch network of each element in; 3.3.3.4.4 If c <C k , let c = c + 1, and change (F s ) c Deposit F s Go to 3.3.3.4.2; if c = C k , indicating the semantic representation center set F s , F s Send to the second head network module of the second network branch, go to 3.3.3.5; 3.3.3.5 The first regression head network of the first head network module enhances the query vector set Q according to the image-text information received from the first multimodal decoder module 1 'Calculate the predicted box position and generate the first network branch to predict the object position The first network branch predicts the object position Send to the prediction value fusion module without post-processing; 3.3.3.6 The second feature enhancement module of the second network branch adopts the second branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F t ' 2 and feature-enhanced visual features F′ v2 , F t ' 2 and F′ v2 Sent to the second multimodal decoder module, F′ v2 and F t ' 2 The dimensions are all B×N text ×E text ; 3.3.3.7 The second multimodal decoder module receives F′ from the second feature enhancement module v2 and F t ' 2 ; The image query attention network of the second multimodal decoder module defines a B×N query ×E query The query vector set Q 2 ; The image query attention network of the second multimodal decoder module defines a B×N query ×4 query vector set initial anchor box position A 2 ; The image query attention network of the second multimodal decoder module is F′ v2 , Q 2 and A 2 , perform image query attention operation, give visual information to the query vector set, and obtain the query vector set with enhanced visual information The second multimodal decoder’s text query attention network is used to calculate F′ v2 , and A 2 Perform text query attention operation to generate a query vector set with enhanced image-text information, denoted as Q 2 ′; Q 2 'Sent to the second head network module of the second network branch; 3.3.3.8 The second classification head network of the second head network module receives Q′ from the second multimodal decoder module 2 , receives F from the second feature enhancement module t ' 2 , calculate Q′ 2 and F t ' 2 The cosine similarity of query ×N text The second prediction category matrix The second classification head network receives F from the classification head network of the first branch network. s , calculate Q′ 2 and F s The cosine similarity of query ×N text The second visual-semantic prediction category The second regression head network of the second head network module receives Q′ from the second multimodal decoder module 2 , the position of the predicted box is calculated as The dimension size is B×N query ×4; The loss function calculation module sent to the target detection system; 3.3.4 The supervised truth value construction module in the loss function calculation module receives the second classification head network Receive D from the image preprocessing module t The optimized position truth labels of the M images and the classification truth labels corresponding to the position truth labels are used to construct the Nth b The supervised truth value building module uses the Hungarian algorithm to convert D t The classification truth labels in Perform one-to-one pairing for Generate the corresponding supervised truth value Size is B×N query ×N text ; D t The position truth labels and Perform one-to-one pairing for Generate the corresponding supervised truth value Size is B×N query ×4; visual-semantic prediction category is an auxiliary classification prediction, and its supervised true value is Will Send to the loss value calculation module; 3.3.5 The loss value calculation module in the loss function calculation module receives the supervision truth value construction module The loss value is calculated for the object category predicted by the second network branch, the visual-semantic prediction category and the location information and the supervised true value, and the network weight parameters of the second network branch are updated by gradient back propagation; the total loss function for: in YesP cls and The classification loss function used for calculation is Focal Loss; It is a visual-semantic contrast loss function, using Focal Loss; It is the regression loss function, using L1 Loss; It is the intersection-over-union loss function, using GIoUloss; is the weight of the classification loss function, set to 1; is the weight of the visual-semantic contrast loss function, set to 1; is the weight of the regression loss function, set to 5; W iou is the weight of the intersection-over-union loss function, set to 2; 3.3.6 If N b ≤M / B, let N b =N b +1, go to 3.3.1; if N b >M / B, that is, the Nth epoch Step length training, let N b =1, go to 3.3.7; 3.3.7 If N epoch =11, let learning_rate = learning_rate × 0.1, go to 3.3.8; if N epoch ≠11, directly switch to 3.3.8; 3.3.8 If N epoch ≤11, let N epoch =N epoch +1, go to 3.3.1; if N epoch =12, save the model parameters of the second network branch at this time, indicating that the training is completed, and go to 3.3.9; 3.3.9 Load the network weight parameters of the second network branch obtained through training into the second network branch of the target detection system to obtain the trained target detection system; Step 4: Use the trained target detection system to perform category and position detection on the image to be detected input by the user, the method is: 4.1 The image preprocessing module uses the image scaling and normalization method described in step 2.2.4 to optimize the image to be detected input by the user to obtain a set of images to be detected consisting of a normalized image; the set of images to be detected is sent to the main feature extraction module; 4.2 Main feature extraction module uses the feature extraction method described in step 3.3.2 to extract visual features of the image set to be detected, and obtains the visual feature FF of the image set to be detected v ; For predefined known class category text I text Extract text features from unknown category text to obtain predefined category text features FF t ; FF v and FF t Send to the dual-branch feature fine-tuning module; 4.3 The first feature enhancement module in the dual-branch feature fine-tuning module uses the first branch cross-attention method in step 3.3.3.1 to enhance the FF v and FF t Perform feature enhancement and alignment to obtain the visual feature FF′ of the first branch feature enhancement of the image set to be detected v1 And feature-enhanced text features FF′ t1 , FF′ v1 and FF′ t1 Send it to the first multimodal decoder module; at the same time, the second feature enhancement module in the dual-branch feature fine-tuning module adopts the second branch cross-attention method in step 3.3.3.6 to FF v and FF t Perform feature enhancement and alignment to obtain the visual feature FF′ of the second branch feature enhancement of the image set to be detected v2 and text feature FF′ t2 , FF′ v2 and FF′ t2 Sending to a second multimodal decoder module; 4.4 The first multimodal decoder module of the first network branch receives FF′ from the first feature enhancement module v1 and FF′ t1 , using the query vector set Q of the first network branch 1 and the initial anchor box position a of the first network branch 1 , the image query attention network of the first multimodal decoder module is FF′ v1 , Q 1 and A 1 Perform image query attention operation, give visual information to the query vector set, and obtain a query vector set with enhanced visual information The first multimodal decoder’s text query attention network for FF′ v1 , and A 1 Perform text query attention operation to generate a query vector set QQ with enhanced image-text information 1 ′, QQ 1 ′ is sent to the first head network module of the first network branch; at the same time, the second multimodal decoder module of the second network branch receives the FF′ of the second feature enhancement module v2 and FF′ t2 , using the query vector set Q of the second network branch 2 and the initial anchor box position A of the first network branch 1 , the image query attention network of the second multimodal decoder module is FF′ v2 , Q 2 and A 1 , perform image query attention operation, give visual information to the query vector set, and obtain the query vector set with enhanced visual information The second multimodal decoder’s text query attention network is used to calculate FF′ v2 , and A 1 Perform text query attention operation to generate a query vector set with enhanced image-text information, denoted as QQ 2 '; QQ 2 'Sent to the second head network module of the second network branch; 4.5 The first classification head network of the first head network module receives QQ from the first multimodal decoder module 1 , receiving FF′ from the first feature enhancement module t1 , calculate QQ′ 1 and FF′ t1 The cosine similarity of query ×N text The predicted category of the first network branch The elements of represent the predicted probability values ​​of the multimodal query vector set of the first network branch in a certain category of text; the regression head network module of the first network branch receives QQ′ from the first multimodal decoder module 1 , calculate the position of the prediction box and get the predicted object position of the first network branch At the same time, the classification head network module of the second network branch receives QQ″ from the second multimodal decoder module 2 , receiving FF′ from the second feature enhancement module t2 , calculate QQ″ 2 and FF′ t2 The cosine similarity of query ×N text The predicted category of the second network branch The regression head network module of the second network branch receives QQ″ from the second multimodal decoder module 2 , calculate the position of the prediction box and get the predicted object position of the second network branch Will Send the prediction value fusion module without post-processing; 4.6 Category-related prediction box screening module without post-processing prediction value fusion module receives from the first network branch Received from the second network branch Filtering operation fusion using category-related prediction boxes and Get the object position of the cth category of the image to be detected in, express The predicted box position of the cth class, express The predicted box position of the cth class; 4.7 Geometric mean fusion module without post-processing prediction value fusion module receives from the first network branch The second network branch receives Fusion using geometric mean operation and Get the object category score of the cth category of the image to be detected in, express The prediction score of the cth class, C k Indicates the category of known objects, C uk represents the category of the unknown object, α is the geometric mean parameter; The fifth step is to obtain the object category and object position of the image set to be detected, and the target detection is completed.

2. The intelligent driving target detection method based on fine-tuning of a large vision-language model as claimed in claim 1, Features The known class texts mentioned in the first step refer to the categories in the training dataset, including Pedestrian and Car; the unknown class texts refer to the categories outside the training dataset, including Vehicle, Roadblock, and Obstacle.

3. The intelligent driving target detection method based on fine-tuning of a large visual-language model as claimed in claim 1, characterized in that the number of category texts N text =6, the dimension of text features E text =256; the number of query vector sets N query =300, the feature dimension of the query vector set is E query =256.

4. The method for intelligent driving target detection based on fine-tuning of a large vision-language model as claimed in claim 1, characterized in that the M images and corresponding labels in the training set are enhanced by an image preprocessing module in step 2.2.2 to obtain the enhanced training set D t The method is: 2.2.2.1 Let variable m = 1 and initialize the optimized training set D t is empty; let the mth image have N m Annotation boxes, N m ≥1, let N m The i-th annotation box among the annotation boxes is the position truth label, Represents the i-th annotation box The coordinates of the upper left corner, Represents the i-th annotation box The coordinates of the lower right corner; let N m The i-th annotation box among the annotation boxes Category for The classification truth label, 1≤i≤N s ; 2.2.2.2 Randomly flip the training set D t Flip the mth image in the image to get the mth flipped image, and the position truth label corresponding to the mth image Flip to get the true value label of the flipped position; the flip probability of the random flip method is set to 0.5; 2.2.2.3 The m-th flipped image is cropped using a random cropping method to obtain the m-th cropped image. The position truth label of the m-th image after flipping is cropped to obtain the cropped position truth label. The cropping probability of the random cropping method is set to 0.

5. 2.2.2.4 Use the random scaling method to scale the mth flipped image to obtain the mth scaled image, and scale the position truth value label after the mth image is cropped to obtain the optimized position truth value label; the scaling probability of the random scaling method is set to 0.5; 2.2.2.5 Use the image normalization operation to normalize the mth scaled image to obtain the mth normalized image. Put the mth normalized image, the optimized position truth label of the mth image, and the classification truth label corresponding to the position truth label into D t ; 2.2.2.6 If m≤M, go to 2.2.2.2; if m>M, obtain the enhanced training set D consisting of M normalized images, M image optimized position truth labels, and classification truth labels corresponding to the position truth labels t .

5. The method for intelligent driving target detection based on fine-tuning of a large vision-language model as described in claim 1, characterized in that the image preprocessing module in step 2.2.4 optimizes T images and corresponding labels in the test set using an image scaling normalization method, and obtains a new test set Test consisting of the normalized T images, the optimized position truth labels of the T images, and the classification truth labels corresponding to the position truth labels by: 2.2.4.1 Let variable t = 1 and initialize the new test set Test to be empty; 2.2.4.2 Use the scaling operation to scale the t-th image in the test set to a size of 800×1333 to obtain the t-th scaled image, and scale the position truth label of the t-th image to obtain the optimized position truth label; 2.2.4.3 Use image normalization operation to normalize the t-th scaled image to obtain the normalized t-th image, and put the normalized t-th image, the position truth label of the t-th image after scaling, and the classification truth label corresponding to the position truth label into a new test set Test; 2.2.4.4 If t≤T, go to 2.2.3.2; if t>T, obtain a new test set Test consisting of T normalized images, T image optimized position truth labels, and T image classification truth labels.

6. The intelligent driving target detection method based on fine-tuning of a large vision-language model as claimed in claim 1, Features 3.3.3.1 The first feature enhancement module of the first network branch adopts the first branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F′ t1 and feature-enhanced visual features F′ v1 The method is as follows: the text-image cross-attention network of the first feature enhancement module receives the text feature F from the text feature extraction module t , the text-image cross-module attention mechanism is used to align the text features and obtain the feature-enhanced text feature F′ t1 ; The image-text cross-attention network of the first feature enhancement module receives the visual features F from the visual feature extraction module v , using the image-text cross-modal attention mechanism to align visual features F v , and obtain the feature-enhanced visual feature F′ v1 .

7. The intelligent driving target detection method based on fine-tuning of a large vision-language model as claimed in claim 1, Features 3.3.3.6 The second feature enhancement module of the second network branch adopts the second branch cross-attention method to t and F v Perform feature enhancement and alignment to obtain feature-enhanced text features F′ t2 and feature-enhanced visual features F′ v2 The method is as follows: the text-image cross-attention network of the second feature enhancement module receives F from the text feature extraction module t , the text-image cross-module attention mechanism is used to align the text features and obtain the second feature enhanced text feature F′ t2 ; The image-text cross-attention network of the second feature enhancement module receives the visual features F from the visual feature extraction module v , using the image-text cross-modal attention mechanism to align visual features F v , and obtain the visual feature F′ enhanced by the second feature v2 .

8. The intelligent driving target detection method based on fine-tuning of a large vision-language model as claimed in claim 1, Features The screening threshold β described in step 3.3.3.4.2 is set to 0.3; the geometric mean parameter α described in step 4.7 is set to 0.3.

Images