Target positioning identification method fusing binocular vision and vision-language model

Abstract

The invention discloses a target positioning identification method fusing binocular vision and a vision-language model. The method comprises the following steps: 1) image acquisition; 2) calculating depth information of the target; 3) outputting a target coordinate frame; 4) target category prediction; and 5) outputting an identification result. According to the target positioning recognition method, binocular vision and a vision-language model are fused, the category of the target can be accurately judged while efficient distance calculation is guaranteed, and the environment perception ability of intelligent driving is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target positioning and recognition, and in particular to a target positioning and recognition method that integrates binocular vision and vision-language models. Background Art

[0002] In the current field of intelligent driving technology, binocular stereo vision technology has been widely used in target detection. By calculating the disparity map, the distance to the target in front can be effectively estimated. However, this technology has some limitations: 1) Lack of category recognition ability: It cannot provide specific category information of the target, which affects the decision-making accuracy of the autonomous driving system. 2) Insufficient semantic understanding: It is difficult to understand the semantic content of the scene, such as the inability to distinguish different types of targets such as pedestrians, vehicles or traffic signs. This affects the system's comprehensive grasp of complex traffic environments. 3) Poor adaptability to new targets: It is difficult to effectively identify new types of targets that do not appear in the training data, which reduces the system's adaptability in changing environments.

[0003] On the other hand, vision-language models such as CLIP, GLIP, and RegionCLIP have made significant progress in image understanding and cross-modal learning. Through large-scale pre-training, these models are able to effectively associate image and text information, demonstrating strong zero-shot learning capabilities. However, these models also face some challenges in practical applications: 1) Lack of depth information: Pure vision-language models cannot directly provide target distance information, which is crucial in autonomous driving scenarios. 2) Domain-specific adaptability: Pre-trained vision-language models may need to be adaptively adjusted for specific driving scenarios to improve their performance in practical applications.

[0004] Therefore, how to effectively combine the depth perception capability of binocular stereo vision with the semantic understanding capability of the vision-language model to accurately determine the category of the target while ensuring efficient distance calculation is a key issue that needs to be urgently addressed in current intelligent driving technology. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the existing technology, it is hoped to provide a target positioning and recognition method that integrates binocular vision and vision-language models, which can accurately determine the category of the target while ensuring efficient distance calculation, thereby effectively improving the environmental perception ability of intelligent driving.

[0006] The present invention provides a method for object positioning and recognition that integrates binocular vision and vision-language models, comprising the following steps:

[0007] 1) Image acquisition;

[0008] Get the left view image and the right view image of the front scene;

[0009] 2) Calculate the depth information of the target;

[0010] Process the left and right view images to generate a disparity map and calculate the depth information of the target;

[0011] 3) Output target coordinate frame;

[0012] The Depth-DETR model is used to output the target coordinate frame based on the RGB image and disparity map. The RGB image is the left view image or the right view image.

[0013] 4) Target category prediction;

[0014] Use vision-language model to predict the target category in RGB image;

[0015] 5) Output recognition results;

[0016] Output the depth information, target coordinate frame, and category information of the target in front.

[0017] Furthermore, in step 2), calculating the depth information of the target includes the following steps:

[0018] 21) Using binocular stereo vision algorithm to process the left view image and the right view image to generate a disparity map;

[0019] 22) Calculate the disparity d according to the disparity map;

[0020] d=x l -x r ;

[0021] Among them, x l and x r are the x-coordinates of the corresponding points in the left and right images respectively;

[0022] 23) Calculate the actual distance Z of the target;

[0023]

[0024] Where f is the focal length and B is the baseline length.

[0025] Furthermore, in step 3), outputting the target coordinate frame includes the following steps:

[0026] 31) Feature extraction;

[0027] A convolutional neural network is used to extract features from the RGB image to generate RGB features, and features are extracted from the disparity map to generate depth features, and the extracted RGB features and depth features are flattened;

[0028] 32) Position coding;

[0029] The flattened RGB features are added with position encoding to generate f rgb , add the flattened depth feature to the position code to generate f d ;

[0030] 33) Feature fusion;

[0031] Through the cross-modal attention mechanism, f rgb and f d Perform feature fusion;

[0032]

[0033] Where Q is the query matrix, take f d ; K is the key matrix, take f rgb ; V is the value matrix, take f rgb ;

[0034] d k is the dimension of the key vector;

[0035] 34) The fused features are further processed by the encoder of the self-attention mechanism to capture global context information;

[0036] 35) The decoder receives the processed features from the encoder and combines them with a set of predefined queries to generate an embedding vector;

[0037] 36) Use a multi-layer perceptron to classify the embedded vector and predict the bounding box, and output the target coordinate box.

[0038] Furthermore, in step 4), the loss function L of the Depth-DETR model is total for:

[0039] L total =α*(L cls +L bbox )+β*L BiCL ;

[0040] Among them, α and β are hyperparameters for adjusting the loss ratio;

[0041] L cls is the classification loss; L bbox is the bounding box regression loss; L BiCL is the two-way contrast loss;

[0042]

[0043] Where N is the number of true label boxes in the image;

[0044] The predicted index after the predicted label box in the image matches the true label box;

[0045] c i is the category of the true label box corresponding to the predicted label box;

[0046] For the The category probability of the predicted label box;

[0047] Bounding box regression loss L bbox To include L1 loss and GIoU loss;

[0048]

[0049] in, is the coordinate of the predicted label box; b i is the coordinate of the real label box;

[0050] || ||1 represents the L1 norm;

[0051]

[0052] Among them, the positive sample pair is the disparity map feature and the corresponding RGB map feature (f d 、f rgb );

[0053] Negative sample pairs are disparity map features and other randomly selected RGB map features (f d 、f′ rgb ), as well as RGB map features and other randomly selected disparity map features (f′ d 、f rgb );

[0054] The calculation formula of cosine similarity sim is:

[0055]

[0056] M is the number of samples, τ is the temperature hyperparameter;

[0057] It is the RGB image feature, including positive samples and negative samples;

[0058] is the disparity map feature, including positive samples and negative samples.

[0059] Furthermore, in step 4), target category prediction includes the following steps:

[0060] 41) Based on the target coordinate frame, the corresponding target area image is cropped from the RGB image;

[0061] 42) Input the cropped target area image into the visual encoder of the vision-language model to obtain the visual feature vector v of the area;

[0062] 43) Inputting the prepared target category description text into the text encoder of the vision-language model to obtain the corresponding text feature vector t;

[0063] 44) Calculate the similarity sim(v,t) between the visual feature vector v and each text feature vector t, and the target category description text corresponding to the text feature vector t with the highest similarity is the target category;

[0064]

[0065] Furthermore, the target category description text includes target category information to be identified in the following ways:

[0066] 431) Establish a vocabulary library containing various possible target category phrases;

[0067] 432) adding relevant attribute descriptions to each target category phrase;

[0068] 433) Adding information about possible scenarios in which the target may appear to the target category phrase;

[0069] 434) possible actions or states for each target category phrase;

[0070] 435) Prepare multiple different descriptions for the same target category phrase;

[0071] 436) Add subcategory description to a part of the target category phrase.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] (1) The target positioning and recognition method of the present invention uses the Depth-DETR model to process the RGB image and disparity map and output the target coordinate frame; the loss function of the Depth-DETR model introduces a bidirectional contrast loss. By constructing positive and negative sample pairs, the model is forced to learn the consistency of cross-modal features, practicing the semantic alignment between multimodal features, and greatly improving the accuracy of target recognition.

[0074] (2) The present invention combines binocular vision and vision-language models, and utilizes binocular stereo vision technology to accurately calculate the depth information of the target; by fusing depth information and semantic understanding, it can more accurately identify and locate the target object. Utilizing the zero-sample learning capability of the vision-language model, targets of unknown categories can be better identified, which is particularly important for handling complex and changing driving environments. It can flexibly adapt to different target categories and complex and changing driving scenarios, improving versatility and scalability. Since the reliance on large amounts of manually labeled data is reduced, the development and maintenance costs of the system can be significantly reduced. By combining visual and language information, it can provide a more comprehensive environmental understanding capability and provide richer information support for intelligent decision-making. The method of the present application is efficient, accurate, flexible and has strong adaptability. It is suitable for application scenarios in complex dynamic environments such as autonomous driving and intelligent transportation.

[0075] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0077] Figure 1 A flowchart of the target location and identification method;

[0078] Figure 2 This is the network structure diagram of the Depth-DETR model;

[0079] Figure 3 The process intention predicted for the target category; DETAILED DESCRIPTION

[0080] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0081] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0082] Please refer to Figures 1 to 3 , an embodiment of the present invention provides a method for target positioning and recognition by fusing binocular vision and vision-language model, comprising the following steps:

[0083] 1) Image acquisition;

[0084] Get the left view image and the right view image of the front scene;

[0085] 2) Calculate the depth information of the target;

[0086] 21) Using binocular stereo vision algorithm to process the left view image and the right view image to generate a disparity map;

[0087] 22) Calculate the disparity d according to the disparity map;

[0088] d=x l -x r ;

[0089] Among them, x l and x r are the x-coordinates of the corresponding points in the left and right images respectively;

[0090] 23) Calculate the actual distance Z of the target;

[0091]

[0092] Where f is the focal length and B is the baseline length;

[0093] 3) Output target coordinate frame;

[0094] The Depth-DETR model is used to output the target coordinate frame based on the RGB image and disparity map. The RGB image is the left view image or the right view image.

[0095] 31) Feature extraction;

[0096] A convolutional neural network is used to extract features from the RGB image to generate RGB features, and features are extracted from the disparity map to generate depth features, and the extracted RGB features and depth features are flattened;

[0097] 32) Position coding;

[0098] The flattened RGB features are added with position encoding to generate f rgb , add the flattened depth feature to the position code to generate f d ;

[0099] 33) Feature fusion;

[0100] Through the cross-modal attention mechanism, f rgb and f d Perform feature fusion;

[0101]

[0102] Where Q is the query matrix, take f d ; K is the key matrix, take frgb ; V is the value matrix, take f rgb ;

[0103] d k is the dimension of the key vector;

[0104] 34) The fused features are further processed by the encoder of the self-attention mechanism to capture global context information;

[0105] 35) The decoder receives the processed features from the encoder and combines them with a set of predefined queries to generate an embedding vector;

[0106] 36) Classify the embedded vector and predict the bounding box through a multi-layer perceptron, and output the target coordinate box;

[0107] Among them, the loss function L of the Depth-DETR model total for:

[0108] L total =α*(L cls +L bbox )+β*L BiCL ;

[0109] Among them, α and β are hyperparameters for adjusting the loss ratio;

[0110] L cls is the classification loss; L bbox is the bounding box regression loss; L BiCL is the two-way contrast loss;

[0111]

[0112] Where N is the number of true label boxes in the image;

[0113] The predicted index after the predicted label box in the image matches the true label box;

[0114] c i is the category of the true label box corresponding to the predicted label box;

[0115] For the The category probability of the predicted label box;

[0116] Bounding box regression loss L bbox To include L1 loss and GIoU loss;

[0117]

[0118] in, is the coordinate of the predicted label box; b iis the coordinate of the real label box;

[0119] || ||1 represents the L1 norm;

[0120]

[0121] Among them, the positive sample pair is the disparity map feature and the corresponding RGB map feature (f d 、f rgb );

[0122] Negative sample pairs are disparity map features and other randomly selected RGB map features (f d 、f′ rgb ), as well as RGB map features and other randomly selected disparity map features (f′ d 、f rgb );

[0123] The calculation formula of cosine similarity sim is:

[0124]

[0125] M is the number of samples, τ is the temperature hyperparameter;

[0126] It is the RGB image feature, including positive samples and negative samples;

[0127] is the disparity map feature, including positive samples and negative samples;

[0128] 4) Target category prediction;

[0129] Use vision-language model to predict the target category in RGB image;

[0130] 41) Based on the target coordinate frame, the corresponding target area image is cropped from the RGB image;

[0131] 42) Input the cropped target area image into the visual encoder of the vision-language model to obtain the visual feature vector v of the area;

[0132] 43) Inputting the prepared target category description text into the text encoder of the vision-language model to obtain the corresponding text feature vector t;

[0133] The target category description text includes the target category information to be identified in the following ways:

[0134] 431) Establish a vocabulary library containing various possible target category phrases;

[0135] 432) adding relevant attribute descriptions to each target category phrase;

[0136] 433) Adding information about possible scenarios in which the target may appear to the target category phrase;

[0137] 434) possible actions or states for each target category phrase;

[0138] 435) Prepare multiple different descriptions for the same target category phrase;

[0139] 436) Add subcategory description to a part of the target category phrase;

[0140] 44) Calculate the similarity sim(v,t) between the visual feature vector v and each text feature vector t, and the target category description text corresponding to the text feature vector t with the highest similarity is the target category;

[0141]

[0142] 5) Output recognition results;

[0143] Output the depth information, target coordinate frame, and category information of the target in front.

[0144] In this embodiment, the Depth-DETR model is designed to address two core challenges of target detection in intelligent driving scenarios: (1) The difficulty of multimodal information fusion: In traditional methods, RGB images provide rich texture and color information but lack accurate depth perception; while disparity maps can provide depth information, they lack semantic content. How to efficiently fuse the information of the two modalities to improve target positioning accuracy is a difficult problem that needs to be solved urgently. (2) The limitations of unknown category target detection: Existing supervised learning-based detection frameworks rely on a large amount of labeled data. When faced with categories that do not appear in the training set, the positioning ability is significantly reduced and the category cannot be determined.

[0145] Unlike conventional DETR, which relies solely on RGB image features, the Depth-DETR model introduces the disparity map as a depth prior. Through a cross-modality attention mechanism, it uses the disparity map features as a query to dynamically guide the RGB features to focus on the target area. This design significantly improves the robustness of target localization in complex scenarios (such as occlusion and low light).

[0146] The basic principle of the cross-modal attention mechanism is to use features from one modality as the query and features from the other modality as the key and value, and then use the attention mechanism to calculate the fused features. During this process, the attention mechanism calculates the similarity between the query and the key to obtain attention weights. These weights are used to weight features from different modalities, thereby achieving effective information fusion. Because the disparity map clearly defines which areas contain and do not contain objects, disparity map features are used as the query, and RGB image features are used as the key and value.

[0147] The Cross Modality Attention mechanism can effectively fuse information from different modalities (RGB and depth), allowing for more focus on areas with objects during target detection. Specifically, the benefits of fusing multi-modal information are:

[0148] Information complementarity: RGB images and disparity maps contain color and texture information and distance information respectively. The combination of the two can relatively make up for the shortcomings of a single modality, thereby enhancing the expressive ability of the overall features.

[0149] Improved accuracy: By integrating features from different modalities, the model can gain a more comprehensive understanding of complex scenarios, thereby improving the accuracy of target detection.

[0150] Optimized training: The cross-modal attention mechanism helps balance the importance of features from different modalities, thereby optimizing the training effect and performance of the model.

[0151] Enhanced robustness: The fusion of multimodal information makes the model more robust to adverse factors such as noise and occlusion, enabling more reliable target detection.

[0152] Compared with existing fusion methods (such as early fusion or late decision fusion), this application implements global context modeling through a self-attention encoder, combined with a bilateral contrastive loss (Bilateral Contrastive Loss), forcing the model to learn the semantic consistency of RGB image features and disparity image features, effectively solving the problems of information redundancy and conflict between modalities.

[0153] Through the semantic guidance of the vision-language model, the Depth-DETR model can utilize the prior knowledge of text descriptions in the positioning stage, reduce dependence on labeled data, improve the generalization ability of positioning targets of unknown categories, and break through the limitations of traditional supervised learning frameworks.

[0154] The core concept of the Depth-DETR model is to detect objects by fusing RGB images and disparity maps. It extracts features using a convolutional neural network (CNN) and applies a multimodal attention mechanism to the disparity and RGB image features using a transformer encoder. This allows the extracted features to incorporate information from both modalities, while focusing the model more closely on target areas. The fused features are then fed into a self-attention encoder to obtain global information, and a decoder to learn object boundaries. This allows for better utilization of scene depth information, thereby improving object detection accuracy.

[0155] The Depth-DETR model of this application introduces a bidirectional contrast loss based on the InfoNCE principle, and simultaneously calculates the contrast loss in two directions: disparity map to RGB image and RGB image to disparity map, to ensure the bidirectional alignment of information between modalities and avoid feature bias caused by single-direction supervision. By randomly sampling negative sample pairs, the model's robustness to noise and modal differences is enhanced, and the risk of overfitting is reduced. Traditional multimodal fusion methods usually optimize the loss of each modality independently, resulting in insufficient utilization of inter-modal information. This application dynamically balances the weights of classification, regression tasks, and contrastive learning tasks through hyperparameters to achieve collaborative optimization of multiple tasks.

[0156] In the Depth-DETR model's object detection algorithm, the loss function is a key component of model training and optimization. Bidirectional contrastive loss is designed for cross-modal scenarios. By using contrastive learning to reduce reliance on labeled data, the model can maintain high performance even with a small amount of labeled data, significantly reducing labeling costs.

[0157] The vision-language model is key to determining the target category in this application, primarily for determining categories not supported by the detection algorithm. Advanced vision-language models (such as CLIP, GLIP, or RegionCLIP) are used to calculate the similarity between visual and text features, enabling accurate classification of the target.

[0158] In the target category description text:

[0159] (1) Vocabulary: such as motor vehicles, non-motor vehicles, pedestrians, animals, buildings, traffic lights, traffic signs, fences, telephone poles, obstacles, etc.

[0160] (2) Attribute description: such as color, shape, size, etc., to increase the richness and differentiation of the description;

[0161] (3) Scene information, such as “cars on the road”, “pedestrians holding umbrellas”, etc.

[0162] (4) Action or state, such as "a moving car", "a standing pedestrian", etc.;

[0163] (5) Prepare multiple different descriptions for the same target category phrase to increase the robustness of the model;

[0164] (6) Subcategory description, such as "car", "truck", "bus", "commercial concrete truck", etc.

[0165] In this way, the language model is able to cover a wide range of target category information and improve the accuracy and flexibility of classification.

[0166] Example

[0167] (1) Dataset:

[0168] The 2d_object subset of the KITTI dataset is used. Jointly developed by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute America, the KITTI dataset is currently the world's largest evaluation dataset for computer vision algorithms in autonomous driving scenarios. KITTI contains real-world image data collected in urban, rural, and highway scenarios. The categories include Car, Cyclist, Don'tCare, Misc, Pedestrian, Person_sitting, Tram, Truck, and Van. Each image contains up to 15 vehicles and 30 pedestrians, with varying degrees of occlusion and truncation. The dataset is quite rich.

[0169] (2) Comparison method:

[0170] The currently popular advanced target detection methods RT-DETR, Yolov9m, Yolov10m, and Yolov11m are selected for comparison.

[0171] RT-DETR, a real-time object detection model developed by Baidu, is based on the Transformer architecture and aims to provide high accuracy and real-time performance. It uses an efficient hybrid encoder to process multi-scale features, eliminating the need for post-processing steps such as non-maximum suppression (NMS), making it suitable for scenarios that require fast processing.

[0172] YOLOv9m is a medium-sized model in the Ultralytics YOLOv9 series. The YOLO series is known for its speed and accuracy. YOLOv9m strikes a balance between model complexity and detection performance, reducing parameters and computation while improving accuracy.

[0173] YOLOv10m is a mid-size model in the YOLOv10 series. It introduces a dual-label assignment strategy to enhance supervision and deployment efficiency. It outperforms previous YOLO versions in accuracy and efficiency, making it suitable for real-time applications.

[0174] YOLOv11m is the latest medium-sized model in the YOLOv11 series, continuing the series' improvement trend. Architectural enhancements such as the C3k2 block and SPPF component improve feature extraction and performance.

[0175] (3) Measurement indicators:

[0176] Compare the recall rate, precision rate and F1 value of the method of this application with those of existing methods.

[0177] TP, True Positive, means the prediction is correct, it is a positive sample and is predicted as a positive sample;

[0178] TN, True Negative, means the prediction is correct, it is a negative sample and is predicted as a negative sample;

[0179] FP, False Positive, indicates a prediction error, which is a negative sample but predicted as a positive sample; the larger the value, the higher the false positive;

[0180] FN, False Negative, indicates a prediction error, where a sample is positive but predicted as a negative sample. The larger the value, the higher the chance of missed detection.

[0181] Recall = TP / (TP+FN);

[0182] Precision = TP / (TP+FP);

[0183] F1 value: F1 value is a comprehensive indicator to measure the accuracy of the detection algorithm. It is the harmonic mean of precision and recall.

[0184] The calculation formula is: F1 value = 2*Precision*Recall / (Precision+Recall).

[0185] (4) Comparison of detection performance:

[0186] Table 1 Detection performance comparison

[0187] Recall Precision F1 RT-DETR 0.922 0.905 0.913 Yolov9m 0.929 0.926 0.927 Yolov10m 0.931 0.938 0.934 Yolov11m 0.935 0.947 0.941 Depth-DETR 0.947 0.972 0.959

[0188] In terms of inference speed, on the NVIDIA 4090 GPU, FP32, with an input image size of 640*640, the tested inference speed is 46fps, meeting the real-time requirements.

[0189] As shown in Table 1, Depth-DETR achieves improved object detection accuracy by fusing features from RGB images and disparity maps and leveraging cross-modal attention and self-attention mechanisms. This enables the model to more comprehensively understand the scene. Leveraging the zero-shot learning capabilities of vision-language models, the system is able to better identify objects of unknown categories, thereby improving classification accuracy and flexibility.

[0190] The specific analysis is as follows:

[0191] Reasons for improved recall:

[0192] Depth-guided positioning: The Depth-DETR model uses binocular stereo vision to generate disparity maps, providing precise depth information. Combined with a cross-modal attention mechanism, this allows the model to focus more on the target area. For example, in occluded scenes, depth features can help distinguish foreground from background, reducing missed detections and improving recall.

[0193] Insufficient comparison methods: The YOLO series and RT-DETR rely only on RGB data, lack depth perception capabilities, and are easily affected by lighting changes or occlusions, resulting in some targets not being detected.

[0194] Reasons for improved precision:

[0195] Zero-shot classification for vision-language models: Leveraging the semantic alignment capabilities of models like CLIP, the Depth-DETR model can accurately distinguish between target categories. For example, the fine-grained features of text descriptions (e.g., "truck" vs. "bus") reduce false positives when encountering unknown categories. Bidirectional contrast loss optimization: Enforces alignment of RGB image features with disparity map features, reducing false positives due to interference such as shadows.

[0196] Insufficient comparison methods: The YOLO series and RT-DETR rely on supervised learning, which can easily force unknown categories to be classified as known categories. If the training set does not cover certain categories, it is easy to cause false detection.

[0197] (5) Ablation experiment:

[0198] Table 2. Comparison of ablation experiments

[0199] Configuration Recall Precision F1 Complete model 0.947 0.972 0.959 No two-way contrast loss 0.928 0.951 0.939 Only RGB data 0.901 0.918 0.909

[0200] Table 2 compares the performance impact of each module of the Depth-DETR model; among them, only RGB data is used to represent the input. Only RGB data is used without depth information, and the transformer encoder module does not perform cross attention but self-attention, and there is no bidirectional contrast loss. Not using bidirectional contrast loss means Figure 2 Remove the auxiliaryloss part.

[0201] Parameter sensitivity analysis:

[0202] Adjusting the loss weight hyperparameters α and β, when α = 0.7 and β = 0.3, F1 can reach a maximum value of 0.959. A β that is too high (> 0.5) will lead to insufficient weighting of the localization and classification losses, while a β that is too low (< 0.1) will weaken the cross-modal alignment effect, both of which will affect the final results.

[0203] The method of the present application combines binocular stereo vision (for obtaining depth and target coordinate information) with a vision-language model (for target classification), solving the problem that traditional binocular vision technology cannot directly identify target categories.

[0204] Multimodal learning to locate targets: The Depth-DETR model fuses disparity map features with RGB map features by introducing depth information, and strengthens the network's attention to target areas by introducing auxiliary loss (bidirectional contrast loss), ultimately achieving more efficient and accurate target detection.

[0205] Multimodal learning to determine categories: The visual-language model can automatically identify the category of the target through comparative learning of text and images without the need for a large amount of labeled data, which greatly improves the efficiency and accuracy of classification.

[0206] Improving the perception capabilities of intelligent driving systems: Through the combined method of this application, not only can the distance of the target be determined, but the category of the target can also be accurately identified, providing more comprehensive perception results, thereby optimizing autonomous driving decisions and ensuring driving safety.

[0207] Throughout this specification, terms such as "one embodiment" or "some embodiments" mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0208] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A target location and recognition method integrating binocular vision and vision-language model, characterized in that: The steps include: 1) Image acquisition; Get the left view image and the right view image of the front scene; 2) Calculate the depth information of the target; Process the left and right view images to generate a disparity map and calculate the depth information of the target; 3) Output target coordinate frame; The Depth-DETR model is used to output the target coordinate frame based on the RGB image and disparity map. The RGB image is the left view image or the right view image. 4) Target category prediction; Use vision-language model to predict the target category in RGB image; 5) Output recognition results; Output the depth information, target coordinate frame, and category information of the target in front.

2. The target positioning and recognition method integrating binocular vision and vision-language model according to claim 1, characterized in that: In step 2), calculating the depth information of the target includes the following steps: 21) Using binocular stereo vision algorithm to process the left view image and the right view image to generate a disparity map; 22) Calculate the disparity d according to the disparity map; d=x l -x r ; Among them, x l and x r are the x-coordinates of the corresponding points in the left and right images respectively; 23) Calculate the actual distance Z of the target; Where f is the focal length and B is the baseline length.

3. The target positioning and recognition method integrating binocular vision and vision-language model according to claim 1, characterized in that: In step 3), outputting the target coordinate frame includes the following steps: 31) Feature extraction; A convolutional neural network is used to extract features from the RGB image to generate RGB features, and features are extracted from the disparity map to generate depth features, and the extracted RGB features and depth features are flattened; 32) Position coding; The flattened RGB features are added with position encoding to generate f rgb , add the flattened depth feature to the position code to generate f d ; 33) Feature fusion; Through the cross-modal attention mechanism, f rgb and f d Perform feature fusion; Where Q is the query matrix, take f d ; K is the key matrix, take f rgb ; V is the value matrix, take f rgb ; d k is the dimension of the key vector; 34) The fused features are further processed by the encoder of the self-attention mechanism to capture global context information; 35) The decoder receives the processed features from the encoder and combines them with a set of predefined queries to generate an embedding vector; 36) Use a multi-layer perceptron to classify the embedded vector and predict the bounding box, and output the target coordinate box.

4. The target positioning and recognition method integrating binocular vision and vision-language model according to claim 3, characterized in that: In step 3), the loss function L of the Depth-DETR model is total for: L total =α*(L cls +L bbox )+β*L BiCL ; Among them, α and β are hyperparameters for adjusting the loss ratio; L cls is the classification loss; L bbox is the bounding box regression loss; L BiCL is the two-way contrast loss; Where N is the number of true label boxes in the image; The predicted index after the predicted label box in the image matches the true label box; c i is the category of the true label box corresponding to the predicted label box; For the The category probability of the predicted label box; Bounding box regression loss L bbox To include L1 loss and GIoU loss; in, is the coordinate of the predicted label box; b i is the coordinate of the real label box; || ||1 represents the L1 norm; Among them, the positive sample pair is the disparity map feature and the corresponding RGB map feature (f d 、f rgb ); Negative sample pairs are disparity map features and other randomly selected RGB map features (f d 、f′ rgb ), as well as RGB map features and other randomly selected disparity map features (f d ′、f rgb ); The calculation formula of cosine similarity sim is: M is the number of samples, τ is the temperature hyperparameter; It is the RGB image feature, including positive samples and negative samples; is the disparity map feature, including positive samples and negative samples.

5. The target positioning and recognition method integrating binocular vision and vision-language model according to claim 1, characterized in that: In step 4), target category prediction includes the following steps: 41) Based on the target coordinate frame, the corresponding target area image is cropped from the RGB image; 42) Input the cropped target area image into the visual encoder of the vision-language model to obtain the visual feature vector v of the area; 43) Inputting the prepared target category description text into the text encoder of the vision-language model to obtain the corresponding text feature vector t; 44) Calculate the similarity sim(v,t) between the visual feature vector v and each text feature vector t, and the target category description text corresponding to the text feature vector t with the highest similarity is the target category; 6. The target positioning and recognition method integrating binocular vision and vision-language model according to claim 5, characterized in that: The target category description text includes target category information to be identified in the following ways: 431) Establish a vocabulary library containing various possible target category phrases; 432) adding relevant attribute descriptions to each target category phrase; 433) Adding information about possible scenarios in which the target may appear to the target category phrase; 434) possible actions or states for each target category phrase; 435) Prepare multiple different descriptions for the same target category phrase; 436) Add subcategory description to a part of the target category phrase.

Images