3D target detection method and system based on binocular vision and open vocabularies

Abstract

The invention discloses a 3D target detection method and system based on binocular vision and open vocabularies, and relates to the technical field of target detection. The method comprises the following steps: calculating pixel features of each image by adopting an open vocabulary 2D detection model; pixel features of the left view and the right view are aggregated to each three-dimensional point; obtaining a parallax estimation map based on the pixel features of the left and right views; introducing a time aggregation guide parallax consistency constraint to obtain a depth parallax estimation map; converting the depth parallax estimation map into a 3D point cloud, and extracting features to obtain 3D features; and carrying out correlation integration on the text features, the 2D fusion feature vectors and the 3D features by utilizing object-level image guide detection loss, object-level cross comparison alignment and object-level self-comparison learning to obtain a three-dimensional target detection model, realizing three-dimensional target detection and distinguishing different target objects. According to the invention, accurate detection and distinguishing of the target can be realized, so that the flexibility and universality of 3D target identification are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a 3D target detection method and system based on binocular vision and open vocabulary. Background Art

[0002] With the rapid development of technologies such as autonomous driving, robotics, and augmented reality, 3D object detection, as one of the most important perception tasks in the computer vision community, has made great progress in recent years, especially after the emergence of deep learning based on neural networks. 3D object detection aims to automatically detect and identify three-dimensional objects in a scene from images or point cloud data, which is especially important in applications such as autonomous driving and robotic perception. Unlike traditional 2D object detection, 3D object detection requires accurate estimation of the object's three-dimensional position, size, and posture, so it is more challenging.

[0003] Some researchers have focused on extracting accurate 3D information from LiDAR data for 3D object detection. However, LiDAR data is expensive to collect and annotate, and has a limited sensing range in some cases. In contrast, vision-based methods have attracted increasing attention in recent years. Binocular stereo images have different viewpoints, and image pairs can reconstruct depth information through pixel-to-pixel correspondence or stereo geometry, making them better at detecting 3D objects without the need to introduce expensive LiDAR data.

[0004] However, traditional 3D object detection is trained with the supervision of benchmark datasets designed for specific tasks, and usually relies on pre-defined target categories (e.g., cars, pedestrians, traffic signs, etc.); these categories are fixed and cannot adapt to new, unseen categories. Object detection has many applications in image understanding, autonomous driving, obstacle avoidance, etc., but most of its methods only detect objects with fixed categories, which is a long-standing and inherent challenge. This challenge limits the ability and applicability of open scenes, especially in complex and changing background conditions. It is particularly difficult to achieve real-time and efficient target recognition. In addition, among 3D detection methods, compared with methods using sensors such as lidar, methods based on binocular vision are low-cost and easy to deploy, suitable for a wide range of applications, especially in low-cost such as small drones, robots, etc. In some scenes with fast changes such as fast-moving objects, binocular vision is more suitable than other sensors. However, in scenes with long distances or less texture, binocular vision may face the problem of inaccurate depth estimation, especially when there is no obvious visual difference between the object and the background, errors will occur in disparity calculation. Summary of the invention

[0005] In order to overcome the above problems or at least partially solve the above problems, the present invention provides a 3D target detection method and system based on binocular vision and open vocabulary, which optimizes image feature extraction and target classification strategies by introducing binocular stereo vision technology and a powerful open vocabulary visual language model, realizes accurate detection and distinction of targets, thereby improving the flexibility and versatility of 3D target recognition.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a 3D object detection method based on binocular vision and open vocabulary, comprising the following steps:

[0008] The pixel features of each image are calculated using a pre-trained open vocabulary 2D detection model to obtain the pixel features of the corresponding left and right views;

[0009] Aggregate the pixel features of the left and right views to each 3D point to form a 2D fused feature vector for each point;

[0010] Obtain a disparity estimation map based on pixel features of the left and right views;

[0011] Based on the disparity estimation map, temporal aggregation is introduced to guide the disparity consistency constraint to obtain the depth disparity estimation map;

[0012] Convert the depth disparity estimation map into a 3D point cloud;

[0013] A pre-trained 3D convolutional network is used to extract features from the 3D point cloud geometric information to obtain 3D features;

[0014] The text features are associated and integrated with the 2D fusion feature vector and 3D features using object-level image-guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning to obtain a 3D object detection model.

[0015] Three-dimensional target detection is realized based on the three-dimensional target detection model, and different target objects are distinguished.

[0016] The present invention introduces binocular stereo vision technology and a powerful visual language model to optimize image feature extraction and target classification strategies, thereby improving the flexibility and versatility of 3D target recognition and providing strong technical support for applications in various fields.

[0017] Based on the first aspect, further, the above-mentioned temporal aggregation guided disparity consistency constraint is a cross-view disparity consistency constraint between the teacher model and the student model.

[0018] Based on the first aspect, further, the method for obtaining a depth disparity estimation map by introducing a temporal aggregation-guided disparity consistency constraint based on the disparity estimation map comprises the following steps:

[0019] Use the pre-trained model to initialize the student model and the teacher model;

[0020] The teacher model is used to generate pseudo-annotations for unlabeled images and used to progressively enhance the student model which is updated via standard gradient descent; in return, the teacher model receives temporal updates from the student model via an exponential moving average.

[0021] Perform data augmentation on labeled images and unlabeled images and input them into the teacher model and student model respectively;

[0022] The student model is trained and optimized on limited annotated images and a large number of pseudo-annotated images;

[0023] The output disparity map of the teacher model is used to temporally aggregate the knowledge of the student model to guide the disparity estimation of the student model and achieve temporal aggregation-guided disparity consistency.

[0024] Based on the first aspect, further, the method for associating and integrating text features with 2D fusion feature vectors and 3D features by using object-level image guided detection loss, object-level cross-contrast alignment and object-level self-contrast learning includes the following steps:

[0025] The detection head uses object-level IoU-guided detection loss and introduces distance IoU loss for bounding box regression.

[0026] Introducing object-level cross-contrast alignment task to improve the performance of the cross-fusion module;

[0027] Self-contrastive loss is used to motivate the model to capture features that distinguish objects, and the IoU threshold is used to select positive and negative samples for self-contrastive learning; by optimizing the self-contrastive loss, the fusion module can distinguish different objects.

[0028] Based on the first aspect, further, the 3D object detection method based on binocular vision and open vocabulary includes:

[0029] Based on the predictions b p and the ground truth value b gt , calculate the IoU between them, and have the following regression loss:

[0030]

[0031] Where c is the diagonal length of the smallest enclosing box covering the two boxes, ρ 2 (·,·) is the Euclidean distance;

[0032] The object-level image guided detection loss OID is obtained using DIoU loss and label smoothing. The OID loss expression is as follows:

[0033]

[0034] in,

[0035] Among them, y p is the weight of proposal p in the soft label, y p According to the total number of proposals K, δ is a given pre-set knowledge value threshold and ε is a weight factor.

[0036] Based on the first aspect, further, the method for introducing the object-level cross-comparison alignment task comprises the following steps:

[0037] In the training phase, the target detection box of the real target is introduced, and all the prediction boxes with IoU greater than the preset threshold are selected as positive samples, and the remaining prediction boxes are used as negative samples;

[0038] Align features of positive samples with language embeddings and push features of negative samples with contrastive loss to achieve cross-modal understanding;

[0039] There is the following contrast loss as the loss function for the OCC task:

[0040]

[0041] Among them, H p represents the embedding of proposal p, T represents the embedding of text, s(·,·) represents the similarity scoring function used to measure the similarity between two types of features, using P pos ={p|IoU(b p ,b gt )≥δ} as the proposal set containing positive samples, P neg ={p|IoU(b p ,b gt )<δ} is taken as the proposal set containing negative samples, IoU(·,·) represents the IoU score between two bounding boxes, and δ is the IoU threshold.

[0042] Based on the first aspect, further, the self-contrast loss function is:

[0043] Among them, H p Denote the embedding of proposal p, using P pos ={p|IoU(b p ,b gt )≥δ} as the proposal set containing positive samples, Pneg ={p|IoU(b p ,b gt )<δ} is taken as the proposal set containing negative samples, IoU(·,·) represents the IoU score between two bounding boxes, and δ is the IoU threshold.

[0044] In a second aspect, the present invention provides a 3D target detection system based on binocular vision and open vocabulary, comprising a 2D feature extraction module, a 2D feature fusion module, a disparity estimation map acquisition module, a depth map acquisition module, a 3D point cloud conversion module, a 3D feature extraction module, a feature fusion module and a target detection module, wherein:

[0045] A 2D feature extraction module for computing pixel features of each image using a pre-trained open vocabulary 2D detection model to obtain pixel features of corresponding left and right views;

[0046] 2D feature fusion module, used to aggregate the pixel features of the left and right views to each 3D point to form a 2D fused feature vector for each point;

[0047] A disparity estimation map acquisition module, used to acquire a disparity estimation map based on pixel features of left and right views;

[0048] A depth map acquisition module is used to introduce a time aggregation-guided disparity consistency constraint based on a disparity estimation map to obtain a depth disparity estimation map;

[0049] 3D point cloud conversion module, used to convert the depth disparity estimation map into 3D point cloud;

[0050] A 3D feature extraction module is used to extract features from 3D point cloud geometric information using a pre-trained 3D convolutional network to obtain 3D features;

[0051] A feature fusion module is used to associate and integrate text features with 2D fusion feature vectors and 3D features using object-level image-guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning to obtain a three-dimensional object detection model;

[0052] The target detection module is used to realize three-dimensional target detection based on the three-dimensional target detection model and to distinguish different target objects.

[0053] This system optimizes image feature extraction and target classification strategies through the cooperation of multiple modules including 2D feature extraction module, 2D feature fusion module, disparity estimation map acquisition module, depth map acquisition module, 3D point cloud conversion module, 3D feature extraction module, feature fusion module and target detection module, by introducing binocular stereo vision technology and powerful visual language model, thereby improving the flexibility and versatility of 3D target recognition and providing strong technical support for applications in various fields.

[0054] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0056] The present invention has at least the following advantages or beneficial effects:

[0057] 1. Improve the fusion accuracy of 3D points and image features: By combining 2D and 3D feature extraction methods, the present invention can more accurately align 3D point clouds with pixel features in images. By integrating back-projection and binocular views, the complementarity of image information and geometric information is fully utilized, significantly improving the representation capability and detection accuracy of 3D points. This hybrid feature strategy not only enhances the robustness of the model, but also effectively copes with 3D target detection tasks in complex environments.

[0058] 2. Solve the depth estimation problem under limited annotated data: The pseudo-supervision strategy proposed in this paper significantly improves the stability and accuracy of depth estimation by using disparity consistency constraints and cross-view agreement strategies under limited annotated data. By guiding the base model (student model) to perform depth estimation, and combining temporal aggregation and disparity consistency constraints, the model can continuously learn and accurately estimate depth without a large amount of annotated data, thereby achieving more efficient 3D target positioning.

[0059] 3. Enhance the target detection capability of the 3D visual language model: This invention significantly improves the performance of the 3D target detection model in multimodal data fusion by introducing a visual language model and designing innovative tasks such as object-level image-guided detection loss (OID), object-level cross-contrast alignment (OCC), and object-level self-contrast learning (OSC). Through these tasks, the model can not only accurately detect 3D targets, but also effectively distinguish different objects. The fusion of the visual language model further improves the generalization ability of the model, making it more adaptable and recognizable in a variety of application scenarios.

[0060] 4. Efficient use of unlabeled data: The present invention uses a pseudo-supervised learning method to make full use of unlabeled data for training when labeled data is scarce. By generating high-quality pseudo-annotations and combining self-supervised learning and cross-view constraints, the learning efficiency and accuracy of the model can be improved without the support of a large amount of manually labeled data. This not only reduces the labeling cost, but also enables the model to be effectively trained on large-scale unlabeled data, thereby improving the practical application value of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 It is an overall block diagram of a 3D object detection method based on binocular vision and open vocabulary according to an embodiment of the present invention;

[0063] Figure 2 is a structural diagram of depth disparity estimation in an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of a cross-modal alignment method in an embodiment of the present invention;

[0065] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0066] Description of reference numerals: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0068] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0070] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0071] In the description of the embodiments of the present invention, "plurality" means at least 2.

[0072] Example:

[0073] like Figure 1 As shown, in a first aspect, an embodiment of the present invention provides a 3D target detection method based on binocular vision and open vocabulary, comprising the following steps:

[0074] S1. Calculate the pixel features of each image using a pre-trained open vocabulary 2D detection model to obtain the pixel features of the corresponding left and right views;

[0075] S2, aggregate the pixel features of the left and right views to each 3D point to form a 2D fusion feature vector for each point;

[0076] S3, obtaining a disparity estimation map based on pixel features of the left and right views;

[0077] S4. Based on the disparity estimation map, a temporal aggregation guided disparity consistency constraint is introduced to obtain a depth disparity estimation map; the temporal aggregation guided disparity consistency constraint is a cross-view disparity consistency constraint between the teacher model and the student model.

[0078] Furthermore, it includes: initializing the student model and the teacher model using a pre-trained model; generating pseudo annotations for unlabeled images using the teacher model, and using them to gradually enhance the student model updated by standard gradient descent; in return, the teacher model is temporally updated from the student model through an exponential moving average; data enhancement is performed on labeled images and unlabeled images, and they are input into the teacher model and the student model respectively; the student model is trained and optimized on a limited number of labeled images and a large number of pseudo-labeled images; and the knowledge of the student model is temporally aggregated using the output disparity map of the teacher model to guide the disparity estimation of the student model, thereby achieving temporal aggregation-guided disparity consistency.

[0079] In some embodiments of the present invention, Figure 2 The depth estimation architecture is shown in Figure 1. Temporal aggregation is introduced to guide disparity consistency, i.e., cross-view disparity consistency constraints between the teacher model and the student model to achieve robust and improved depth estimation. The unannotated images are exploited using a teacher-student framework, where the student model and the teacher model learn from each other. The student model and the teacher model are initialized with pre-trained models. The teacher model is used to generate pseudo-annotations for the unlabeled images and is used to progressively enhance the student model that is updated by standard gradient descent. In return, the teacher model is temporally updated from the student model by an exponential moving average as follows:

[0080]

[0081] Among them, φ T and φ S Represent the parameters from the teacher model and the student model respectively, α is a smoothing factor to prevent the teacher model from overfitting the student model when updating, and α is set to 0.999 here. In order to encourage more effective self-ensemble learning, data enhancement methods are used to enhance the diversity of labeled images and unlabeled images. Since the environmental factors such as lighting of the left and right images in different scenes are slightly different, the model should adjust the different patterns between pixel correspondences through asymmetric color enhancement, where the left and right images are color enhanced differently, and the unlabeled images are subjected to random horizontal flipping as weak enhancement and input into the teacher model. Symmetric geometric enhancement and asymmetric color enhancement are used as strong enhancement for labeled and unlabeled images and sent to the student model. The student model is trained on a limited number of labeled images and a large number of pseudo-labeled images, and the optimization objective is as follows: L = L a (X a )+λ μ L μ (X μ )+λ cons L TAG (X μ );

[0082] Where L a(X a ) represents the supervised loss applied to the labeled image, including classification, localization, and disparity estimation losses, as follows: L a (X a )=L cls (X a )+L reg (X a )+L disp (X a ), where λ μ and λ cons is the weight factor to control the pseudo-supervision loss L μ (X μ ) is similar in form to L a (X a ). Here, set λ μ and λ cons are 0.1 and 1.0 respectively. The pseudo annotation will pass the initial threshold τ cls = 0.9 is refined on the classification and the proposed cross-view agreement strategy.

[0083] The consistency between the output disparity map and the output disparity map by applying a view is as follows:

[0084]

[0085] in, and denote the disparity values ​​of the left and right views at position (i, j), respectively. However, insufficient annotations lead to poor disparity estimation of the base model, resulting in unreliable cross-view disparity consistency. The output disparity map of the teacher model is used to temporally aggregate the knowledge of the student model to guide the disparity estimation of the student model. Specifically, the output disparity map of the teacher model for the left and right images is denoted as d lt and d rt , denoted as d for the student model ls and d rs After exchanging the disparity maps of the views, the photometric consistency between the teacher and student models is as follows:

[0086] The right disparity map of the student model projected by the left disparity map of the teacher model should be consistent with the right disparity map of the teacher model projected by the left disparity map of the student model, because they are pointing to the same target. After mirroring, the corresponding loss on the opposite view can be obtained. In order to further strengthen the disparity consistency between the student model and the teacher model, the photometric consistency between the teacher model and the student model is introduced without exchanging the disparity map, as shown below:

[0087] Finally, the disparity map of the student model should also be aligned with the disparity map of the teacher model, as shown below:

[0088]

[0089] Finally, the final disparity consistency loss for unannotated images is as follows:

[0090]

[0091] Wherein, W and H are the width and height of the downsampled disparity map.

[0092] Image feature extraction: Given a visible light image with resolution H×W, compute per-pixel embedding from the frozen 2D detection model, denoted as I i ∈R H×W×C , where C is the feature dimension and i is the index spanning the total number of images.

[0093] 2D-3D pairing (the ultimate goal of pairing is to obtain the feature vector of each point by fusing the two-dimensional pixels of each point, and finally construct a feature point cloud): The point cloud P∈R of a scene with M points M×3 A three-dimensional surface point P∈R 3 , when the frame I is provided i The intrinsic matrix and the world-to-camera extrinsic matrix E i , calculate the corresponding pixel u = (u, v).

[0094] Fusion function of each pixel: Through 2D-3D pairing, the two-dimensional feature corresponding to the i-th frame of point p can be written as f i =I i (u)∈R C Now, the total number of views is 2 (left view and right view), for point f 2D =φ(f 1 , f 2 ) can be fused into a 2D pixel embedding to obtain a feature vector, where φ: R K×C →R C is an average pooling operator for multi-view features.

[0095] 3D Distillation: Feature Cloud F 2D It can be directly used for 3D scene understanding, however, due to the potential noise in the underlying 2D predictions. In addition, some tasks only provide 3D point clouds or meshes. Therefore, it is possible to extract these 2D visual language knowledge into a 3D point network that only takes 3D point positions as input. Specifically, given an input point cloud P, we try to learn an encoder that outputs an embedding for each point. Figure 1The 3D feature extraction encoder in

[14] adopts PointMLP. For the input point cloud, the residual point MLP block is used to gradually extract local features. At each stage, PointMLP first uses the geometric affine module to perform affine transformation on the local points, and then extracts deep features through several residual MLP modules. A simple maximum pooling operation is used to aggregate multiple local points into one point to describe the local information, and the residual MLP module is used again to extract features. In order to make the network F 3D The output and fusion feature F 2D Keep it consistent and use cosine similarity loss to calculate the distillation loss L D :L D =1-cos(F 2D ,F 3D ).

[0096] S5, converting the depth disparity estimation map into a 3D point cloud;

[0097] S6, using a pre-trained 3D convolutional network to extract features from the 3D point cloud geometric information to obtain 3D features;

[0098] S7, using object-level image-guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning to associate and integrate text features with 2D fusion feature vectors and 3D features to obtain a three-dimensional object detection model;

[0099] Furthermore, it includes: using object-level IoU-guided detection loss in the detection head, introducing distance IoU loss for bounding box regression; introducing object-level cross-contrast alignment task to improve the performance of the cross-fusion module; using self-contrast loss to motivate the model to capture the features of distinguishing objects, using IoU threshold to select positive and negative samples for self-contrast learning; by optimizing self-contrast loss, the fusion module distinguishes different objects. The above self-contrast loss function is: Among them, H p Denote the embedding of proposal p, using P pos ={p|IoU(b p ,b gt )≥δ} as the proposal set containing positive samples, P neg ={p|IoU(b p ,b gt )<δ} is taken as the proposal set containing negative samples, IoU(·,·) represents the IoU score between two bounding boxes, and δ is the IoU threshold.

[0100] Further, including: based on the predicted recommendations b p and the ground truth value bgt , calculate the IoU between them, and have the following regression loss: Where c is the diagonal length of the smallest enclosing box covering the two boxes, ρ 2 (·,·) is the Euclidean distance;

[0101] The object-level image guided detection loss OID is obtained using DIoU loss and label smoothing. The OID loss expression is as follows:

[0102] in,

[0103] Among them, y p is the weight of proposal p in the soft label, y p According to the total number of proposals K, δ is a given pre-set knowledge value threshold and ε is a weight factor.

[0104] Furthermore, it includes: in the training phase, introducing the target detection box of the real target, and selecting all the prediction boxes with IoU greater than a preset threshold as positive samples, and the remaining prediction boxes as negative samples; aligning the features of the positive samples with the language embedding, and pushing the features of the negative samples with the contrast loss to achieve cross-modal understanding; there is the following contrast loss as the loss function of the OCC task:

[0105]

[0106] Among them, H p represents the embedding of proposal p, T represents the embedding of text, s(·,·) represents the similarity scoring function used to measure the similarity between two types of features, using P pos ={p|IoU(b p ,b gt )≥δ} as the proposal set containing positive samples, P neg ={p|IoU(b p ,b gt )<δ} is taken as the proposal set containing negative samples, IoU(·,·) represents the IoU score between two bounding boxes, and δ is the IoU threshold.

[0107] In some embodiments of the present invention, although the 2D fusion feature F can be used 2D Or the feature F of 3D distillation 3D To perform open vocabulary query, we introduce a 2D-3D integration method to obtain hybrid features for better open vocabulary detection performance. First, we use the CLIP text encoder, represented as T = {t 1 ,···,t N}∈R N×C, where N is the number of textual prompts and C is the feature dimension. For each 3D point, we get its 2D fusion and 3D distilled embedding f 2D and f 3D Based on 2D-3D feature integration, hybrid features are obtained to enhance model performance; 2D features are used to predict small objects or geometric shapes, and 3D features can generate predictions for different objects. The purpose of integration is to combine the best of both worlds. Finally, the text features are associated with these two sets of features through cosine similarity, object-level cross-comparison alignment, and object-level self-comparison alignment.

[0108] The object-level IoU guided detection loss is adopted in the detection head. The distance IoU (DIoU) loss is introduced for bounding box regression. According to the predicted proposal b p and the ground truth value b gt , calculate the IoU between them, and have the following regression loss:

[0109] The detailed cross-modal alignment method is as follows Figure 3 As shown:

[0110] Object-level Cross-Contrastive Alignment (OCC): Contrastive learning can provide insights into embedding alignment in different distributions. An object-level cross-contrastive alignment task is introduced to improve the performance of the cross-fusion module. Specifically, during the training phase, the object detection box of the true target is introduced, and all the prediction boxes with IoU greater than a pre-set threshold are selected as positive samples, because they semantically point to the target object and should have similar characteristics. The remaining predicted boxes are considered to be negative samples, representing proposals of other objects or backgrounds. Then, the features of the positive samples are aligned with the language embeddings, and the features of the negative samples are pushed away with the contrastive loss to achieve better cross-modal understanding. Formally, there is the following contrastive loss as the loss function for the OCC task:

[0111]

[0112] Among them, H p represents the embedding of p, and T represents the embedding of text.

[0113] Object-level self-contrastive learning (OSC): In semantic 3D scene understanding, the presence of similar objects in the scene can significantly affect the matching performance of the model. Therefore, it is necessary to use self-contrastive loss to encourage the model to capture features that distinguish objects, rather than pairwise loss to effectively distinguish objects, to improve the model's semantic understanding of the scene. Based on the ideas in the OCC task, the IoU threshold is used to select positive and negative samples for self-contrastive learning. By optimizing the self-contrastive loss, the features of proposals for ground truth targets are encouraged to be as different as possible from other proposals, so that the fusion module can easily distinguish different objects. This contrast loss takes the following form:

[0114]

[0115] The above design helps the model to more effectively identify similarities and differences between objects in the scene. Therefore, the OSC task at the object level improves the performance of the object detector for downstream tasks.

[0116] S8. Implement three-dimensional target detection based on the three-dimensional target detection model, and distinguish different target objects.

[0117] like Figure 1 As shown, we first compute pixel features for each image using a pre-trained open vocabulary 2D detection model. Then, we aggregate the pixel features of the left and right views to each 3D point to form a per-point fused feature vector. At the same time, we use the left and right views to obtain a disparity estimation map, which is then converted into a depth map and converted into a 3D point cloud. Next, we extract a 3D network that reproduces the fused features using only the 3D point cloud as input. The fused 2D features are then integrated, and the 3D features are refined into a per-point feature (each point in 3D space, which can be a point in the point cloud data or a point in the 3D space model), and use it to answer open vocabulary queries.

[0118] In 3D scenes, a 3D network embedding point is trained by building associations between 3D points and pixels from images, supervised by CLIP embedding features. This method aligns 3D points with pixels in feature space and aligns pixels with text features, thus enabling open vocabulary queries on 3D points. The feature extraction method combines 2D and 3D convolutions. First, the positions of 3D points are back-projected into each image and features of relevant pixels are aggregated by fusion of binocular views. Next, a sparse 3D convolutional network is trained to extract features from only the 3D point cloud geometry information, and the loss function optimizes the network by minimizing the difference in aggregated pixel features. Finally, the features generated by the 2D fusion and 3D convolutional networks are merged to form a single feature for each 3D point. This hybrid 2D-3D feature strategy enables the method to exploit salient patterns in both 2D images and 3D geometry, making it more robust and descriptive than features from a single domain.

[0119] In the case of limited annotated stereo images, depth estimation is a key stage for accurately locating 3D objects, but its stability is poor, which can easily lead to inaccurate object positioning. However, the inherent constraints between each pair of stereo images can effectively improve the performance of depth estimation. The base model, as a student model, performs disparity estimation and should follow the disparity estimation of the teacher model to achieve continuous knowledge accumulation from the base model. Therefore, the use of temporal aggregation to guide the disparity consistency constraint can improve the performance of depth estimation in additional data without annotations. At the same time, a cross-view agreement strategy is used to generate more accurate pseudo-annotations, which includes a 3D-2D agreement constraint to ensure the consistency of pseudo-annotations between 2D and 3D views, and a left-right agreement constraint to maintain pseudo-annotations with high similarity between left and right views in the feature space. This pseudo-supervision method effectively utilizes unlabeled data and enhances the detection ability of the model.

[0120] In order to improve the generalization ability of 3D visual language embedding, the object-level image-guided detection (OID) loss is used to help optimize the target bounding box. At the same time, the distance IoU (DIoU) loss and object-level label smoothing method are combined to accelerate convergence and improve performance. In addition, through the object-level cross-contrast alignment (OCC) task and the object-level self-contrast learning (OSC) task, the alignment of objects and descriptions is achieved respectively, and different objects in the scene are distinguished.

[0121] The present invention can more accurately align the three-dimensional point cloud with the pixel features in the image by combining two-dimensional and three-dimensional feature extraction methods. By back-projection and the fusion of binocular views, the complementarity of image information and geometric information is fully utilized, and the representation ability and detection accuracy of three-dimensional points are significantly improved. This hybrid feature strategy not only enhances the robustness of the model, but also effectively copes with three-dimensional target detection tasks in complex environments. The pseudo-supervision strategy proposed in the present invention significantly improves the stability and accuracy of depth estimation by using disparity consistency constraints and cross-view agreement strategies in the case of limited annotated data. By guiding the basic model (student model) to perform depth estimation, and combining time aggregation and disparity consistency constraints, the model can continuously learn and accurately estimate the depth without a large amount of annotated data, thereby achieving more efficient three-dimensional target positioning. The present invention introduces a visual language model and designs innovative tasks such as object-level image guided detection loss (OID), object-level cross-contrast alignment (OCC) and object-level self-contrast learning (OSC), which significantly improves the performance of the three-dimensional target detection model in multimodal data fusion. Through these tasks, the model can not only achieve accurate detection of three-dimensional targets, but also effectively distinguish different objects. The fusion of the visual language model further improves the generalization ability of the model, making it more adaptable and recognizable in a variety of application scenarios. The present invention uses a pseudo-supervised learning method to make full use of unlabeled data for training when labeled data is scarce. By generating high-quality pseudo-annotations and combining self-supervised learning and cross-view constraints, the learning efficiency and accuracy of the model can be improved without the support of a large amount of manually labeled data. This not only reduces the labeling cost, but also enables the model to be effectively trained on large-scale unlabeled data, thereby enhancing the practical application value of the entire system.

[0122] In a second aspect, an embodiment of the present invention provides a 3D target detection system based on binocular vision and open vocabulary, including a 2D feature extraction module, a 2D feature fusion module, a disparity estimation map acquisition module, a depth map acquisition module, a 3D point cloud conversion module, a 3D feature extraction module, a feature fusion module and a target detection module, wherein:

[0123] A 2D feature extraction module for computing pixel features of each image using a pre-trained open vocabulary 2D detection model to obtain pixel features of corresponding left and right views;

[0124] 2D feature fusion module, used to aggregate the pixel features of the left and right views to each 3D point to form a 2D fused feature vector for each point;

[0125] A disparity estimation map acquisition module, used to acquire a disparity estimation map based on pixel features of left and right views;

[0126] A depth map acquisition module is used to introduce a time aggregation-guided disparity consistency constraint based on a disparity estimation map to obtain a depth disparity estimation map;

[0127] 3D point cloud conversion module, used to convert the depth disparity estimation map into 3D point cloud;

[0128] A 3D feature extraction module is used to extract features from 3D point cloud geometric information using a pre-trained 3D convolutional network to obtain 3D features;

[0129] A feature fusion module is used to associate and integrate text features with 2D fusion feature vectors and 3D features using object-level image-guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning to obtain a three-dimensional object detection model;

[0130] The target detection module is used to realize three-dimensional target detection based on the three-dimensional target detection model and to distinguish different target objects.

[0131] This system optimizes image feature extraction and target classification strategies through the cooperation of multiple modules including 2D feature extraction module, 2D feature fusion module, disparity estimation map acquisition module, depth map acquisition module, 3D point cloud conversion module, 3D feature extraction module, feature fusion module and target detection module, by introducing binocular stereo vision technology and powerful visual language model, thereby improving the flexibility and versatility of 3D target recognition and providing strong technical support for applications in various fields.

[0132] like Figure 4 As shown, in a third aspect, an embodiment of the present application provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method in the first aspect described above is implemented.

[0133] The memory 101, the processor 102 and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0134] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0135] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0136] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0138] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 102, a method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0139] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0140] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A 3D object detection method based on binocular vision and open vocabulary, characterized in that: The following steps are involved: The pixel features of each image are calculated using a pre-trained open vocabulary 2D detection model to obtain the pixel features of the corresponding left and right views; Aggregate the pixel features of the left and right views to each 3D point to form a 2D fused feature vector for each point; Obtain a disparity estimation map based on pixel features of the left and right views; Based on the disparity estimation map, temporal aggregation is introduced to guide the disparity consistency constraint to obtain the depth disparity estimation map; Convert the depth disparity estimation map into a 3D point cloud; A pre-trained 3D convolutional network is used to extract features from the 3D point cloud geometric information to obtain 3D features; The text features are associated and integrated with 2D fusion feature vectors and 3D features using object-level image-guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning to obtain a 3D object detection model. Three-dimensional target detection is realized based on the three-dimensional target detection model, and different target objects are distinguished.

2. The 3D object detection method based on binocular vision and open vocabulary according to claim 1, characterized in that: The temporal aggregation guided disparity consistency constraint is a cross-view disparity consistency constraint between the teacher model and the student model.

3. The 3D object detection method based on binocular vision and open vocabulary according to claim 2, characterized in that: The method for obtaining a depth disparity estimation map by introducing a time aggregation-guided disparity consistency constraint based on the disparity estimation map comprises the following steps: Use the pre-trained model to initialize the student model and the teacher model; The teacher model is used to generate pseudo-annotations for unlabeled images and used to progressively enhance the student model which is updated via standard gradient descent; in return, the teacher model receives temporal updates from the student model via an exponential moving average. Perform data augmentation on labeled images and unlabeled images and input them into the teacher model and student model respectively; The student model is trained and optimized on limited annotated images and a large number of pseudo-annotated images; The output disparity map of the teacher model is used to temporally aggregate the knowledge of the student model to guide the disparity estimation of the student model and achieve temporal aggregation-guided disparity consistency.

4. The 3D object detection method based on binocular vision and open vocabulary according to claim 1, characterized in that: The method for associating and integrating text features with 2D fusion feature vectors and 3D features by using object-level image guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning comprises the following steps: The detection head uses object-level IoU-guided detection loss and introduces distance IoU loss for bounding box regression. Introducing object-level cross-contrast alignment task to improve the performance of the cross-fusion module; Self-contrastive loss is used to motivate the model to capture features that distinguish objects, and the IoU threshold is used to select positive and negative samples for self-contrastive learning; by optimizing the self-contrastive loss, the fusion module can distinguish different objects.

5. The 3D object detection method based on binocular vision and open vocabulary according to claim 4, characterized in that: include: Based on the predictions b p and the ground truth value b gt , calculate the IoU between them, and have the following regression loss: Where c is the diagonal length of the smallest enclosing box covering the two boxes, ρ 2 (·,·) is the Euclidean distance; The object-level image guided detection loss OID is obtained using DIoU loss and label smoothing. The OID loss expression is as follows: in, Among them, y p is the weight of proposal p in the soft label, y p According to the total number of proposals K, δ is a given pre-set knowledge value threshold and ε is a weight factor.

6. The 3D object detection method based on binocular vision and open vocabulary according to claim 4, characterized in that: The method of introducing the object-level cross-comparison alignment task comprises the following steps: In the training phase, the target detection box of the real target is introduced, and all the prediction boxes with IoU greater than the preset threshold are selected as positive samples, and the remaining prediction boxes are used as negative samples; Align features of positive samples with language embeddings and push features of negative samples with contrastive loss to achieve cross-modal understanding; There is the following contrast loss as the loss function for the OCC task: Among them, H p represents the embedding of proposal p, T represents the embedding of text, s(·,·) represents the similarity scoring function used to measure the similarity between two types of features, using P pos ={p|IoU(b p ,b gt )≥δ} as the proposal set containing positive samples, P neg ={p|IoU(b p ,b gt )<δ} is taken as the proposal set containing negative samples, IoU(·,·) represents the IoU score between two bounding boxes, and δ is the IoU threshold.

7. The 3D object detection method based on binocular vision and open vocabulary according to claim 4, characterized in that: The self-contrastive loss function is: Among them, H p Denote the embedding of proposal p, using P pos ={p|IoU(b p ,b gt )≥δ} as the proposal set containing positive samples, P neg ={p|IoU(b p ,b gt )<δ} is taken as the proposal set containing negative samples, IoU(·,·) represents the IoU score between two bounding boxes, and δ is the IoU threshold.

8. A 3D object detection system based on binocular vision and open vocabulary, characterized in that: It includes 2D feature extraction module, 2D feature fusion module, disparity estimation map acquisition module, depth map acquisition module, 3D point cloud conversion module, 3D feature extraction module, feature fusion module and target detection module. in: A 2D feature extraction module for computing pixel features of each image using a pre-trained open vocabulary 2D detection model to obtain pixel features of corresponding left and right views; 2D feature fusion module, used to aggregate the pixel features of the left and right views to each 3D point to form a 2D fused feature vector for each point; A disparity estimation map acquisition module, used to acquire a disparity estimation map based on pixel features of left and right views; A depth map acquisition module is used to introduce a time aggregation-guided disparity consistency constraint based on a disparity estimation map to obtain a depth disparity estimation map; 3D point cloud conversion module, used to convert the depth disparity estimation map into 3D point cloud; A 3D feature extraction module is used to extract features from 3D point cloud geometric information using a pre-trained 3D convolutional network to obtain 3D features; A feature fusion module is used to associate and integrate text features with 2D fusion feature vectors and 3D features using object-level image-guided detection loss, object-level cross-contrast alignment, and object-level self-contrast learning to obtain a three-dimensional object detection model; The target detection module is used to realize three-dimensional target detection based on the three-dimensional target detection model and to distinguish different target objects.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images