Three-dimensional visual positioning method based on comparative learning and visual angle alignment

Abstract

The invention discloses a three-dimensional visual positioning method based on comparative learning and visual angle alignment. The method comprises the following steps: learning features of an input text, attribute guidance-based point cloud features of an object in an original three-dimensional scene, and attribute guidance-based point cloud features of an object in the three-dimensional scene after angle transformation; using self-comparison loss and cross-modal comparison loss to train comparison network alignment text features and object original point cloud feature pairs in a visual angle, aligning the text features and object point cloud feature pairs under a specific visual angle, and obtaining multi-modal features of objects in an original scene and a scene under a random visual angle; training a cross-view-angle comparison network by using cross-view-angle comparison loss, and aligning the multi-modal features of the object under different view angles; and training a visual positioning network by using the positioning consistency loss and the positioning loss, and aligning the object positioning confidence under different visual angles. During reasoning, for a query text and an original point cloud, an object with the highest confidence degree is selected from predicted visual positioning results to serve as a final positioning result. According to the invention, by focusing on alignment of object point cloud features in a specific view angle and cross view angles, the understanding of text description related to the view angle is improved, and the accuracy of three-dimensional visual positioning and the robustness of view angle change are improved.

Description

Technical Field

[0001] The present invention belongs to the intersection of computer vision technology and natural language processing, and specifically, is a three-dimensional visual positioning method based on contrast learning and perspective alignment. Background Art

[0002] The 3D visual positioning task aims to understand a given free-form language description, find objects that match the language description from the point cloud scene, and predict their position coordinates. This task spans the visual scene of the 3D point cloud and the language scene of the text description, and is of great significance for promoting the large-scale application and development of embodied intelligence and autonomous driving, especially assisting the intelligent agent to understand its current spatial position, thereby realizing subsequent accurate path planning, automatic navigation and other scene interaction operations.

[0003] Existing 3D visual localization methods mainly follow a two-stage framework, which uses the self-attention mechanism in the transformer network to generate representations of pre-detection proposals and descriptions respectively. Then, the object point cloud features and text features are fused by calculating the cross-attention mechanism to generate multimodal features, thereby inferring the matching relationship between the candidate region and the text to predict the final localization score for each candidate region. Existing methods mainly focus on object localization in point cloud scenes of specific perspectives. However, due to the irregularity of text representation, it often involves perspective when describing the target object, such as "When you face the door, please choose the leftmost chair." At this time, the correspondence between the object and the text obtained based on the cross-attention of the object point cloud features and the text features is easily affected by the change of perspective and causes positioning failure. After that, MVT, the method closest to the present invention, projects the original point cloud scene to different perspectives, fuses the features under different perspectives to generate perspective-robust point cloud features for the candidate region, and then calculates its matching relationship with the text to enhance the localization accuracy. However, the alignment of text features and object point cloud features under a specific perspective still lacks clear supervision constraints, which limits the further improvement of the model's visual localization accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a three-dimensional visual positioning method based on contrastive learning and viewpoint alignment, which gradually aligns the spatial attributes of different modalities to narrow the matching gap between text and target objects.

[0005] The technical solution to achieve the purpose of the present invention is: a three-dimensional visual positioning method based on contrast learning and view alignment, comprising the following steps:

[0006] Step 1: Unimodal feature extraction, including text features of given language description, object point cloud features guided by attributes in the original point cloud, and object point cloud features guided by attributes after rotating at random angles;

[0007] Step 2: Based on the acquired original point cloud features and text feature pairs of the object, and the point cloud features and text feature pairs of the object after the perspective transformation, the self-contrast loss and cross-modal contrast loss are used to train the intra-visual contrast network to obtain the original object multimodal features and the multimodal features of the object after the random perspective transformation respectively;

[0008] Step 3: Based on the original multimodal features obtained in step 2 and the multimodal features after perspective transformation, the multimodal features of objects under different perspectives are aligned using cross-perspective contrast loss;

[0009] Step 4: Predict the visual localization results based on the multimodal features of the objects under different perspectives obtained in step 3, train the visual localization network using the localization consistency loss and localization loss, and align the localization confidence of the objects under different perspectives;

[0010] Step 5: For query text and point cloud, use the trained visual localization network to obtain the positioning confidence of different objects, and select the object with the highest confidence as the 3D visual localization result.

[0011] Compared with the prior art, the present invention has the following significant advantages:

[0012] (1) The present invention constructs an intra-view contrast network and designs a self-contrast loss to adaptively balance the intrinsic information and spatial attribute information of an object to obtain a compact and complete visual representation of the object; the present invention designs a cross-modal contrast loss to align spatial attributes across modalities before multimodal interaction to promote more effective multimodal fusion. Compared with previous methods, the present invention captures the matching relationship between objects and texts in a specific view by aligning spatial attributes, thereby improving the accuracy of the final visual positioning.

[0013] (2) The present invention constructs a cross-view contrast network, designs a cross-view contrast loss to penalize the difference in multimodal features of the same candidate region before and after visual transformation, and designs a positioning consistency loss to penalize the difference in positioning results of multimodal features before and after visual transformation, so as to encourage learning the equivariant matching relationship between objects and texts from different perspectives. Compared with previous methods that focus on fusing object point cloud features from different perspectives to obtain a view-robust feature representation, the present invention learns a view-robust matching relationship by aligning the matching relationships from different perspectives, thereby improving the accuracy of the final visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is an overall framework diagram of the three-dimensional visual positioning network based on contrast learning and view alignment in the method of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings.

[0016] like Figure 1As shown, a three-dimensional visual positioning method based on contrast learning and perspective alignment of the present invention comprises the following steps:

[0017] Step 1: Unimodal feature extraction, including text features of given language description, object point cloud features guided by attributes in the original point cloud, and object point cloud features guided by attributes after rotating at random angles. The specific process is as follows:

[0018] Step 1.1: For the input point cloud S, use the pre-trained 3D object detector to generate N object candidate regions O = (o1, o2, ..., o N ). Use the feature extraction network PointNet++ to generate raw point cloud features for the candidate area in corresponds to the oth i The d-dimensional point cloud features of the candidate regions;

[0019] Step 1.2: For each candidate region o i , the object size features are calculated based on the center point and vertex coordinates of its bounding box; for any two candidate regions o i and j , the spatial relationship features of the object are calculated based on the Euclidean distance of the center point coordinates of the two and the angle between the normal vectors; the attribute features of the object are obtained by horizontally connecting the size features and the spatial relationship features;

[0020] Step 1.3: Laterally connect the original point cloud features and attribute features of the object to obtain the original point cloud features of the object guided by attributes;

[0021] Step 1.4: For the original point cloud scene, randomly rotate it clockwise along the z-axis by an angle of θ, where θ∈(45,90,135,225,270,315);

[0022] Step 1.5: For the rotated point cloud, use the feature extraction network PointNet++ to generate the point cloud features after the viewpoint transformation for the candidate area in corresponds to the oth i The d-dimensional point cloud features of the candidate regions after the perspective transformation are obtained; similar to step 1.3, the point cloud features and attribute features of the object after the perspective transformation are horizontally connected to obtain the attribute-guided point cloud features of the object after the perspective transformation.

[0023] Step 1.6: For the input text T containing L words, use the pre-trained Glove and Bert models to extract text features F T =(t1,...,t L ), where t i is the d-dimensional text feature corresponding to the i-th word;

[0024] Step 2: Based on the acquired original point cloud features and text feature pairs of the object, and the point cloud features and text feature pairs of the object after the perspective transformation, the self-contrast loss and cross-modal contrast loss are used to train the visual intra-contrast network to obtain the original object multimodal features and the multimodal features of the object after the random perspective transformation, respectively. The specific process is as follows:

[0025] Step 2.1: For the N generated object candidate regions, calculate their intersection-over-union ratio with the real object; select the candidate regions with an intersection-over-union ratio greater than 0.5 as the positive sample set O pos , the remaining candidate regions are the negative sample set O neg ;

[0026] Step 2.2: Use self-contrastive loss to enhance the attribute-guided point cloud features of each candidate region, which encourages the point cloud features of different candidate regions in the positive sample set to be close in the feature space, while increasing the distance between the point cloud features of different candidate regions in the negative sample set in the feature space. The specific process is as follows:

[0027]

[0028] Among them, sim() is the dot product calculation operation, τ=0.1 is the hyperparameter that controls the similarity, It is o i Point cloud features guided by the attributes of candidate regions;

[0029] Step 2.3: Use cross-modal contrast loss to align the attribute-guided point cloud features and text features of each candidate region. While encouraging the attribute-guided point cloud features to be close to the text features, the text features are also encouraged to be close to the attribute-guided point cloud features. The specific process is as follows:

[0030]

[0031] Step 2.4: Based on the enhanced text features and object point cloud features in step 2.3, calculate the cross attention between the two to generate the multimodal features of the candidate region Similarly, the multimodal features of the candidate regions after the perspective transformation are calculated

[0032] Step 3: Based on the original multimodal features obtained in step 2 and the multimodal features after perspective transformation, the multimodal features of objects under different perspectives are aligned using cross-perspective contrast loss. The specific process is as follows:

[0033]

[0034] Step 4: Predict the visual localization results based on the multimodal features of the objects under different perspectives obtained in step 3, train the visual localization network using the localization consistency loss and localization loss, and align the localization reliability of the objects under different perspectives. The specific process is as follows:

[0035] Step 4.1: Based on the original multimodal features E of the candidate regions obtained in step 3 s , use the visual positioning head to directly predict the positioning result, and the calculation process is:

[0036]

[0037] in, For o i The location confidence score of the candidate area, MLP is a multi-layer perceptron network;

[0038] Step 4.2: Similarly, multimodal features of candidate regions after view transformation Predict the positioning results; use the positioning consistency loss and positioning loss to train the visual positioning network. The positioning loss penalizes the difference between the predicted positioning result and the actual positioning result, while the positioning consistency loss penalizes the difference between the positioning prediction results before and after the perspective transformation of the same candidate area. The specific calculation process is:

[0039]

[0040] Among them, p gt is the real positioning result of the current point cloud scene, For o i The positioning confidence score of the candidate region after the perspective transformation, the first item represents the positioning loss and the second item represents the positioning consistency loss;

[0041] Step 4.3: Calculate the value of the loss function according to the above formula, use the gradient descent method to update the network parameters until convergence, and finally obtain a trained 3D visual positioning model based on contrastive learning and perspective alignment;

[0042] Step 5: For query text and point cloud, use the trained visual localization network to obtain the positioning confidence of different objects, and select the object with the highest confidence as the 3D visual localization result.

[0043] In summary, the three-dimensional visual positioning method based on contrast learning and perspective alignment of the present invention includes: learning the features of input text, the point cloud features of objects in the original three-dimensional scene based on attribute guidance, and the point cloud features of objects in the three-dimensional scene after angle transformation based on attribute guidance; using self-contrast loss and cross-modal contrast loss to train the intra-view contrast network to align the text features and the original point cloud feature pairs of the object, align the text features and the point cloud feature pairs of the object under a specific perspective, and obtain the multimodal features of the objects in the original scene and the scene under a random perspective; using cross-view contrast loss to train the cross-view contrast network to align the multimodal features of objects under different perspectives; using positioning consistency loss and positioning loss to train the visual positioning network to align the positioning confidence of objects under different perspectives. During reasoning, for the query text and the original point cloud, the object with the highest confidence is selected from the predicted visual positioning results as the final positioning result.

[0044] The method of the present invention first designs and uses an intra-view contrast network to enhance the point cloud feature representation of the candidate region, and uses self-contrast loss and cross-modal contrast loss to maximize the feature similarity of matching point cloud features and text feature pairs, while minimizing the feature similarity of unmatched pairs, thereby balancing the inherent attribute features and spatial features within each candidate region. In addition, the method designs and uses a cross-view contrast network to reduce the sensitivity of model learning to viewpoint changes, and uses cross-view contrast loss and positioning consistency loss to promote the network to learn the equivariant matching relationship between object visual features and text features under different viewpoints, ultimately achieving accurate three-dimensional visual positioning.

[0045] The present invention is further described below in conjunction with embodiments. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, the present invention provides a technical solution: a three-dimensional visual positioning method based on contrast learning and perspective alignment, comprising the following steps:

[0047] Step 1: Based on the public ScanRefer dataset or the point cloud scene dataset with text description collected and annotated by yourself, obtain the text features of the language description, the object point cloud features guided by attributes in the original point cloud, and the object point cloud features guided by attributes after rotating at random angles. The specific process is as follows:

[0048] Step 1.1: For the input point cloud S, use the pre-trained 3D object detector GroupFree to generate N = 256 object candidate regions O = (o1, o2, ..., o N). Sample 1024 points for each candidate region and use the feature extraction network PointNet++ to generate raw point cloud features for the candidate region. in corresponds to the oth i d=768-dimensional point cloud features of candidate regions;

[0049] Step 1.2: For each candidate region o i , the object size features are calculated based on the center point and vertex coordinates of its bounding box; for any two candidate regions o i and j , the spatial relationship features of the object are calculated based on the Euclidean distance of the center point coordinates of the two and the angle between the normal vectors; the attribute features of the object are obtained by horizontally connecting the size features and the spatial relationship features;

[0050] Step 1.3: Laterally connect the original point cloud features and attribute features of the object to obtain the original point cloud features of the object guided by attributes;

[0051] Step 1.4: For the original point cloud scene, randomly rotate it clockwise along the z-axis by an angle of θ, where θ∈(45,90,135,225,270,315);

[0052] Step 1.5: For the rotated point cloud, use the feature extraction network PointNet++ to generate the point cloud features after the viewpoint transformation for the candidate area in corresponds to the oth i The d=768-dimensional point cloud features of the candidate regions after the perspective transformation are obtained; similar to step 1.3, the point cloud features and attribute features of the object after the perspective transformation are horizontally connected to obtain the attribute-guided point cloud features of the object after the perspective transformation.

[0053] Step 1.6: For the input text T containing L words, use the pre-trained Glove and Bert models to extract text features F T =(t1,...,t L ), where t i is the d=768-dimensional text feature corresponding to the i-th word;

[0054] Step 2: Based on the acquired original point cloud features and text feature pairs of the object, and the point cloud features and text feature pairs of the object after the perspective transformation, the self-contrast loss and cross-modal contrast loss are used to train the visual intra-contrast network to obtain the original object multimodal features and the multimodal features of the object after the random perspective transformation, respectively. The specific process is as follows:

[0055] Step 2.1: For the generated N = 256 object candidate regions, calculate their intersection-over-union ratio with the real object; select the candidate regions with an intersection-over-union ratio greater than 0.5 as the positive sample set O pos , the remaining candidate regions are the negative sample set O neg ;

[0056] Step 2.2: Use self-contrastive loss to enhance the attribute-guided point cloud features of each candidate region, which encourages the point cloud features of different candidate regions in the positive sample set to be close in the feature space, while increasing the distance between the point cloud features of different candidate regions in the negative sample set in the feature space. The specific process is as follows:

[0057]

[0058] Among them, sim() is the dot product calculation operation, τ=0.1 is the hyperparameter that controls the similarity, It is o i The attribute-guided point cloud features of the candidate regions have a feature dimension of 1536;

[0059] Step 2.3: Use cross-modal contrast loss to align the attribute-guided point cloud features and text features of each candidate region. While encouraging the attribute-guided point cloud features to be close to the text features, the text features are also encouraged to be close to the attribute-guided point cloud features. The specific process is as follows:

[0060]

[0061] Step 2.4: Based on the enhanced text features and object point cloud features in step 2.3, calculate the cross attention between the two to generate the multimodal features of the candidate region Similarly, the multimodal features of the candidate regions after the perspective transformation are calculated

[0062] Step 3: Based on the original multimodal features obtained in step 2 and the multimodal features after perspective transformation, the multimodal features of objects under different perspectives are aligned using cross-perspective contrast loss. The specific process is as follows:

[0063]

[0064] Step 4: Predict the visual localization results based on the multimodal features of the objects under different perspectives obtained in step 3, train the visual localization network using the localization consistency loss and localization loss, and align the localization reliability of the objects under different perspectives. The specific process is as follows:

[0065] Step 4.1: Based on the original multimodal features E of the candidate regions obtained in step 3 s , use the visual positioning head to directly predict the positioning result, and the calculation process is:

[0066]

[0067] in, For o i The localization confidence score of the candidate region, the MLP is set to a 2-layer fully connected layer;

[0068] Step 4.2: Similarly, multimodal features of candidate regions after view transformation Predict the positioning results; use the positioning consistency loss and positioning loss to train the visual positioning network. The positioning loss penalizes the difference between the predicted positioning result and the actual positioning result, while the positioning consistency loss penalizes the difference between the positioning prediction results before and after the perspective transformation of the same candidate area. The specific calculation process is:

[0069]

[0070] Among them, p gt is the real positioning result of the current point cloud scene, For o i The positioning confidence score of the candidate region after the perspective transformation, the first item represents the positioning loss and the second item represents the positioning consistency loss;

[0071] Step 4.3: Calculate the value of the loss function according to the above formula, use the gradient descent method to update the network parameters, train the training set for 100 rounds until convergence, and finally obtain the trained 3D visual positioning model based on contrast learning and view alignment; in order to balance the losses of each part, the weights of the intra-view contrast loss and the cross-view contrast loss are set to 0.1 and 0.1 respectively.

[0072] Step 5: For query text and point cloud, use the trained visual localization network to obtain the positioning confidence of different objects, and select the object with the highest confidence as the 3D visual localization result.

[0073] The above embodiments should be regarded as illustrative and non-restrictive, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A three-dimensional visual positioning method based on contrastive learning and view alignment, characterized in that: The following steps are involved: Step 1: Unimodal feature extraction, including text features of given language description, object point cloud features guided by attributes in the original point cloud, and object point cloud features guided by attributes after rotating at random angles; Step 2: Based on the acquired original point cloud features and text feature pairs of the object, and the point cloud features and text feature pairs of the object after the perspective transformation, the self-contrast loss and cross-modal contrast loss are used to train the intra-visual contrast network to obtain the original object multimodal features and the multimodal features of the object after the random perspective transformation respectively; Step 3: Based on the original multimodal features obtained in step 2 and the multimodal features after perspective transformation, the multimodal features of objects under different perspectives are aligned using cross-perspective contrast loss; Step 4: Predict the visual localization results based on the multimodal features of the objects under different perspectives obtained in step 3, train the visual localization network using the localization consistency loss and localization loss, and align the localization confidence of the objects under different perspectives; Step 5: For query text and point cloud, use the trained visual localization network to obtain the positioning confidence of different objects, and select the object with the highest confidence as the 3D visual localization result.

2. The three-dimensional visual positioning method based on contrast learning and view alignment according to claim 1, characterized in that: The specific implementation process in step 1 is: Step 1.1: For the input point cloud S, use the pre-trained 3D object detector to generate N object candidate regions O = (o1, o2, ..., o N ); Use the feature extraction network PointNet++ to generate raw point cloud features for the candidate area in corresponds to the oth i The d-dimensional point cloud features of the candidate regions; Step 1.2: For each candidate region o i , the object size features are calculated based on the center point and vertex coordinates of its bounding box; for any two candidate regions o i and j , the spatial relationship features of the objects are calculated based on the Euclidean distance of the center point coordinates and the angle between the normal vectors; Horizontally connect the size features and the spatial relationship features to obtain the attribute features of the object; Step 1.3: Laterally connect the original point cloud features and attribute features of the object to obtain the original point cloud features of the object guided by attributes; Step 1.4: For the original point cloud scene, randomly rotate it clockwise along the z-axis by an angle of θ, where θ∈(45,90,135,225,270,315); Step 1.5: For the rotated point cloud, use the feature extraction network PointNet++ to generate the point cloud features after the viewpoint transformation for the candidate area in corresponds to the oth i The d-dimensional point cloud features after the perspective transformation of the candidate regions; The point cloud features and attribute features of the object after the perspective transformation are horizontally connected to obtain the attribute-guided point cloud features of the object after the perspective transformation; Step 1.6: For the input text T containing L words, use the pre-trained Glove and Bert models to extract text features F T =(t1, ..., t L ), where t i is the d-dimensional text feature corresponding to the i-th word.

3. The three-dimensional visual positioning method based on contrastive learning and perspective alignment according to claim 1, characterized in that: The implementation process of step 2 is as follows: Step 2.1: For the N generated object candidate regions, calculate their intersection-over-union ratio with the real object; select the candidate regions with an intersection-over-union ratio greater than 0.5 as the positive sample set O pos , the remaining candidate regions are the negative sample set O neg ; Step 2.2: Use self-contrastive loss to enhance the attribute-guided point cloud features of each candidate region, which encourages the point cloud features of different candidate regions in the positive sample set to be close in the feature space, while increasing the distance between the point cloud features of different candidate regions in the negative sample set in the feature space. The specific process is as follows: Among them, sim() is the dot product calculation operation, τ=0.1 is the hyperparameter that controls the similarity, It is o i Point cloud features guided by the attributes of candidate regions; Step 2.3: Use cross-modal contrast loss to align the attribute-guided point cloud features and text features of each candidate region. While encouraging the attribute-guided point cloud features to be close to the text features, the text features are also encouraged to be close to the attribute-guided point cloud features. The specific process is as follows: Step 2.4: Based on the enhanced text features and object point cloud features in step 2.3, calculate the cross attention between the two to generate the multimodal features of the candidate region Similarly, the multimodal features of the candidate regions after the perspective transformation are calculated 4. The three-dimensional visual positioning method based on contrastive learning and view alignment according to claim 1, characterized in that: In step 3, the original multimodal features E of the object are aligned using cross-view contrast loss s And multimodal features after perspective transformation The specific process is:

5. The three-dimensional visual positioning method based on contrast learning and view alignment according to claim 1, characterized in that The implementation process of step 4 is as follows: Step 4.1: Based on the original multimodal features E of the obtained candidate regions s , use the visual positioning head to directly predict the positioning result, and the calculation process is: in, For o i The location confidence score of the candidate area, MLP is a multi-layer perceptron network; Step 4.2: Multimodal features of candidate regions after view transformation Predict the positioning results; use the positioning consistency loss and positioning loss to train the visual positioning network. The positioning loss penalizes the difference between the predicted positioning result and the actual positioning result, while the positioning consistency loss penalizes the difference between the positioning prediction results before and after the perspective transformation of the same candidate area. The specific calculation process is: Among them, p gt is the real positioning result of the current point cloud scene, For o i The positioning confidence score of the candidate region after the perspective transformation. The first item represents the positioning loss, and the second item represents the positioning consistency loss. Step 4.3: Calculate the value of the loss function according to the above formula, use the gradient descent method to update the network parameters until convergence, and finally obtain a trained 3D visual positioning model based on contrastive learning and perspective alignment.

Images