Three-dimensional visual positioning method based on contrast learning and perspective alignment

Abstract

The application discloses a three-dimensional visual positioning method based on contrast learning and perspective alignment, comprising: learning the features of input text, the attribute-guided point cloud features of objects in the original three-dimensional scene, and the attribute-guided point cloud features of objects in the three-dimensional scene after angle transformation; training the text features and the object original point cloud feature pairs in the perspective by using self-contrast loss and cross-modal contrast loss, training the text features and the object point cloud feature pairs in a specific perspective, obtaining the multi-modal features of objects in the original scene and the scene under a random perspective; training the multi-modal features of objects under different perspectives by using cross-perspective contrast loss; training the object positioning confidence under different perspectives by using positioning consistency loss and positioning loss; in inference, selecting the object with the highest confidence from the predicted visual positioning result as the final positioning result. The application focuses on the alignment of object point cloud features in a specific perspective and across perspectives to improve the understanding of text descriptions related to perspectives and the accuracy of three-dimensional visual positioning and the robustness to perspective changes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross field of computer vision technology and natural language processing, and in particular to a three-dimensional visual positioning method based on contrast learning and perspective alignment. BACKGROUND

[0002] The three-dimensional visual positioning task aims to understand the given free-form language description, find the object in the point cloud scene that matches the language description, and predict its position coordinates. This task spans the visual scene of three-dimensional point clouds and the language scene of text descriptions, and is of great significance to the large-scale application and development of embodied intelligence and autonomous driving, especially to the intelligent agent's understanding of the current spatial position, so as to realize subsequent precise path planning, automatic navigation and other scene interaction operations.

[0003] Existing three-dimensional visual positioning methods mainly follow a two-stage framework, which uses self-attention mechanisms in the transformer network to generate pre-detection proposals and description representations respectively. Then, the multi-modal features are generated by calculating the cross-attention mechanism to fuse the object point cloud features and the text features, so as to infer the matching relationship between the candidate regions and the text and predict the final positioning score for each candidate region. Existing methods mainly focus on object positioning in point cloud scenes of specific perspectives, but due to the irregularity of text representation, it often involves perspective when describing the target object, such as "when you face the door, please choose the leftmost chair". At this time, the object and text correspondence obtained by the cross-attention of the object point cloud features and the text features is easily affected by the change of perspective, resulting in positioning failure. After that, MVT, which is the closest method to the present application, projects the original point cloud scene to different perspectives, fuses the features under different perspectives to generate perspective-robust point cloud features for the candidate regions, and then calculates the matching relationship between the features and the text to enhance the positioning accuracy. However, the alignment of text features and object point cloud features under specific perspectives still lacks explicit supervision constraints, which limits the further improvement of the visual positioning accuracy of the model. SUMMARY

[0004] The present application aims to provide a three-dimensional visual positioning method based on contrast learning and perspective alignment, which gradually aligns the spatial attributes of different modalities to narrow the matching gap between the text and the target object.

[0005] The technical solution to achieve the purpose of the present application is: a three-dimensional visual positioning method based on contrast learning and perspective alignment, comprising the following steps:

[0006] Step 1: single-modal feature extraction, including text features of given language description, object point cloud features based on attribute guidance in original point cloud, and object point cloud features based on attribute guidance after rotating a random angle;

[0007] Step 2: Based on the obtained object original point cloud feature and text feature pair, the object point cloud feature after view transformation and the text feature pair, the intra-view contrast network is trained by using the self-contrast loss and the cross-modal contrast loss, and the original object multi-modal feature and the multi-modal feature of the object after random view transformation are obtained respectively;

[0008] Step 3: Based on the original multi-modal feature and the multi-modal feature after view transformation obtained in step 2, the multi-modal features of the object under different views are aligned by using the cross-view contrast loss;

[0009] Step 4: The visual positioning result is predicted based on the object multi-modal features under different views obtained in step 3, and the visual positioning network is trained by using the positioning consistency loss and the positioning loss, so as to align the object positioning confidence under different views;

[0010] Step 5: For the query text and the point cloud, the positioning confidence of different objects is obtained by using the trained visual positioning network, and the object with the highest confidence is selected as the three-dimensional visual positioning result.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] (1) The intra-view contrast network is constructed, the self-contrast loss is designed to adaptively balance the inherent information and spatial attribute information of the object, so as to obtain a compact and complete object visual representation; the cross-modal contrast loss is designed to align the spatial attributes across modalities before multi-modal interaction, so as to promote more effective multi-modal fusion. Compared with the previous method, the matching relationship between the object and the text under a specific view is captured by aligning the spatial attributes, so as to improve the accuracy of the final visual positioning.

[0013] (2) The cross-view contrast network is constructed, the cross-view contrast loss is designed to punish the difference between the multi-modal features of the same candidate region before and after visual transformation, and the positioning consistency loss is designed to punish the difference between the positioning results of the multi-modal features before and after visual transformation, so as to encourage learning the isomorphic matching relationship between the object and the text under different views. Compared with the previous method which focuses on fusing the object point cloud features under different views to obtain a view-robust feature representation, the matching relationship under different views is aligned to learn a view-robust matching relationship, so as to improve the accuracy of the final visual positioning. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is the overall framework diagram of the three-dimensional visual positioning network based on contrast learning and view alignment in the method of the present application. DETAILED DESCRIPTION

[0015] The present application will be further described below in conjunction with the drawings of the specification.

[0016] As Figure 1As shown, a three-dimensional visual positioning method based on contrast learning and perspective alignment of the application comprises the following steps:

[0017] Step 1: Single-modal feature extraction, including text features of given language description, object point cloud features based on attribute guidance in original point cloud, and object point cloud features based on attribute guidance after rotating random angle, the specific process is as follows:

[0018] Step 1.1: For the input point cloud S, use the pre-trained 3D object detector to generate N object candidate regions O=(o1, o2,...,oN). N Use the feature extraction network PointNet++ to generate original point cloud features for candidate regions, where is the d-dimensional point cloud feature corresponding to the o i th candidate region;

[0019] Step 1.2: For each candidate region o i , calculate the object size feature based on the center point and vertex coordinates of its bounding box; for any two candidate regions o i and o j , calculate the object spatial relationship feature based on the Euclidean distance and normal vector angle of the center point coordinates of the two; connect the size feature and the spatial relationship feature transversely to obtain the attribute feature of the object;

[0020] Step 1.3: Connect the object original point cloud feature and the attribute feature transversely to obtain the object attribute-guided original point cloud feature;

[0021] Step 1.4: For the original point cloud scene, rotate it along the z-axis direction by a random angle θ clockwise, where θ∈(45, 90, 135, 225, 270, 315);

[0022] Step 1.5: For the rotated point cloud, use the feature extraction network PointNet++ to generate the point cloud feature after perspective transformation for the candidate regions where is the d-dimensional point cloud feature after perspective transformation corresponding to the o i th candidate region; similar to step 1.3, connect the object point cloud feature after perspective transformation and the attribute feature transversely to obtain the object attribute-guided point cloud feature after perspective transformation.

[0023] Step 1.6: For the input text T containing L words, use the pre-trained Glove and Bert models to extract text features F T =(t1,...,tL), where t L is the d-dimensional text feature corresponding to the i i th word;

[0024] Step 2: Based on the obtained object original point cloud feature and text feature pair, the object point cloud feature and text feature pair after view transformation, the intra-visual contrast network is trained using self-contrast loss and cross-modal contrast loss, respectively, to obtain the original object multi-modal feature and the multi-modal feature of the object after random view transformation, and the specific process is as follows:

[0025] Step 2.1: For the generated N object candidate regions, calculate the intersection over union with the real object; select the candidate regions with intersection over union greater than 0.5 as the positive sample set O pos The remaining candidate regions are negative sample set O neg ;

[0026] Step 2.2: Use self-contrast loss to enhance the attribute-guided point cloud feature of each candidate region, which encourages the point cloud features of different candidate regions in the positive sample set to be close in the feature space, while the point cloud features of different candidate regions in the negative sample set are far away in the feature space. The specific process is as follows:

[0027]

[0028] Where sim() is the dot product operation, τ = 0.1 is the hyperparameter controlling the similarity, is the attribute-guided point cloud feature of the o i th candidate region;

[0029] Step 2.3: Use cross-modal contrast loss to align the attribute-guided point cloud feature and text feature of each candidate region, which encourages the attribute-guided point cloud feature to be close to the text feature, and also encourages the text feature to be close to the attribute-guided point cloud feature. The specific process is as follows:

[0030]

[0031] Step 2.4: Based on the enhanced text feature and object point cloud feature in step 2.3, calculate the cross attention between the two to generate the multi-modal feature of the candidate region Similarly, calculate the multi-modal feature of the candidate region after view transformation

[0032] Step 3: Based on the original multi-modal feature and the multi-modal feature after view transformation obtained in step 2, use cross-view contrast loss to align the object multi-modal features under different views, and the specific process is as follows:

[0033]

[0034] Step 4: Based on the object multi-modal features obtained in step 3 under different perspectives, the visual positioning network is trained to predict the visual positioning result, and the positioning consistency loss and the positioning loss are used to align the object positioning confidence under different perspectives. The specific process is as follows:

[0035] Step 4.1: Based on the original multi-modal features E s of the candidate region obtained in step 3, the visual positioning head is used to directly predict the positioning result, and the calculation process is as follows:

[0036]

[0037] wherein, is the positioning confidence score of the o i th candidate region, and MLP is a multi-layer perception network;

[0038] Step 4.2: Similarly, based on the multi-modal features of the candidate region after perspective transformation, the positioning result is predicted; the visual positioning network is trained using the positioning consistency loss and the positioning loss, wherein the positioning loss penalizes the difference between the predicted positioning result and the true positioning result, and the positioning consistency loss penalizes the difference between the positioning prediction results before and after perspective transformation of the same candidate region. The specific calculation process is as follows:

[0039]

[0040] wherein, p gt is the true positioning result of the current point cloud scene, is the positioning confidence score of the o i th candidate region after perspective transformation, and the first term represents the positioning loss while the second term represents the positioning consistency loss;

[0041] Step 4.3: According to the above formula, the value of the loss function is calculated, and the network parameters are updated using the gradient descent method until convergence, and finally the trained three-dimensional visual positioning model based on contrast learning and perspective alignment is obtained.

[0042] Step 5: For the query text and point cloud, the visual positioning network is used to obtain the positioning confidence of different objects, and the object with the highest confidence is selected as the three-dimensional visual positioning result.

[0043] In summary, the three-dimensional visual positioning method based on contrast learning and view alignment provided by the application comprises the following steps: learning the features of input text, the attribute-guided point cloud features of objects in the original three-dimensional scene, and the attribute-guided point cloud features of objects in the three-dimensional scene after angle transformation; training the intra-view contrast network to align the text features and the object original point cloud feature pair by using the self-contrast loss and the cross-modal contrast loss, aligning the text features and the object point cloud feature pair under a specific view, and obtaining the multi-modal features of objects in the original scene and the scene under a random view; training the cross-view contrast network by using the cross-view contrast loss to align the multi-modal features of objects under different views; and training the visual positioning network by using the positioning consistency loss and the positioning loss to align the positioning confidence of objects under different views. During inference, the object with the highest confidence is selected as the final positioning result from the predicted visual positioning result for the query text and the original point cloud.

[0044] The method provided by the application first designs and uses an intra-view contrast network to enhance the point cloud feature representation of the candidate region, maximizes the feature similarity of the matched point cloud feature and text feature pair by using the self-contrast loss and the cross-modal contrast loss, and minimizes the feature similarity of the unmatched pair, so as to balance the inherent attribute features and spatial features inside each candidate region. In addition, the method designs and uses a cross-view contrast network to reduce the sensitivity of model learning to view changes, promotes the network to learn the isometric matching relationship between the visual features of objects under different views and the text features by using the cross-view contrast loss and the positioning consistency loss, and finally realizes accurate three-dimensional visual positioning.

[0045] The application will be further described below in conjunction with the embodiments. The described embodiments are only a part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0046] As shown in Figure 1 The application provides a technical solution: a three-dimensional visual positioning method based on contrast learning and view alignment, comprising the following steps:

[0047] Step 1: based on the public ScanRefer dataset or a point cloud scene dataset with text description collected and labeled by oneself, text features of language description, attribute-guided object point cloud features in the original point cloud, and attribute-guided object point cloud features after rotating a random angle are obtained respectively, and the specific process is as follows:

[0048] Step 1.1: for the input point cloud S, a pre-trained 3D object detector GroupFree is used to generate N=256 object candidate regions O=(o1, o2,..., oN), and the specific process is as follows: N). 1024 points are sampled for each candidate region, and the original point cloud feature of the candidate region is generated using the feature extraction network PointNet++ where is the d = 768-dimensional point cloud feature corresponding to the o i th candidate region;

[0049] Step 1.2: For each candidate region o i , the object size feature is calculated based on the center point and vertex coordinates of its bounding box; for any two candidate regions o i and o j , the object spatial relationship feature is calculated based on the Euclidean distance and normal vector angle of the center point coordinates of the two; the size feature and the spatial relationship feature are transversely connected to obtain the attribute feature of the object;

[0050] Step 1.3: The attribute-guided original point cloud feature of the object is obtained by transversely connecting the object original point cloud feature and the attribute feature;

[0051] Step 1.4: For the original point cloud scene, a random rotation of angle θ clockwise along the z-axis direction is performed, where θ ∈ (45, 90, 135, 225, 270, 315);

[0052] Step 1.5: For the rotated point cloud, the perspective-transformed point cloud feature of the candidate region is generated using the feature extraction network PointNet++ where is the d = 768-dimensional perspective-transformed point cloud feature corresponding to the o i th candidate region; similar to Step 1.3, the perspective-transformed point cloud feature of the object and the attribute feature are transversely connected to obtain the perspective-transformed attribute-guided point cloud feature of the object.

[0053] Step 1.6: For the input text T containing L words, the text feature F T = (t1,...,t L ) is extracted using the pre-trained Glove and Bert models, where t i is the d = 768-dimensional text feature corresponding to the i

[0054] Step 2: Based on the obtained pairs of object original point cloud features and text features, and the pairs of perspective-transformed object point cloud features and text features, the intra-visual contrast network is trained using the self-contrast loss and the cross-modal contrast loss to obtain the original object multi-modal feature and the multi-modal feature of the object after random perspective transformation, respectively, and the specific process is as follows:

[0055] Step 2.1: For the generated N = 256 object candidate regions, calculate their intersection over union with the real object; select the candidate regions with intersection over union greater than 0.5 as the positive sample set O pos The remaining candidate regions are the negative sample set O neg ;

[0056] Step 2.2: Use self-contrast loss to enhance the attribute-guided point cloud feature of each candidate region, which encourages the point cloud features of different candidate regions in the positive sample set to be close in the feature space, while pulling the point cloud features of different candidate regions in the negative sample set away in the feature space. The specific process is as follows:

[0057]

[0058] Where sim() is the dot product calculation operation, τ = 0.1 is the hyperparameter controlling the similarity, is the attribute-guided point cloud feature of the o i th candidate region, with a feature dimension of 1536;

[0059] Step 2.3: Use cross-modal contrast loss to align the attribute-guided point cloud feature and the text feature of each candidate region, encouraging the attribute-guided point cloud feature to be close to the text feature, and also encouraging the text feature to be close to the attribute-guided point cloud feature. The specific process is as follows:

[0060]

[0061] Step 2.4: Based on the enhanced text feature and object point cloud feature in step 2.3, calculate the cross-attention between the two to generate the multi-modal feature of the candidate region Similarly, calculate the multi-modal feature of the candidate region after perspective transformation

[0062] Step 3: Based on the original multi-modal feature and the perspective-transformed multi-modal feature obtained in step 2, use cross-perspective contrast loss to align the object multi-modal features under different perspectives. The specific process is as follows:

[0063]

[0064] Step 4: Based on the object multi-modal features under different perspectives obtained in step 3, predict the visual positioning result, use the positioning consistency loss and positioning loss to train the visual positioning network, and align the object positioning confidence under different perspectives. The specific process is as follows:

[0065] Step 4.1: Based on the original multi-modal feature E s of the candidate region obtained in step 3, use the visual positioning head to directly predict the positioning result, the calculation process is as follows:

[0066]

[0067] wherein, is the o i the location confidence score of the candidate region, and the MLP is set as a full connection layer with 2 layers;

[0068] Step 4.2: Similarly, based on the multi-modal features of the candidate region after the perspective transformation, the location prediction result is predicted; the visual positioning network is trained by using the location consistency loss and the location loss, wherein the location loss punishes the difference between the predicted location result and the real location result, and the location consistency loss punishes the difference between the location prediction results before and after the perspective transformation of the same candidate region, and the specific calculation process is as follows:

[0069]

[0070] wherein, p gt is the real location result of the current point cloud scene, is the o i the location confidence score of the candidate region after the perspective transformation, the former represents the location loss, and the second represents the location consistency loss;

[0071] Step 4.3: The value of the loss function is calculated according to the above formula, the network parameters are updated by using the gradient descent method, the training set is trained for 100 rounds until convergence, and finally the three-dimensional visual positioning model based on contrast learning and perspective alignment is obtained after training; in order to balance the loss of each part, the weights of the intra-perspective contrast loss and the cross-perspective contrast loss are set to 0.1 and 0.1 respectively.

[0072] Step 5: For the query text and the point cloud, the location confidence of different objects is obtained by using the trained visual positioning network, and the object with the highest confidence is selected as the three-dimensional visual positioning result.

[0073] The above embodiments should be regarded as exemplary and non-limiting, and the protection scope of the present application is not limited thereto, and any modification, equivalent replacement and improvement made by any person skilled in the art within the technical range disclosed by the present application, within the spirit and principle of the present application, should be covered within the protection scope of the present application.​

Claims

1. A three-dimensional visual positioning method based on contrastive learning and view alignment, characterized in that, Comprising the following steps: Step 1: Single-modal feature extraction, including text features of given language description, object point cloud features based on attribute guidance in original point cloud, object point cloud features based on attribute guidance after rotating random angle; Step 2: Based on the obtained object original point cloud features and text feature pairs, object point cloud features and text feature pairs after view transformation, train the visual intra-contrast network using self-contrast loss and cross-modal contrast loss, respectively, to obtain the original object multi-modal features and the multi-modal features of the object after random view transformation; The implementation process of step 2 is as follows: Step 2.1: For the generated N object candidate regions, calculate the intersection over union with the real object; select the candidate regions with intersection over union greater than 0.5 as the positive sample set O pos The remaining candidate regions are the negative sample set O neg ; Step 2.2: Use self-contrast loss to enhance the attribute-guided point cloud features of each candidate region, which encourages the point cloud features of different candidate regions in the positive sample set to be close in the feature space, while the point cloud features of different candidate regions in the negative sample set are far away in the feature space, and the specific process is as follows: where sim() is a dot product computation operation, τ = 0.1 is a hyperparameter to control the similarity, is the attribute-guided point cloud feature of the o i th candidate region. Step 2.3: Use cross-modal contrast loss to align the attribute-guided point cloud features and text features of each candidate region, which encourages the attribute-guided point cloud features to be close to the text features, and also encourages the text features to be close to the attribute-guided point cloud features, and the specific process is as follows: Step 2.4: Based on the enhanced text features and object point cloud features in step 2.3, calculate the cross-attention of the two to generate the multi-modal features of the candidate region Similarly, calculate the multi-modal features of the candidate region after the perspective transformation Step 3: Based on the original multi-modal features and the multi-modal features after view transformation obtained in step 2, use cross-view contrast loss to align the object multi-modal features under different views; Align the original multi-modal features E of the object using the cross-view contrast loss s and the multi-modal features after view transformation The specific process is as follows: Step 4: Based on the object multi-modal features under different views obtained in step 3, predict the visual positioning result, train the visual positioning network using the positioning consistency loss and the positioning loss, and align the object positioning confidence under different views; Step 5: For query text and point cloud, use the trained visual positioning network to obtain the positioning confidence of different objects, and select the object with the highest confidence as the three-dimensional visual positioning result.

2. The three-dimensional visual positioning method based on contrastive learning and view alignment according to claim 1, characterized in that, The specific implementation process of step 1 is as follows: Step 1.1: For the input point cloud S, use a pre-trained 3D object detector to generate N object candidate regions O = (o1, o2,..., oN) using a pre-trained 3D object detector; use a feature extraction network PointNet++ to generate raw point cloud features N for the candidate regions where is the d-dimensional point cloud feature corresponding to the o i th candidate region; Step 1.2: For each candidate region o i , compute object size features based on its center point and vertex coordinates of the bounding box; For any two candidate regions o i and o j , compute object spatial relation features based on the Euclidean distance of their center points and the angle of their normal vectors. Transversely connect size features and spatial relationship features to obtain object attribute features; Step 1.3: Transversely connect object original point cloud features and attribute features to obtain object attribute-guided original point cloud features; Step 1.4: For the original point cloud scene, rotate it clockwise by θ angle along the z-axis direction, where θ ∈ (45, 90, 135, 225, 270, 315); Step 1.5: For the rotated point cloud, use the feature extraction network PointNet++ to generate the point cloud features of the candidate region after perspective transformation wherein is the d-dimensional point cloud feature of the o i th candidate region after perspective transformation; Transversely connect object view-transformed point cloud features and attribute features to obtain object attribute-guided point cloud features after view transformation; Step 1.6: For an input text T containing L words, extract text features F using pre-trained Glove and Bert models T = (t1,...,t L ), where t i is the d-dimensional text feature corresponding to the i-th word.

3. The three-dimensional visual positioning method based on contrastive learning and view alignment according to claim 1, characterized in that The implementation process of step 4 is as follows: Step 4.1: Based on the obtained raw multimodal features E of the candidate region s directly predict the positioning result by using the visual positioning head, and the calculation process is: wherein, is the o i a localization confidence score for the candidate region, MLP is a multi-layer perceptron network; Step 4.2: Multimodal features based on the perspective-transformed candidate region predicting the positioning result; training the visual positioning network by using the positioning consistency loss and the positioning loss, wherein the positioning loss punishes the difference between the predicted positioning result and the real positioning result, and the positioning consistency loss punishes the difference between the positioning prediction results before and after perspective transformation of the same candidate region, and the specific calculation process is as follows: Where, p gt This represents the actual localization result for the current point cloud scene. For the o i The location reliability score after the candidate region perspective transformation, the first term represents the location loss, and the second term represents the location consistency loss; Step 4.3: Calculate the value of the loss function according to the above formula, update the network parameters using gradient descent method until convergence, and finally obtain the trained three-dimensional visual positioning model based on contrast learning and view alignment.

Images