A 3D visual positioning method and system based on adaptive position coding and parallel decoding

Abstract

The present invention discloses a 3D visual positioning method and system based on adaptive position coding and parallel decoding, wherein the method comprises: using a text encoder to process input natural language text to obtain text tokens and features; using a visual encoder to process 3D point cloud input to obtain seed points and features; using a cross encoder to intermodulate visual and text features to update text features and visual features; predicting the confidence scores of seed points and sorting them, and selecting the candidate points with the highest scores as queries; dividing text tokens into target object attribute tokens and surrounding space environment tokens according to the semantics of natural language descriptions; using a dual-branch parallel decoder to generate new query features, and generating coarse prediction boxes by a box prediction head; projecting the generated query features into position features and object semantic features for calculating losses and training the network; using the projection results of the query features to score the coarse prediction boxes, and taking the one with the highest score as the visual positioning result.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal deep learning and 3D visual positioning, and in particular to a 3D visual positioning method and system based on adaptive position coding and parallel decoding. Background Art

[0002] Multimodal learning is key to enhancing AI's understanding and cognition of the real world. Vision and language are the most fundamental and important modalities. Visual grounding is an emerging visual language task that requires models to identify and locate objects in a scene based on natural language descriptions. While significant progress has been made in visual grounding on 2D images, 3D visual grounding remains a significant challenge given the sparse and complex structure of 3D point clouds and the diverse language descriptions inherent to the three-dimensional environment.

[0003] With significant progress in single-modal 3D object detection, the current main goal of 3D visual localization tasks is to address the "correct classification but incorrect localization" problem, which requires models to better understand the spatial layout of 3D scenes. Some methods attempt to primarily process visual features to better perceive the spatial relationships between detected objects in the scene. For example, "Viewpoint-Aware Visual Grounding in 3D Scenes" (2024 IEEE Conference on Computer Vision and Pattern Recognition) introduced a model that predicts the descriptor's viewpoint based on scene information. Other methods focus on processing natural language descriptions to achieve cross-modal alignment of positional and semantic information. For example, "EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding" (2023 IEEE Conference on Computer Vision and Pattern Recognition) achieves semantic understanding and spatial relationship matching by decoupling text; "ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance" (2023 IEEE International Conference on Computer Vision) expands a single base text into multiple geometrically consistent descriptions through the diverse language knowledge of a large language model.

[0004] However, current methods have two main problems. On the one hand, because the existing serial encoder-decoder architecture uses the same set of attention mechanisms to handle recognition and localization problems, object attribute features (such as shape and color) and spatial environment features are coupled together. The former requires the attention map to pay more attention to points near the prediction box, while the latter requires it to pay more attention to points that can form a spatial relationship with the target object in the description. The previous serial structure cannot decouple these two types of attention. On the other hand, the cross-attention module that processes visual features does not use the spatial information carried by the text. Without the guidance of natural language descriptions, queries can only roughly learn the features of neighboring points, which causes the query to focus on redundant and irrelevant spatial information, thereby causing attention to be distracted. Summary of the Invention

[0005] The present invention provides a 3D visual positioning method and system based on adaptive position encoding and parallel decoding. The designed dual-branch parallel decoder can not only enable the attention map to focus on the target object features and the spatial layout around it respectively, but also the adaptive position encoding methods of the two branches can make full use of multimodal input to guide the model to focus on points carrying valid information.

[0006] A 3D visual positioning method based on adaptive position coding and parallel decoding includes the following steps:

[0007] (1) Use the pre-trained RoBERTa model as a text encoder to process the input natural language text and obtain L text tokens and corresponding features D is the feature dimension;

[0008] (2) Use PointNet++ as a visual encoder to process the input 3D point cloud features and sample N seed points and corresponding features

[0009] (3) Use the cross encoder module of BUTD-DETR to intermodulate visual and text features to obtain the text features of text tokens and the visual features of the seed point

[0010] (4) Predict the confidence scores of N seed points and sort them, and select the top K candidate points with the highest scores as queries, whose characteristics are

[0011] (5) According to the semantics of natural language description, L text tokens are divided into L m target object attribute tokens and L s The surrounding space environment tokens are characterized by and

[0012] (6) The decoder has N layers. In each layer, the query first passes through the self-attention layer, whose output will be used as the query for each subsequent cross-attention layer. The decoder has two parallel branches, one for extracting the attribute features of the target object, called the target object branch, and the other for perceiving the spatial environment around each query, called the spatial environment branch. The visual features of the seed point The features T of the target object token and spatial environment token generated in step (5) serve as the key and value of the visual cross-attention layer in both branches. m and T sThey are used as the key and value of the text cross attention layer in the two branches respectively; finally, the visual-guided features and text-guided features in each branch are first fused together, and then the features from the two branches are fused. The output becomes the new query feature, and the box prediction head generates a coarse prediction box;

[0013] (7) Project the query features in each layer of the decoder into position features and object semantic features, both of which and the coarse prediction box in step (6) are used to calculate the loss and train the network;

[0014] (8) Use the projection result of the query feature of the last layer in step (7) as the coarse prediction box score of the decoder of the last layer in step (6), and take the one with the highest score as the visual positioning result.

[0015] In step (3), in the cross encoder module, the visual and text features are intermodulated, and the two branches with cross attention blocks take the features of one modality as query and the features of the other modality as key and value, respectively, to obtain the text features of the text token and the visual features of the seed point

[0016] In step (6), for the visual cross-attention layer, position encoding is added to the calculation of the attention map for refinement. The calculation process of the attention map is as follows:

[0017] First, the relative position between the seed point and the rough prediction box is calculated. The K queries filtered out in step (4) are processed separately to obtain K rough prediction boxes. The point on the surface of each box that is closest to each seed point is found, and the offset between the two in three dimensions is obtained. Δε=(e x ,e y ,e z ); then relative position encoding is performed, the nonlinear function F(·) is used to project the offset to the specified interval, and the three-dimensional offset is projected to a higher dimension using an MLP-based network. They will be assigned to each attention head as position encoding; finally, the relative position encoding is added to the attention map and the softmax activation function is applied to complete the calculation of visual attention.

[0018] The calculation formula of the attention map is:

[0019]

[0020] in and They represent the query and key-value in the attention calculation respectively, and ε represents the position encoding. The calculation formula is as follows:

[0021] ε=MLP(F(Δε)),

[0022] Where F(·) is a nonlinear function that projects the three-dimensional offset Δε into a specified interval. The MLP-based network then projects the offset into a higher dimension and assigns it to each attention head.

[0023] In step (6), in the spatial environment branch, the text-guided method is used as the gate to refine the attention map, and its calculation formula is updated as follows:

[0024]

[0025] Specifically, first calculate the N seed point features and L s The confidence between the spatial environment tokens; then take L s The maximum value of the confidence scores is used as the confidence score of each seed point. At this time, the higher the confidence score, the more relevant the seed point is to the spatial information in the text description. The sigmoid function is then used to map the confidence score to the [0,1] interval and replicate K times to generate Acts as a gate to enhance the attention map in the spatial context branch.

[0026] A 3D visual positioning system based on adaptive position coding and parallel decoding includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned 3D visual positioning method.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. We design a parallel decoder with two branches, each of which has a different attention map, respectively decoding the features of the target object and the surrounding spatial layout, which allows the query to focus on points related to the specific target of each branch.

[0029] 2. The present invention proposes an adaptive position encoding method for the two branches of the decoder, which makes full use of multimodal input to guide the module to focus on points carrying valid information. In particular, the spatial layout information carried by the natural language description is taken into account in the visual attention calculation of the spatial layout branch, so that the query can be focused on the spatial position in the description.

[0030] 3. Compared with other baseline algorithms, this invention has better model performance and surpasses the previous state-of-the-art technology on two widely used 3D visual positioning datasets ScanRefer and Nr3D. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the overall framework of a 3D visual positioning method based on adaptive position coding and parallel decoding according to the present invention;

[0032] Figure 2 This is a flow chart of the adaptive position coding in the present invention.

[0033] Figure 3 Schematic diagram of the calculation of the three-digit offset of the relative position in the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0035] like Figure 1 As shown in (a), a 3D visual localization method based on adaptive encoding and parallel decoding is proposed. The entire framework includes encoders for both visual and textual modalities, a cross encoder, and a parallel decoder with adaptive encoding. Specifically, the following steps are involved:

[0036] S01, unimodal encoder. For language modality, use pre-trained RoBERTa as a text encoder to process the input natural language text and obtain L text tokens and their features. The feature dimension D is set to 288, and L is determined by the length of the text. For the visual modality, PointNet++ is used as the visual encoder to process the input 3D point cloud features, sampling N seed points and their features. The feature dimension D is also set to 288, and N is 1024.

[0037] S02, Cross Encoder. The cross encoder module of BUTD-DETR is used to interoperate visual and text features. Two branches with cross attention blocks take the features of one modality as query and the features of the other modality as key and value to obtain the features of text tokens. and the visual features of the seed point

[0038] Before entering the parallel decoder, visual features and text features should be processed separately. For visual features, predict the confidence scores of N seed points and sort them, and select the top K candidate points with the highest scores as queries, whose features are At this time, K is set to 256. For language features, L text tokens are divided into L m target object attribute tokens and L s The surrounding space environment tokens are characterized by and For example, in the sentence “There is a dark brown wooden chair, placed in the table of the kitchen.”, {target object}-{chair} and {attribute adjective}-{dark brown wooden} are attribute tokens of the target object, while other parts such as {auxiliary object}-{table of the kitchen} and {relative adverb}-{placed in} are spatial context tokens. They serve as the key and value of the text cross-attention layer in the two branches of the subsequent S03.

[0039] S03, dual-branch parallel decoder. Figure 1 As shown in (b), in the hth layer, query V q h First through the self-attention layer:

[0040]

[0041] Its output will be used as the query for each subsequent cross-attention layer; the decoder has two parallel branches, one for extracting the attribute features of the target object, called the target object branch (m), and the other for perceiving the spatial environment around each query, called the spatial environment branch (s); the features of the seed point Serves as the key and value of the visual cross-attention layer (v) in both branches:

[0042]

[0043] Features T of target object token and spatial environment token m and T s As the key and value of the text cross attention layer (t) in the two branches respectively:

[0044]

[0045] Finally, the visually guided features and text-guided features in each branch are fused together:

[0046]

[0047]

[0048] Then the features from the two branches are fused:

[0049]

[0050] Its output becomes the new query feature and is input to the h+1 layer. The decoder has 6 layers in total.

[0051] For visual cross-attention layer The present invention adds position coding to the calculation of the attention map for refinement. The calculation formula of the attention map is:

[0052]

[0053] in and They represent the query and key-value in the attention calculation respectively, and ε represents the position encoding. The calculation formula is as follows:

[0054] ε=MLP(F(Δε)),

[0055] Where F(·) is a nonlinear function that projects the three-dimensional offset Δε into a specified interval. Then, the MLP-based network projects the offset into eight dimensions and distributes it to eight attention heads.

[0056] In order to represent the relative positions between N seed points (N=1024) and K coarse prediction boxes (K=256), the present invention finds the point closest to each seed point on the box surface and uses the three-dimensional offset

[0057] For the prediction box B=(x c ,y c ,z c ,l,w,h), if the seed point A=(x0,y0,z0) is outside the box, such as Figure 3 As shown, it is possible to (x c ,y c ,z c ) is used as the origin to establish a relative coordinate system. At this time, the relative coordinates of point A can be expressed as

[0058] For any point M = (x, y, z) on the surface of the prediction box, its relative distance MA to the seed point can be expressed as Since M is on the surface of the box, its coordinates should satisfy At the same time, since A is outside the box, its coordinates should satisfy Due to the symmetry of the relative coordinate system, only the positive semi-axis region in the three-dimensional space needs to be considered, such as Figure 3 As shown in the green part, point M and point A are mapped to and At this time, MA can be expressed as

[0059] In order to find the point on the box surface closest to the seed point, that is, to find the minimum value of MA, considering the above constraints, At this time, the three-bit offset can be expressed as:

[0060] Similarly, if point A' is inside the box, Its coordinates should satisfy Mapping points A′ and M to the positive semi-axis region of the coordinate system, the minimum value of MA′ can be expressed as If MA′ reaches its minimum value in the x direction, that is, Then e x Set to -MA′, the offset in the other two directions is 0; e y and e z Similarly, the offset is the negative of the distance to distinguish it from the seed points outside the box.

[0061] The position encoding mentioned above only uses relative positions to provide confidence for the attention map, which is inaccurate in the spatial context branch because it cannot determine whether the seed point carries valid spatial layout information. Figure 2 As shown, based on this, the present invention proposes a text-guided method as a gate to refine the attention map and update the calculation formula as follows:

[0062]

[0063] Specifically, first calculate the visual features of the seed point and surrounding space environment token features Confidence between:

[0064] confidence=V(T s ) T ,

[0065] in Then take the maximum confidence for the N seed points:

[0066] gate0=σ(max(conf idence)),

[0067] in σ is the sigmoid function, which is used to map the maximum confidence to the range [0,1]. When the confidence is high, the gate value is close to 1, which means that the corresponding value in the attention map can be effectively updated by the seed point; on the contrary, when the gate value is close to 0, the update of the attention map is ignored. Finally, gate0 is repeated K times to obtain This is used as the gate value to refine the attention map between the seed point features and the query features.

[0068] The present invention is trained and tested on two commonly used 3D vision-language datasets, ScanRefer and Nr3D, and compared with many other algorithms in the past, as shown in the table below. ScanRefer provides 51,583 manually annotated language descriptions of 11,046 objects in 800 3D scenes. Nr3D contains 41,503 language annotations, describing 76 object categories from 707 indoor scenes. For ScanRefer, Acc@mIoU is used as the evaluation metric, where m is set to 0.25 and 0.5. This metric indicates the proportion of the union intersection (IoU) of the predicted bounding box and the true bounding box that is greater than m. The higher the metric, the better the performance. For Nr3D, all experiments are evaluated under Acc@0.25IoU.

[0069] Table 1

[0070]

[0071] Table 2

[0072]

[0073] Table 1 shows the experimental results of our method compared with previous studies on the ScanRefer dataset. As can be seen, our method outperforms all previous methods in Acc@0.25IoU and Acc@0.5IoU, achieving 56.6% and 45.5% respectively. It outperforms our baseline method EDA in Acc@0.5IoU by 3.2% and the previous state-of-the-art method VPP-Net by 1.7%. It is worth noting that the methods marked with an "*" in the table use additional 2D image input. In contrast, our method achieves superior results without relying on additional 2D features.

[0074] Table 2 shows the experimental results of our method on the Nr3D dataset, comparing it to previous studies. The table provides test results for two baseline protocols: a ground truth protocol that takes the true labels as input, and a classification protocol that takes the classification results of the PointNet++ classifier on Nr3D as input. In the ground truth protocol, our method outperforms all previous methods by 67.8%, surpassing the previous state-of-the-art method, 3DRP-Net, by 1.9%. In the classification protocol, our method achieves 59.1% Acc@0.5IoU, which is 2.2% higher than the best result reported by VPP-Net and 7.0% higher than our baseline method, EDA.

[0075] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D visual positioning method based on adaptive position coding and parallel decoding, characterized in that: The steps include: (1) Use the pre-trained RoBERTa model as a text encoder to process the input natural language text and obtain L text tokens and corresponding features D is the feature dimension; (2) Use PointNet++ as a visual encoder to process the input 3D point cloud features and sample N seed points and corresponding features (3) Use the cross encoder module of BUTD-DETR to intermodulate visual and text features to obtain the text features of text tokens and the visual features of the seed point (4) Predict the confidence scores of N seed points and sort them, and select the top K candidate points with the highest scores as queries. Its characteristics are (5) According to the semantics of natural language description, L text tokens are divided into L m target object attribute tokens and L S A token of the surrounding space environment, Its characteristics are and (6) The decoder has N layers. In each layer, the query first passes through the self-attention layer, whose output will be used as the query for each subsequent cross-attention layer. The decoder has two parallel branches, one for extracting the attribute features of the target object, called the target object branch, and the other for perceiving the spatial environment around each query, called the spatial environment branch. The visual features of the seed point The features T of the target object token and spatial environment token generated in step (5) serve as the key and value of the visual cross-attention layer in both branches. m and T s They are used as the key and value of the text cross attention layer in the two branches respectively; finally, the visual-guided features and text-guided features in each branch are first fused together, and then the features from the two branches are fused. The output becomes the new query feature, and the box prediction head generates a coarse prediction box; (7) Project the query features in each layer of the decoder into position features and object semantic features, both of which and the coarse prediction box in step (6) are used to calculate the loss and train the network; (8) Use the projection result of the query feature of the last layer in step (7) as the coarse prediction box score of the decoder of the last layer in step (6), and take the one with the highest score as the visual positioning result.

2. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 1, characterized in that: In step (3), in the cross encoder module, the visual and text features are intermodulated, and the two branches with cross attention blocks take the features of one modality as query and the features of the other modality as key and value, respectively, to obtain the text features of the text token and the visual features of the seed point 3. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 1, characterized in that: In step (6), for the visual cross-attention layer, position encoding is added to the calculation of the attention map for refinement. The calculation process of the attention map is as follows: First, the relative position between the seed point and the rough prediction box is calculated. The K queries filtered out in step (4) are processed separately to obtain K rough prediction boxes. The point on the surface of each box that is closest to each seed point is found, and the offset between the two in three dimensions is obtained. Δε=(e x ,e y ,e z ); then relative position encoding is performed, the nonlinear function F(·) is used to project the offset to the specified interval, and the three-dimensional offset is projected to a higher dimension using an MLP-based network. They will be assigned to each attention head as position encoding; finally, the relative position encoding is added to the attention map and the softmax activation function is applied to complete the calculation of visual attention.

4. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 3, characterized in that: The calculation formula of the attention map is: in and They represent the query and key-value in the attention calculation respectively, and ε represents the position encoding. The calculation formula is as follows: ε=MLP(F(Δε)), Where F(·) is a nonlinear function that projects the three-dimensional offset Δε into a specified interval. The MLP-based network then projects the offset into a higher dimension and assigns it to each attention head.

5. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 4, characterized in that: In step (6), in the spatial environment branch, the text-guided method is used as the gate to refine the attention map, and its calculation formula is updated as follows: Specifically, first calculate the N seed point features and L s The confidence between the spatial environment tokens; then take L s The maximum value of the confidence scores is used as the confidence score of each seed point. At this time, the higher the confidence score, the more relevant the seed point is to the spatial information in the text description. The sigmoid function is then used to map the confidence score to the [0,1] interval and replicate K times to generate Acts as a gate to enhance the attention map in the spatial context branch.

6. A 3D visual positioning system based on adaptive position coding and parallel decoding, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the 3D visual positioning method according to any one of claims 1 to 5.

Images