3D visual positioning method and system based on adaptive position coding and parallel decoding

Abstract

The invention discloses a 3D visual positioning method and system based on adaptive position coding and parallel decoding, and the method comprises the steps: processing an inputted natural language text through a text encoder, and obtaining a text token and features; processing 3D point cloud input by using a visual encoder to obtain seed points and features; intermodulation is carried out on the visual features and the text features by adopting a cross encoder, and the text features and the visual features are updated; predicting confidence scores of the seed points and sorting, and selecting a candidate point with the highest score as query; dividing the text tokens into target object attribute tokens and surrounding space environment tokens according to semantics described by a natural language; generating a new query feature by using a double-branch parallel decoder, and generating a coarse prediction frame by using a frame prediction head; projecting the generated query features into the position features and the object semantic features for calculating loss and training a network; and scoring the rough prediction frame by using the projection result of the query feature, and taking the highest score as a visual positioning result.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal deep learning and 3D visual positioning, and in particular to a 3D visual positioning method and system based on adaptive position coding and parallel decoding. Background Art

[0002] Multimodal learning is the key to enhancing artificial intelligence's understanding and cognition of the real world, among which the most basic and important modalities are vision and language. Among them, visual grounding is an emerging visual language task that requires the model to identify and locate target objects in the scene based on natural language descriptions. Although the visual grounding task on 2D images has achieved remarkable results, considering the sparse and complex structure of 3D point clouds and the diversity of language descriptions caused by the three-dimensional space environment, the 3D visual grounding task still faces huge challenges.

[0003] With the significant progress made in the unimodal 3D object detection task, the main goal of the 3D visual positioning task at this stage is to solve the problem of "correct classification but incorrect positioning", which requires the model to better understand the spatial layout in the three-dimensional scene. Some methods try to mainly process visual features to better perceive the spatial relationship between objects detected in the scene, such as "Viewpoint-Aware Visual Grounding in 3D Scenes" (2024 IEEE Conference on Computer Vision and Pattern Recognition), which introduces a model that predicts the descriptor's perspective based on scene information. Other methods focus on processing natural language descriptions to achieve cross-modal alignment of location information and semantic information. For example, "EDA: Explicit Text-Decoupling and Dense Alignmentfor 3D Visual Grounding" (2023 IEEE Conference on Computer Vision and Pattern Recognition) achieves semantic understanding and spatial relationship matching by decoupling text; "ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance" (2023 IEEE International Conference on Computer Vision) expands a single base text into multiple geometrically consistent descriptions through the diverse language knowledge of a large language model.

[0004] However, there are two main problems with current methods. On the one hand, since the existing serial encoder-decoder architecture uses the same set of attention mechanisms to handle recognition and localization problems, object attribute features (such as shape and color) and spatial environment features are coupled together. The former requires the attention map to pay more attention to points near the prediction box, while the latter requires it to pay more attention to points that can form a spatial relationship with the target object in the description. The past serial structure cannot decouple these two types of attention. On the other hand, the cross-attention module that processes visual features does not use the spatial information carried by the text. Without the guidance of natural language descriptions, queries can only roughly learn the features of neighboring points, which causes queries to focus on redundant and irrelevant spatial information, thereby causing attention distraction. Summary of the invention

[0005] The present invention provides a 3D visual positioning method and system based on adaptive position coding and parallel decoding. The designed dual-branch parallel decoder can not only make the attention map focus on the target object features and the spatial layout around it respectively, but also the adaptive position coding methods of the two branches can make full use of multimodal input to guide the model to focus on points carrying valid information.

[0006] A 3D visual positioning method based on adaptive position coding and parallel decoding comprises the following steps:

[0007] (1) Use the pre-trained RoBERTa model as a text encoder to process the input natural language text and obtain L text tokens and corresponding features D is the feature dimension;

[0008] (2) Use PointNet++ as a visual encoder to process the input 3D point cloud features and sample N seed points and corresponding features.

[0009] (3) Use the cross encoder module of BUTD-DETR to interoperate the visual and text features to obtain the text features of the text token and the visual features of the seed point

[0010] (4) Predict the confidence scores of N seed points and sort them, and select the top K candidate points with the highest scores as queries, whose characteristics are

[0011] (5) According to the semantics of the natural language description, L text tokens are divided into L m target object attribute tokens and L s The surrounding space environment tokens are characterized by and

[0012] (6) The decoder has N layers. In each layer, the query first passes through the self-attention layer, whose output will be used as the query for each subsequent cross-attention layer. The decoder has two parallel branches, one for extracting the attribute features of the target object, called the target object branch, and the other for perceiving the spatial environment around each query, called the spatial environment branch. The visual features of the seed point The features T of the target object token and spatial context token generated in step (5) serve as the key and value of the visual cross-attention layer in both branches. m and T sThey are used as the key and value of the text cross attention layer in the two branches respectively; finally, the visual-guided features and text-guided features in each branch are fused together, and then the features from the two branches are fused, and the output becomes the new query feature, and the rough prediction box is generated by the box prediction head;

[0013] (7) Project the query features in each layer of the decoder into position features and object semantic features. Both of them and the coarse prediction box in step (6) are used to calculate the loss and train the network;

[0014] (8) Use the projection result of the query feature of the last layer in step (7) as the coarse prediction box score of the decoder of the last layer in step (6), and take the one with the highest score as the visual positioning result.

[0015] In step (3), in the cross encoder module, the visual and text features are interoperated, and the two branches with cross attention blocks take the features of one modality as the query and the features of the other modality as the key and value, respectively, to obtain the text features of the text token and the visual features of the seed point

[0016] In step (6), for the visual cross-attention layer, position encoding is added to the calculation of the attention map for refinement. At this time, the calculation process of the attention map is as follows:

[0017] First, the relative position between the seed point and the rough prediction box is calculated. The K queries filtered out in step (4) are processed separately to obtain K rough prediction boxes. The point on the surface of each box that is closest to each seed point is found, and the offset between the two in three dimensions is obtained. x ,e y ,e z ); then relative position encoding is performed, the nonlinear function F(·) is used to project the offset to the specified interval, and the three-dimensional offset is projected to a higher dimension using an MLP-based network. They will be assigned to each attention head as position encoding; finally, the relative position encoding is added to the attention map and the softmax activation function is applied to complete the calculation of visual attention.

[0018] The calculation formula of the attention map is:

[0019]

[0020] in and They represent the query and key-value in the attention calculation respectively, and ε represents the position encoding. The calculation formula is as follows:

[0021] ε=MLP(F(Δε)),

[0022] Where F(·) is a nonlinear function that projects the three-dimensional offset Δε to a specified interval. Then the MLP-based network projects the offset to a higher dimension and assigns it to each attention head.

[0023] In step (6), in the spatial environment branch, the text-guided method is used as the gate to refine the attention map, and its calculation formula is updated as follows:

[0024]

[0025] Specifically, first calculate the N seed point features and L s The confidence between the spatial environment tokens; then take L s The maximum value of the confidence scores is used as the confidence of each seed point. At this time, the higher the confidence score, the more relevant the seed point is to the spatial information in the text description. Then the sigmoid function is used to map the confidence to the [0,1] interval and replicate K times to generate Acts as a gate to enhance the attention map in the spatial context branch.

[0026] A 3D visual positioning system based on adaptive position coding and parallel decoding includes a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the above-mentioned 3D visual positioning method.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The present invention designs a parallel decoder with two branches, each branch has a different attention map, which decodes the features of the target object and the surrounding spatial layout respectively, which allows the query to focus on points related to the specific target of each branch.

[0029] 2. The present invention proposes an adaptive position encoding method for the two branches of the decoder, which makes full use of multimodal input to guide the module to focus on points carrying valid information. In particular, the spatial layout information carried by the natural language description is taken into account in the visual attention calculation of the spatial layout branch, so that the query can be focused on the spatial position in the description.

[0030] 3. Compared with other baseline algorithms, the present invention has better model performance and surpasses the most advanced technologies in the past on two widely used 3D visual positioning datasets ScanRefer and Nr3D. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall framework of a 3D visual positioning method based on adaptive position coding and parallel decoding of the present invention;

[0032] Figure 2 This is a flow chart of the adaptive position coding in the present invention.

[0033] Figure 3 It is a schematic diagram of calculating the three-dimensional offset of the relative position in the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0035] like Figure 1 As shown in (a), a 3D visual positioning method based on adaptive encoding and parallel decoding, the entire framework includes encoders for both visual and text modalities, a cross encoder, and a parallel decoder with adaptive encoding. Specifically, the following steps are included:

[0036] S01, unimodal encoder. For the language modality, use the pre-trained RoBERTa as the text encoder to process the input natural language text and obtain L text tokens and their features. The feature dimension D is set to 288, and L is determined by the length of the text. For the visual modality, PointNet++ is used as the visual encoder to process the input 3D point cloud features and sample N seed points and their features. The feature dimension D is also set to 288, and N is 1024.

[0037] S02, Cross Encoder. The cross encoder module of BUTD-DETR is used to interoperate visual and text features. Two branches with cross attention blocks take the features of one modality as query and the features of the other modality as key and value to obtain the features of text tokens. and the visual features of the seed point

[0038] Before entering the parallel decoder, visual features and text features should be processed separately. For visual features, predict the confidence scores of N seed points and sort them, and select the top K candidate points with the highest scores as queries, whose features are At this time, K is set to 256. For language features, L text tokens are divided into L m target object attribute tokens and L s The surrounding space environment tokens are characterized by and For example, in “There is a dark brown wooden chair, placed in the table of the kitchen.”, {target object}-{chair} and {attribute adjective}-{dark brown wooden} are target object attribute tokens, and other parts such as {auxiliary object}-{table of the kitchen}, {relation adverb}-{placed in} are surrounding space environment tokens. They serve as the key and value of the text cross-attention layer in the two branches of the subsequent S03.

[0039] S03, dual-branch parallel decoder. Figure 1 As shown in (b), in the hth layer, query V q h First through the self-attention layer:

[0040]

[0041] Its output will be used as the query for each subsequent cross-attention layer; the decoder has two parallel branches, one for extracting the attribute features of the target object, called the target object branch (m), and the other for perceiving the spatial environment around each query, called the spatial environment branch (s); the features of the seed point Serves as the key and value of the visual cross-attention layer (v) in both branches:

[0042]

[0043] Features T of target object token and spatial environment token m and T s As the key and value of the text cross attention layer (t) in the two branches respectively:

[0044]

[0045] Finally, the visually guided features and text-guided features in each branch are fused together:

[0046]

[0047]

[0048] Then the features from the two branches are fused:

[0049]

[0050] Its output becomes the new query feature and is input to the h+1 layer. The decoder has 6 layers in total.

[0051] For the visual cross-attention layer The present invention adds position coding to the calculation of the attention map for refinement. At this time, the calculation formula of the attention map is:

[0052]

[0053] in and They represent the query and key-value in the attention calculation respectively, and ε represents the position encoding. The calculation formula is as follows:

[0054] ε=MLP(F(Δε)),

[0055] Where F(·) is a nonlinear function that can project the three-dimensional offset Δε to a specified interval. Then the MLP-based network can project the offset into 8 dimensions and assign them to 8 attention heads.

[0056] In order to represent the relative positions between N seed points (N=1024) and K rough prediction boxes (K=256), the present invention finds the point closest to each seed point on the box surface and uses the three-dimensional offset

[0057] For the prediction box B = (x c ,y c ,z c ,l,w,h), if the seed point A=(x0,y0,z0) is outside the box, such as Figure 3 As shown, it is possible to c ,y c ,z c ) is used as the origin to establish a relative coordinate system. At this time, the relative coordinate of point A can be expressed as

[0058] For any point M = (x, y, z) on the surface of the prediction box, its relative distance MA to the seed point can be expressed as Since M is on the surface of the box, its coordinates should satisfy At the same time, since A is outside the box, its coordinates should satisfy Due to the symmetry of the relative coordinate system, only the positive semi-axis region in the three-dimensional space needs to be considered, such as Figure 3 As shown in the green part, point M and point A are mapped to and At this time, MA can be expressed as

[0059] In order to find the point on the box surface closest to the seed point, that is, to find the minimum value of MA, considering the above constraints, At this time, the three-bit offset can be expressed as:

[0060] Similarly, if point A' is inside the box, Its coordinates should satisfy Mapping points A′ and M to the positive semi-axis region of the coordinate system, the minimum value of MA′ can be expressed as If MA′ reaches its minimum value in the x direction, that is, Then x Set to -MA′, the offset in the other two directions is 0; e y and e z Similarly, the offset is the negative of the distance to distinguish it from the seed points outside the box.

[0061] The position encoding mentioned above only uses relative positions to provide confidence for the attention map, which is inaccurate in the spatial context branch because it cannot determine whether the seed point carries valid spatial layout information. Figure 2 As shown, on this basis, the present invention proposes a text-guided method as a gate to refine the attention map and update the calculation formula as follows:

[0062]

[0063] Specifically, first calculate the visual features of the seed point and surrounding space environment token features Confidence between:

[0064] confidence=V(T s ) T ,

[0065] in Then take the maximum confidence for the N seed points:

[0066] gate0=σ(max(conf idence)),

[0067] in σ is the sigmoid function, which is used to map the maximum confidence to the range [0,1]. When the confidence is high, the gate value approaches 1, which means that the corresponding value in the attention map can be effectively updated by the seed point; conversely, when the gate value approaches 0, the update of the attention map is ignored. Finally, gate0 is repeated K times to obtain This is used as the gate value to refine the attention map between the seed point features and the query features.

[0068] The present invention is trained and tested on two commonly used 3D vision-language datasets, ScanRefer and Nr3D, and compared with many other algorithms in the past, as shown in the following table. ScanRefer provides 51,583 manually annotated language descriptions of 11,046 objects in 800 3D scenes. Nr3D contains 41,503 language annotations, describing 76 object categories from 707 indoor scenes. For ScanRefer, Acc@mIoU is used as the evaluation metric, where m is set to 0.25 and 0.5. This metric indicates the proportion of the intersection over union (IoU) of the predicted bounding box and the true bounding box greater than m. The higher the metric, the better the performance. For Nr3D, all experiments were evaluated under Acc@0.25IoU.

[0069] Table 1

[0070]

[0071] Table 2

[0072]

[0073] Table 1 shows the experimental results of the present invention compared with previous studies on the ScanRefer dataset. It can be seen that the performance of the present invention on Acc@0.25IoU and Acc@0.5IoU is better than all previous methods, reaching 56.6% and 45.5% respectively. It is 3.2% higher than the baseline method EDA of the present invention on Acc@0.5IoU, and 1.7% higher than the previous state-of-the-art method VPP-Net. It is worth mentioning that the methods marked with "*" in the table use additional input of 2D images. In comparison, the present method can obtain better results without relying on additional 2D features to assist.

[0074] Table 2 shows the experimental results of our invention compared with previous studies on the Nr3D dataset. The table provides test results for two baseline protocols, namely the truth protocol with the true label as input, and the classification protocol with the classification result of the PointNet++ classifier on Nr3D as input. In the truth protocol, our invention outperforms all previous methods by 67.8%, exceeding the previous state-of-the-art method 3DRP-Net by 1.9%. In the classification protocol, our invention achieves 59.1% Acc@0.5IoU, which is 2.2% higher than the best result reported by VPP-Net and 7.0% higher than our baseline method EDA.

[0075] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D visual positioning method based on adaptive position coding and parallel decoding, characterized in that: The steps include: (1) Use the pre-trained RoBERTa model as a text encoder to process the input natural language text and obtain L text tokens and corresponding features D is the feature dimension; (2) Use PointNet++ as a visual encoder to process the input 3D point cloud features and sample N seed points and corresponding features. (3) Use the cross encoder module of BUTD-DETR to interoperate the visual and text features to obtain the text features of the text token and the visual features of the seed point (4) Predict the confidence scores of N seed points and sort them, and select the top K candidate points with the highest scores as queries. Its characteristics are (5) According to the semantics of the natural language description, L text tokens are divided into L m target object attribute tokens and L S Ambient space tokens, Its characteristics are and (6) The decoder has N layers. In each layer, the query first passes through the self-attention layer, whose output will be used as the query for each subsequent cross-attention layer. The decoder has two parallel branches, one for extracting the attribute features of the target object, called the target object branch, and the other for perceiving the spatial environment around each query, called the spatial environment branch. The visual features of the seed point The features T of the target object token and spatial context token generated in step (5) serve as the key and value of the visual cross-attention layer in both branches. m and T s They are used as the key and value of the text cross attention layer in the two branches respectively; finally, the visual-guided features and text-guided features in each branch are fused together, and then the features from the two branches are fused, and the output becomes the new query feature, and the rough prediction box is generated by the box prediction head; (7) Project the query features in each layer of the decoder into position features and object semantic features. Both of them and the coarse prediction box in step (6) are used to calculate the loss and train the network; (8) Use the projection result of the query feature of the last layer in step (7) as the coarse prediction box score of the decoder of the last layer in step (6), and take the one with the highest score as the visual positioning result.

2. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 1, characterized in that: In step (3), in the cross encoder module, the visual and text features are interoperated, and the two branches with cross attention blocks take the features of one modality as the query and the features of the other modality as the key and value, respectively, to obtain the text features of the text token and the visual features of the seed point 3. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 1, characterized in that: In step (6), for the visual cross-attention layer, position encoding is added to the calculation of the attention map for refinement. At this time, the calculation process of the attention map is as follows: First, the relative position between the seed point and the rough prediction box is calculated. The K queries filtered out in step (4) are processed separately to obtain K rough prediction boxes. The point on the surface of each box that is closest to each seed point is found, and the offset between the two in three dimensions is obtained. x ,e y ,e z ); then relative position encoding is performed, the nonlinear function F(·) is used to project the offset to the specified interval, and the three-dimensional offset is projected to a higher dimension using an MLP-based network. They will be assigned to each attention head as position encoding; finally, the relative position encoding is added to the attention map and the softmax activation function is applied to complete the calculation of visual attention.

4. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 3, characterized in that: The calculation formula of the attention map is: in and They represent the query and key-value in the attention calculation respectively, and ε represents the position encoding. The calculation formula is as follows: ε=MLP(F(Δε)), Where F(·) is a nonlinear function that projects the three-dimensional offset Δε to a specified interval. Then the MLP-based network projects the offset to a higher dimension and assigns it to each attention head.

5. The 3D visual positioning method based on adaptive position coding and parallel decoding according to claim 4, characterized in that: In step (6), in the spatial environment branch, the text-guided method is used as the gate to refine the attention map, and its calculation formula is updated as follows: Specifically, first calculate the N seed point features and L s The confidence between the spatial environment tokens; then take L s The maximum value of the confidence scores is used as the confidence of each seed point. At this time, the higher the confidence score, the more relevant the seed point is to the spatial information in the text description. Then the sigmoid function is used to map the confidence to the [0,1] interval and replicate K times to generate Acts as a gate to enhance the attention map in the spatial context branch.

6. A 3D visual positioning system based on adaptive position coding and parallel decoding, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the 3D visual positioning method according to any one of claims 1 to 5.

Images