A 3D visual positioning method and system with relative position perception capability

Abstract

This paper proposes a 3D visual localization method and system with relative position perception capabilities, belonging to the field of 3D visual localization. The method samples seed points from a 3D point cloud and extracts seed point features. Furthermore, the method extracts text features from a language description sentence. Key points of the target object associated with the language description sentence are sampled from the seed points. A 3D relative position perception network containing a text-visual dual decoder is used to decode and output the predicted bounding box and reference confidence score of the target object. The key point with the highest reference confidence score is selected to regress the target bounding box, and the object within the bounding box is used as the localization result. A soft labeling strategy is also designed to assist training and further improve visual localization performance. Extensive experiments have demonstrated the efficiency and effectiveness of the method.

Description

Technical Field

[0001] The present invention relates to the field of 3D visual positioning, and in particular to a 3D visual positioning method and system with relative position perception capability. Background Art

[0002] The goal of the 3D visual localization task is to locate the target object in a 3D point cloud based on language cues in any format. It has extremely wide applications in the fields of VR / AR and autonomous robotics. Previous 3D visual localization methods can be roughly divided into two categories: two-stage methods and single-stage methods. The former follows the "Detection-and-Rank" paradigm. Due to the considerable flexibility of this architecture, they are mainly used to explore the advantages of different object relationship modeling methods in the task of distinguishing target objects. The latter directly predicts the bounding box of the target object by integrating visual and textual features and enhances the object attribute representation by removing unreliable candidate generation stages.

[0003] For two-stage methods, model performance is highly dependent on the quality of object candidates. Due to the sparsity and irregularity of 3D point clouds, sparse candidates may miss the target object, while dense candidates lead to excessive computational redundancy. Furthermore, existing methods do not consider the relative spatial relationships between objects; however, for humans, relative spatial relationships are crucial for distinguishing a specific object from multiple objects of the same category in a 3D scene. Summary of the Invention

[0004] To alleviate these issues, the present invention proposes a 3D visual localization method and system with relative position awareness. This method combines and further enhances the modeling approach of the two-stage approach and the candidate-free nature of the single-stage approach, while avoiding the shortcomings of both approaches. For relational modeling, the present invention incorporates a novel 3D relative position-aware multi-head attention module that captures object relationships along multiple directions and fully considers the interaction between relative position and object features, which is previously ignored in two-stage approaches.

[0005] To achieve the above object, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention proposes a 3D visual positioning method with relative position perception capability, comprising the following steps:

[0007] 1) Sampling seed points from the 3D point cloud and extracting seed point features; and extracting text features of the language description sentence;

[0008] 2) Sampling key points of the target object related to the language description sentence from the seed point;

[0009] 3) A 3D relative position-aware network containing a text-visual dual decoder is used to decode and output the predicted bounding box and reference confidence score of the target object respectively. The key point with the highest reference confidence score is selected to regress the target bounding box, and the object within the bounding box is used as the positioning result.

[0010] Furthermore, the text-vision dual decoder includes a text decoder and a vision decoder, both of which are implemented based on the Transformer decoder structure, and at least one original attention module in the Transformer decoder is replaced by a multi-head attention module with 3D relative position perception.

[0011] Furthermore, the 3D relative position perception multi-head attention module is referred to as 3DRP-MA, and its calculation process is as follows:

[0012] 3.1) For the input query Q, key K and value V, introduce relative position encoding and calculate the attention weight:

[0013]

[0014] Among them, d ij is the relative distance from point i to point j, a i,j is the attention weight of point i and point j, p(·) is the relative distance mapping function, The relative distance from point i to point j is mapped into a pair of relative position codes after discrete values. The superscript T stands for transposition, and q i For the i-th query, k j is the i-th key, d represents k j Dimensions;

[0015] 3.2) Calculate the output of the attention module for single-head relative position perception based on the attention weight

[0016]

[0017] Among them, M0 is the number of key points, v i The i-th value, z i is the i-th output;

[0018] 3.3) Capture the relative spatial relationship between points from multiple directions, encode the x, y, z coordinates and the relative distance after discretization under the Euclidean metric, and express them as D x 、D y 、D z and D e ; D x 、D y 、D z and D eAs the relative distance mapping function p(·), four single-head relative position-aware attention modules are established and combined to obtain the output of the 3D relative position-aware multi-head attention module.

[0019] Furthermore, the text decoder includes a first 3DRP-MA module, an original multi-head attention module, and a first feedforward network module;

[0020] First, the key point features are used as the input sequence, and after the projection matrix transformation, the query Q, key K and value V are generated, which are input into the first 3DRP-MA module, and the residual connection and normalization calculation are performed through the Add&Norm layer to obtain Z0; based on Z0 and the text features obtained in step 1), the query Q, key K and value V for input into the multi-head attention module are generated, and the output of the multi-head attention module is subjected to the Add&Norm layer for residual connection and normalization calculation to obtain Z′; Z′ is used as the input of the first feedforward network module, and the residual connection and normalization calculation are performed through the Add&Norm layer; that is, all the calculations of the text decoder are completed.

[0021] Furthermore, the visual decoder includes a second 3DRP-MA module, a third 3DRP-MA module, and a second feedforward network module;

[0022] First, the key point features are used as the input sequence, and the query Q, key K and value V are generated after the projection matrix transformation, and are input into the second 3DRP-MA module. The residual connection and normalization calculation are performed through the Add&Norm layer to obtain Z0′; based on Z0′ and the seed point features obtained in step 1), the query Q, key K and value V for input into the third 3DRP-MA module are generated, and the output of the third 3DRP-MA module is subjected to the Add&Norm layer for residual connection and normalization calculation to obtain Z″; Z″ is used as the input of the second feedforward network module, and the residual connection and normalization calculation are performed through the Add&Norm layer; that is, all the calculations of the visual decoder are completed.

[0023] Furthermore, the outputs of the text decoder and visual decoder are passed through a multi-layer perceptron to generate a reference confidence score and a predicted bounding box, respectively.

[0024] Furthermore, the relative distance mapping function in step 3.1) is as follows:

[0025]

[0026] Where [·] is the rounding function; sign(·) is the sign function, α and β are thresholds, p(d) is the relative distance after mapping, k is a natural number, and d is the distance.

[0027] Furthermore, a soft label strategy is used to train the 3D relative position perception network. The soft label strategy is specifically as follows: according to the IoU value ranking of the predicted bounding box, the key points are reordered and the top N key points are ranked. s The target label of the key points with a value greater than 0.25 is set to The target labels of other key points are set to 0, where N s is the threshold, The calculation formula is:

[0028]

[0029] in, is the target label of the key point ranked i, and σ is a parameter that controls the smoothness of the result distribution.

[0030] In a second aspect, the present invention proposes a 3D visual positioning system with relative position perception capability, comprising:

[0031] A seed point extraction module is used to extract seed points and their features from 3D point clouds;

[0032] A text encoding module, which is used to extract text features of language description sentences;

[0033] A sampling module, which is used to sample key points of the target object related to the language description sentence from the seed point;

[0034] A 3D relative position perception network module, which is used to decode and output the predicted bounding box and reference confidence score of the target object using a 3D relative position perception network including a text-visual dual decoder;

[0035] The localization module is used to select the keypoint with the highest reference confidence score to regress the target bounding box, and the object within the bounding box is taken as the localization result.

[0036] Furthermore, the 3D relative position perception network module includes:

[0037] The text decoder is obtained by replacing the first original attention module in the Transformer decoder with a 3D relative position-aware multi-head attention module. It is used to generate decoding results based on key point features and text features.

[0038] The visual decoder is obtained by replacing the two original attention modules in the Transformer decoder with a 3D relative position-aware multi-head attention module, which is used to generate decoding results based on key point features and seed point features;

[0039] a first multi-layer perceptron, configured to generate a reference confidence score according to a decoding result of the text decoder;

[0040] The second multi-layer perceptron is used to generate a predicted bounding box according to the decoding result of the visual decoder.

[0041] There is sufficient evidence to show that the present invention generally outperforms all previous state-of-the-art methods for 3D visual localization tasks. The present invention has the following beneficial effects:

[0042] 1. This paper proposes a dual-decoder, single-stage 3D visual localization framework, called the 3D Relative Position Perception Network, which comprehensively captures the relative spatial relationships between objects in 3D space to achieve accurate target object localization.

[0043] 2. This paper designs a novel 3D relative position-aware multi-head attention module to effectively and synchronously model relative spatial position relationships from different directions. In addition, this paper designs a soft labeling strategy to assist training and further improve visual localization performance.

[0044] 3. Large-scale experiments demonstrate the efficiency and effectiveness of the present invention, which exhibits state-of-the-art performance on three mainstream benchmark datasets: ScanRefer, Nr3D, and Sr3D. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the overall framework of the 3D visual positioning method with relative position perception capability shown in the present invention;

[0046] Figure 2 It is a structural diagram of the multi-head attention module for 3D relative position perception shown in the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described and illustrated below in conjunction with the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the 3D visual positioning method with relative position perception capability proposed in the present invention mainly includes the following steps:

[0049] 1. Use the pre-trained model to sample M seed points from the input 3D point cloud and extract the features of the seed points. Use the sampling module to select the key points of the object related to the language description sentence from the seed points. Use the pre-trained language encoder to encode the text features of the language description sentence. Specifically:

[0050] 1.1 There are two inputs in the 3D visual positioning task, one of which is a 3D point cloud The mathematical form is the 3D coordinates of N points and K-dimensional auxiliary features; the other input is a natural language description statement of any form and length L.

[0051] 1.2 Using the pre-trained PointNet++ network from P cloud Sample M seed points and encode point features of dimension C, and record the seed points as

[0052] 1.3 Use the sampling module to select M0 points related to the language description sentence from M seed points and record them as key points

[0053] 1.4 Use the pre-trained natural language encoder to encode the language description sentence of length L into a text feature of dimension D, recorded as

[0054] 2. Establish a multi-head attention module for 3D relative position perception; specifically:

[0055] 2.1 The key points obtained in step 1.3 As an input sequence to a single-head relative position-aware attention module, the query is calculated key value for:

[0056] q i =p i W Q ,k i =p i W K ,v i =p i W V

[0057] Among them, W Q ,W K ,W V is the projection matrix, q i 、k i 、v i are the query, key and value corresponding to the i-th key point, respectively, p i is the i-th seed point.

[0058] 2.2 According to the relative position between the input key points and the relative distance mapping function, the relative position encoding is introduced into the attention module, and the attention weight a in the relative position attention is calculated. i,j The calculation method is rewritten as:

[0059]

[0060] Among them, d ij is the relative distance from point i to point j; p(·)∈[0,2k) is an index function that maps continuous distance to discrete values, and k is a natural number; is a learnable relative position encoding, is a pair of relative position codes after mapping the relative distance from point i to point j into discrete values, d z is the dimension of the elements in the output sequence of the single-head relative position-aware attention module, and the superscript T represents transposition.

[0061] In a specific implementation of the present invention, since the points in the 3D point cloud are unevenly distributed in the Euclidean space, the relative distances between elements are continuously distributed. In order to enhance the relative spatial information and reduce the computational cost, the continuous 3D relative distances are mapped to discrete integers belonging to a finite set. The index function p(·) used to implement the discreteness is as follows:

[0062]

[0063] Among them, [·] is the rounding function; sign(·) is the sign function, α and β are thresholds, and p(d) is the relative distance of the discretization after mapping.

[0064] 2.3 Combined with the above results, calculate the output of the single-head relative position perception attention module

[0065]

[0066] 2.4 Capturing the relative spatial relationship between points from multiple directions, encoding the x, y, z coordinates and the relative distance after discretization under the Euclidean metric, respectively expressed as D x 、D y 、D z and D e ;

[0067] 2.5 respectively with D x 、D y 、D z and D e As the relative position encoding introduced in step 2.2 above, four single-head relative position perception attention modules are established and combined to establish Figure 2 The 3D relative position-aware multi-head attention module shown is denoted as 3DRP-MA.

[0068] 3. Based on the 3D relative position perception multi-head attention module established in step 2, a text-visual dual decoder structure is established to simulate the relative spatial relationship between objects; specifically:

[0069] 3.1 Build a text decoder based on the decoder in Transformer, input the key point P0 obtained in step 1.3 into the 3DRP-MA layer, and perform residual connection and normalization calculation through the Add&Norm layer to obtain the output result Z0; use Z0 and the text feature T obtained in step 1.4 as the input of the MA (i.e., Multi-Attention) layer, where Z0 is used to calculate the Query and T is used to calculate the Key and Value, and perform residual connection and normalization calculation through the Add&Norm layer to obtain the output result Z′; use Z′ as the input of the FFN (i.e., Feed Forward Network) layer, and perform residual connection and normalization calculation through the Add&Norm layer; that is, complete the construction and all calculations of the text decoder.

[0070] 3.2 Based on the decoder in Transformer, a visual decoder is constructed. The key point P0 obtained in step 1.3 is input into the 3DRP-MA layer, and the residual connection and normalization calculation are performed through the Add&Norm layer to obtain the output result Z0′; Z0′ and the seed point P obtained in step 1.2 are combined. S As the input of the 3DRP-MA layer, Z0′ is used to calculate Query, P S It is used to calculate the Key and Value, and performs residual connection and normalization calculation through the Add&Norm layer to obtain the output result Z″; Z″ is used as the input of the FFN layer, and performs residual connection and normalization calculation through the Add&Norm layer; that is, the construction and all calculations of the visual decoder are completed.

[0071] 3.3 Combining text decoder and visual decoder to build Figure 1 The text-visual dual decoder structure shown;

[0072] 4. The output of the text decoder is calculated through an MLP (Multilayer Perceptron) to obtain a reference confidence score; the output of the visual decoder is calculated through an MLP to obtain a predicted bounding box of the target object.

[0073] 5. Build as Figure 1 The 3D relative position perception network shown in the figure includes a text encoding module, a sampling module, a text-visual dual decoder module, a multi-layer perceptron module, etc. During the training phase, the losses of each module and the final comprehensive training loss are as follows:

[0074] 5.1 Keypoint Sampling Loss L ks : This embodiment adopts the standard Focal Loss loss function;

[0075] 5.2 Detection Loss Ldet : The object detection task includes 5 prediction tasks, namely:

[0076] 1) Object confidence prediction: This embodiment uses conventional confidence loss, denoted as L obj ;

[0077] 2) Bounding box classification: This example uses the standard cross entropy loss, denoted as L cls ;

[0078] 3) Center point offset prediction: This embodiment uses the standard cross entropy loss, denoted as L center-off ;

[0079] 4) Size classification: This example uses the standard cross entropy loss, denoted as L sz-cls ;

[0080] 5) Size offset prediction: This embodiment uses the standard smooth L1 loss, denoted as L sz-off ;

[0081] The final detection loss L det It is a weighted combination of the above five losses:

[0082] L det =β1L obj +β2L cls +β3L center-off +β4L sz-cls +β5L sz-off

[0083] 5.3 Text Classification Loss L text : In this embodiment, the standard cross entropy loss is used;

[0084] 5.4 Visual Localization Loss L vg :N generated using the following formula s Soft labels are used to supervise the predicted reference scores of the M0 key points, and the key points are re-ranked according to the IoU value ranking of the prediction box. s The target label of the key point with an IoU value greater than 0.25 is set to The target labels of other key points are set to 0, where N s is the threshold. And predict the reference score S, supervise the M0 key points;

[0085] In this embodiment, the specific calculation function of the soft label strategy is:

[0086]

[0087] Where i∈{0,…,Ns} represents the IoU ranking is the i-th, Represents the target label of the key point ranked i; i>N s When setting i≤N s When the IoU value of the predicted bounding box corresponding to the i-th key point is ≤ 0.25, set At the same time, set σ=[N s / 3] to control the smooth distribution of results;

[0088] In summary, the visual positioning loss L vg The calculation expression is:

[0089]

[0090] Among them, s i is the predicted reference score of the key point ranked i.

[0091] 5.5 Comprehensive training loss:

[0092] L=α1L vg +α2L ks +α3L det +α4L text

[0093] 6. During the inference phase, the keypoint with the highest reference confidence score is selected to regress the target bounding box and use it as the model output.

[0094] The above method is applied to the following examples to demonstrate the technical effects of the present invention. The specific steps in the examples are not repeated here. The present invention is experimented on the ScanRefer dataset and compared with various advanced previous works.

[0095] The ScanRefer dataset is based on the Scan-Net dataset and annotates 800 scenes with descriptions in 51,583 languages. Each scene has an average of 13.81 objects and 64.48 descriptions. According to the ScanRefer benchmark, the training / validation / test sets are divided into 36,655, 9,508 and 5,410 samples respectively in the experiment. The data can be divided into two categories: "Unique" and "Multiple", depending on whether the target object is the only object of the same type in the scene. In order to comprehensively evaluate the present invention, the following table shows the accuracy and overall accuracy of the present invention on the "Unique" and "Multiple" subsets.

[0096] Table 1 Comparison of the proposed method with other advanced methods on the ScanRefer dataset

[0097]

[0098] In this embodiment, the advanced works compared with the present invention are: 1) ScanRefer: This model adopts a method called language-vision joint representation, which combines natural language descriptions with the geometric shape and semantic features of three-dimensional objects to achieve more accurate object positioning and retrieval; 2) TGNN, which jointly models natural language descriptions and visual scenes to achieve accurate object detection and positioning; 3) 3DVG-Transformer is a Transformer-based 3D visual geometry network model that can extract features from 3D scene data and combine it with natural language queries to achieve accurate positioning of 3D objects; The core of the 3DVG-Transformer model The core is a new Transformer encoder that can process point cloud data in space and establish relationships between point cloud data; 4) MVT is based on Transformer and multi-view representation learning, which can integrate point cloud data from multiple perspectives and natural language queries to achieve accurate 3D object positioning; 5) The 3DJCG model is based on graph convolutional networks and multi-layer perceptrons, which can jointly represent 3D point cloud data and natural language queries to achieve accurate 3D object retrieval and positioning; 6) The 3D-SPS model is based on 3D convolutional neural networks (CNN) and point cloud matching algorithms, which can jointly represent 3D shape and posture information to achieve accurate 3D object retrieval and positioning.

[0099] According to Table 1, the present invention outperforms the best two-stage method by +3.7% at Acc@0.25, outperforms the best single-stage method by +1.95% at Acc@0.5, and outperforms the best single-stage method by +1.95% at Acc@0.5.

[0100] +1.44%. Even compared to 3DJCG, which uses an additional Scan2Cap dataset to assist its training, the present invention still shows advantages in all indicators. Specifically, for the "Multiple" subset, the present invention achieves gains of +2.29% and +1.59% in Acc@0.25 and Acc@0.5 compared to the state-of-the-art single-stage model, which verifies that the proposed 3DRP-MA module is very useful for modeling complex relative position relationships in 3D space and significantly helps distinguish the described target object from multiple interference objects.

[0101] In this embodiment, a 3D visual positioning system with relative position perception capability is also provided, which is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can be a combination of software and / or hardware that implements the predetermined function. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement it in hardware, or a combination of software and hardware.

[0102] This embodiment provides a 3D visual positioning system with relative position perception capability, including:

[0103] A seed point extraction module is used to extract seed points and their features from 3D point clouds;

[0104] A text encoding module, which is used to extract text features of language description sentences;

[0105] A sampling module, which is used to sample key points of the target object related to the language description sentence from the seed point;

[0106] A 3D relative position perception network module, which is used to decode and output the predicted bounding box and reference confidence score of the target object using a 3D relative position perception network including a text-visual dual decoder;

[0107] The localization module is used to select the keypoint with the highest reference confidence score to regress the target bounding box, and the object within the bounding box is taken as the localization result.

[0108] In a specific implementation of the present invention, the 3D relative position perception network module includes:

[0109] The text decoder is obtained by replacing the first original attention module in the Transformer decoder with a 3D relative position-aware multi-head attention module. It is used to generate decoding results based on key point features and text features.

[0110] The visual decoder is obtained by replacing the two original attention modules in the Transformer decoder with a 3D relative position-aware multi-head attention module, which is used to generate decoding results based on key point features and seed point features;

[0111] a first multi-layer perceptron, configured to generate a reference confidence score according to a decoding result of the text decoder;

[0112] The second multi-layer perceptron is used to generate a predicted bounding box according to the decoding result of the visual decoder.

[0113] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0114] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0115] Please note that the various technical features of the above embodiments can be combined in any manner. To simplify the description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there are no contradictions in the combination of these technical features, they should be considered to be within the scope of this specification. It should be pointed out that for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of this application, and these all fall within the scope of protection of this application.

Claims

1. A 3D visual positioning method with relative position perception capability, characterized in that: The following steps are involved: 1) Sampling seed points from the 3D point cloud and extracting seed point features; and extracting text features of the language description sentence; 2) Sampling key points of the target object related to the language description sentence from the seed point; 3) Using a 3D relative position-aware network containing a text-visual dual decoder, the predicted bounding box and reference confidence score of the target object are decoded and output respectively. The keypoint with the highest reference confidence score is selected to regress the target bounding box, and the object within the bounding box is used as the positioning result; The text-visual dual decoder includes a text decoder and a visual decoder, both of which are implemented based on the Transformer decoder structure, and at least one original attention module in the Transformer decoder is replaced by a multi-head attention module with 3D relative position perception; The 3D relative position perception multi-head attention module is referred to as 3DRP-MA, and its calculation process is as follows: 3.1) The key points obtained in step 2) As an input sequence to a single-head relative position-aware attention module, the query is calculated key value for: q i =p i W Q ,k i =p i W K ,v i =p i W V Among them, W Q ,W K ,W V is the projection matrix, q i 、k i 、v i are the query, key and value corresponding to the i-th key point, respectively, p i is the i-th seed point; 3.2) According to the relative position between the input key points and the relative distance mapping function, the relative position encoding is introduced into the attention module, and the attention weight a in the relative position attention is calculated accordingly. i,j : Among them, d ij is the relative distance from point i to point j, a i,j is the attention weight of point i and point j, p(·) is the relative distance mapping function, The relative distance from point i to point j is mapped into a pair of relative position codes after discrete values. The superscript T stands for transposition, and q i For the i-th query, k j is the i-th key, d represents k j Dimensions; 3.3) Calculate the output of the attention module for single-head relative position perception based on the attention weight Among them, M0 is the number of key points, v i is the i-th value, z i is the i-th output; 3.4) Capture the relative spatial relationship between points from multiple directions, encode the x, y, z coordinates and the relative distance after discretization under the Euclidean metric, and express them as D x 、D y 、D z and D e ; D x 、D y 、D z and D e As the relative distance mapping function p(·), four single-head relative position-aware attention modules are established and combined to obtain the output of the 3D relative position-aware multi-head attention module.

2. A 3D visual positioning method with relative position perception capability according to claim 1, characterized in that: The text decoder includes a first 3DRP-MA module, an original multi-head attention module, and a first feedforward network module; First, the key point features are used as the input sequence, and after the projection matrix transformation, the query Q, key K and value V are generated, which are input into the first 3DRP-MA module, and the residual connection and normalization calculation are performed through the Add&Norm layer to obtain Z0; based on Z0 and the text features obtained in step 1), the query Q, key K and value V for input into the multi-head attention module are generated, and the output of the multi-head attention module is subjected to the Add&Norm layer for residual connection and normalization calculation to obtain Z′; Z′ is used as the input of the first feedforward network module, and the residual connection and normalization calculation are performed through the Add&Norm layer; that is, all the calculations of the text decoder are completed.

3. The 3D visual positioning method with relative position perception capability according to claim 1, characterized in that: The visual decoder includes a second 3DRP-MA module, a third 3DRP-MA module, and a second feedforward network module; First, the key point features are used as the input sequence, and the query Q, key K and value V are generated after the projection matrix transformation, and are input into the second 3DRP-MA module. The residual connection and normalization calculation are performed through the Add&Norm layer to obtain Z0′; based on Z0′ and the seed point features obtained in step 1), the query Q, key K and value V for input into the third 3DRP-MA module are generated, and the output of the third 3DRP-MA module is subjected to the Add&Norm layer for residual connection and normalization calculation to obtain Z″; Z″ is used as the input of the second feedforward network module, and the residual connection and normalization calculation are performed through the Add&Norm layer; that is, all the calculations of the visual decoder are completed.

4. The 3D visual positioning method with relative position perception capability according to claim 1, characterized in that: The outputs of the text decoder and visual decoder are passed through a multi-layer perceptron to generate reference confidence scores and predicted bounding boxes, respectively.

5. The 3D visual positioning method with relative position perception capability according to claim 1, characterized in that: The relative distance mapping function in step 3.1) is as follows: Where [·] is the rounding function; sign(·) is the sign function, α and β are thresholds, p(d) is the relative distance after mapping, k is a natural number, and d is the distance.

6. The 3D visual positioning method with relative position perception capability according to claim 1, characterized in that: The soft label strategy is used to train the 3D relative position perception network. The soft label strategy is as follows: according to the IoU value ranking of the predicted bounding box, the key points are reordered and the top N key points are ranked. s The target label of the key points with a value greater than 0.25 is set to The target labels of other key points are set to 0, where N s is the threshold, The calculation formula is: in, is the target label of the key point ranked i, and σ is a parameter that controls the smoothness of the result distribution.

7. A 3D visual positioning system with relative position perception capability, used to implement the 3D visual positioning method with relative position perception capability according to claim 1, characterized in that the system include: A seed point extraction module is used to extract seed points and their features from 3D point clouds; A text encoding module, which is used to extract text features of language description sentences; A sampling module, which is used to sample key points of the target object related to the language description sentence from the seed point; A 3D relative position perception network module, which is used to decode and output the predicted bounding box and reference confidence score of the target object using a 3D relative position perception network including a text-visual dual decoder; The localization module is used to select the keypoint with the highest reference confidence score to regress the target bounding box, and the object within the bounding box is taken as the localization result.

8. The 3D visual positioning system with relative position perception capability according to claim 7, characterized in that: The 3D relative position perception network module includes: The text decoder is obtained by replacing the first original attention module in the Transformer decoder with a 3D relative position-aware multi-head attention module. It is used to generate decoding results based on key point features and text features. The visual decoder is obtained by replacing the two original attention modules in the Transformer decoder with a 3D relative position-aware multi-head attention module, which is used to generate decoding results based on key point features and seed point features; a first multi-layer perceptron, configured to generate a reference confidence score according to a decoding result of the text decoder; The second multi-layer perceptron is used to generate a predicted bounding box according to the decoding result of the visual decoder.

Images