A 3D visual positioning method and system based on adaptive view fusion and 2D auxiliary language alignment

Abstract

This invention discloses a 3D visual localization method and system based on adaptive view fusion and 2D-assisted language alignment, comprising: extracting target bounding box information, spatial location information, and 3D point cloud features from 3D point cloud data; rendering the 3D point cloud data into image data and obtaining image features from it; extracting language features from text data; processing the target bounding box information using a bounding box encoder and processing the spatial location information using a spatial encoder; performing multimodal fusion of image features, target bounding box features, and language features through a 2D-assisted language alignment module; obtaining multimodal fused features by fusing the initially fused features with the 3D point cloud features; performing view fusion of the multimodal fused features and spatial features through an adaptive view fusion module; and obtaining a 3D localization result based on the obtained target fused features. This invention can achieve better localization and effectively alleviate the localization deviation problems caused by multi-view feature bias and insufficient cross-modal alignment.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, specifically to a 3D visual positioning method and system based on adaptive view fusion and 2D auxiliary language alignment. Background Technology

[0002] 3D vision positioning is a core technology in embedded intelligence, robot interaction, and other fields. Its core objective is to accurately locate target objects in 3D scenes based on natural language descriptions, achieving cross-modal semantic alignment between language and 3D vision. This provides crucial support for applications such as robot navigation, intelligent scene interaction, and industrial inspection. With the rapid development of related applications, higher demands are being placed on the positioning accuracy, robustness, and scene adaptability of 3D vision positioning methods.

[0003] Currently, most 3D visual positioning methods use a single view or fixed weights to fuse features from multiple views, resulting in inconsistent features across multiple views and an inability to dynamically adapt to the differences in perspectives of different 3D scenes. At the same time, 3D point cloud features are mainly geometric information, while natural language descriptions are mainly semantic information. There are inherent differences between the two feature spaces, and existing alignment methods are difficult to achieve fine-grained semantic matching, resulting in low cross-modal alignment accuracy and inaccurate target localization. Summary of the Invention

[0004] This invention provides a 3D visual positioning method and system based on adaptive view fusion and 2D auxiliary language alignment to achieve better positioning performance and effectively alleviate the positioning deviation problem caused by multi-view feature bias and insufficient cross-modal alignment.

[0005] In a first aspect, the present invention provides a 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment, the method comprising:

[0006] Extract target bounding box information, spatial location information, and 3D point cloud features from 3D point cloud data, render the 3D point cloud data into image data, and extract image features from the image data; extract language features from user-provided text data, wherein the text data corresponds to the 3D point cloud data.

[0007] The target bounding box information is processed using a bounding box encoder to obtain target bounding box features, and the spatial location information is processed using a spatial encoder to obtain spatial features.

[0008] The image features, the target bounding box features, and the language features are fused using a 2D auxiliary language alignment module to obtain initial fused features; and the initial fused features are then fused with the 3D point cloud features to obtain multimodal fused features.

[0009] The adaptive view fusion module performs view fusion of the multimodal fusion features and the spatial features to obtain the target's fusion features; and based on the target's fusion features, a positioning score is performed to obtain a 3D positioning result.

[0010] In some embodiments of the present invention, the 2D auxiliary language alignment module includes a language-visual cross-module, an object-language enhanced cross-module, and an object-language cross-module;

[0011] The step of using a 2D-assisted language alignment module to perform multimodal fusion of the image features, the target bounding box features, and the language features to obtain initial fused features includes:

[0012] Using the language features as the first query, the image features as the first key and the first value, and the language-visual cross-module to perform cross-attention calculation on the first query, the first key and the first value, language-enhanced visual features are obtained.

[0013] The target bounding box features are used as the second query, and the language-enhanced visual features are used as the second key and the second value. The object-language enhancement cross-module is used to perform cross-attention calculation on the second query, the second key, and the second value to obtain the updated target bounding box features.

[0014] Using the language features as the third query, the updated target box features as the third key and the third value, the object-language cross-interaction module is used to perform cross-attention calculation on the third query, the third key, and the third value to obtain the initial fused features.

[0015] In some embodiments of the present invention, the step of using the object-language enhanced cross-payment module to perform cross-attention calculation on the second query, the second key, and the second value to obtain updated target box features includes:

[0016] The object-language enhanced cross-payment module is used to perform cross-attention calculation on the second query, the second key, and the second value to obtain the initial updated target box features;

[0017] The initially updated target bounding box features are adaptively fused with the target bounding box features to obtain the updated target bounding box features.

[0018] In some embodiments of the present invention, fusing the initial fused features with the 3D point cloud features to obtain multimodal fused features includes:

[0019] The initial fused features are used as the fourth key, the 3D point cloud features as the fourth value, and the language features as the fourth query. The attention mechanism in the Transformer module is used to perform cross-attention calculation on the fourth key, the fourth value, and the fourth query to obtain the multimodal fused features.

[0020] In some embodiments of the present invention, the adaptive view fusion module includes a feature enhancement module, a feature rearrangement module, a weight estimation network, and a weighted aggregation layer;

[0021] The step of using an adaptive view fusion module to perform view fusion of the multimodal fusion features and the spatial features to obtain the fusion features of the target includes:

[0022] The feature enhancement module enhances the multimodal fusion features to obtain enhanced features.

[0023] The enhanced features are concatenated with the spatial features to obtain concatenated features, which are then input into the feature rearrangement module to obtain rearranged features.

[0024] The relative importance of each view is evaluated using the weight estimation network to obtain attention weights;

[0025] Based on the attention weights, and by using the weighted aggregation layer to sum all the views in a weighted manner, the fusion features of the target are obtained.

[0026] In some embodiments of the present invention, the weight estimation network includes a first fully connected layer. and the second fully connected layer ;

[0027] The process of using the weight estimation network to evaluate the relative importance of each view and obtain attention weights includes:

[0028] View The data is sequentially input into the first fully connected layer. and the second fully connected layer Obtain relative importance values :

[0029] ;

[0030] In the view dimension, the relative importance value Perform a Softmax operation to obtain the normalized attention weights.

[0031] In some embodiments of the present invention, extracting 3D point cloud features from 3D point cloud data includes:

[0032] The 3D point cloud features are extracted from the 3D point cloud data using the PointNet++ network.

[0033] In some embodiments of the present invention, the step of extracting image features from the image data includes:

[0034] The image features are extracted from the image data using the ViT model.

[0035] In some embodiments of the present invention, the step of extracting language features from user-provided text data includes:

[0036] The BERT model is used to extract the language features from the text data.

[0037] Secondly, the present invention also provides a 3D visual positioning system based on adaptive view fusion and 2D auxiliary language alignment, the system comprising:

[0038] The first feature extraction module is used to extract target bounding box information, spatial location information, and 3D point cloud features from 3D point cloud data, render the 3D point cloud data into image data, extract image features from the image data, and extract language features from user-provided text data, wherein the text data corresponds to the 3D point cloud data.

[0039] The second feature extraction module is used to process the target box information using a box encoder to obtain target box features, and to process the spatial location information using a spatial encoder to obtain spatial features.

[0040] The multimodal fusion module is used to perform multimodal fusion on the image features, the target box features, and the language features through the 2D auxiliary language alignment module to obtain initial fused features; and to fuse the initial fused features with the 3D point cloud features to obtain multimodal fused features.

[0041] The view fusion and localization module is used to perform view fusion of the multimodal fusion features and the spatial features through the adaptive view fusion module to obtain the fusion features of the target; and to perform localization scoring based on the fusion features of the target to obtain 3D localization results.

[0042] In the 3D visual localization method and system based on adaptive view fusion and 2D auxiliary language alignment provided by this invention, the 2D auxiliary language alignment module performs multimodal fusion of image features, target bounding box features, and language features to obtain initial fused features. These initial fused features are then fused with 3D point cloud features to obtain multimodal fused features, thereby enabling the transfer of 2D language reasoning capabilities to 3D tasks and improving the cross-modal alignment accuracy between 3D features and natural language. The adaptive view fusion module performs view fusion of the multimodal fused features and spatial features to obtain the fused features of the target. The adaptive view fusion module dynamically evaluates the reliability of each view feature using attention entropy and visibility weighting, adaptively assigns weights, and fuses complementary information from different views to generate a unified and robust 3D feature representation, solving the problem of inconsistent multi-view features. The method provided by this invention enables multi-module collaborative work, improves the method's scene adaptability, and can be widely applied to various scenarios such as robot navigation, industrial inspection, and intelligent scene interaction, meeting diverse localization needs. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is one of the flowcharts of the 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment provided in the embodiments of the present invention;

[0045] Figure 2 This is the second flowchart of the 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment provided in the embodiments of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0048] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0049] The use of "applies to" or "configured to" in this invention implies an open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more conditions or values ​​may in practice be based on additional conditions or values ​​beyond those conditions.

[0050] In this invention, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0051] Terminology Explanation:

[0052] 3D Visual Grounding (3DVG) is a core task in the fields of computer vision and multimodal interaction. It aims to accurately locate target objects in 3D scenes based on natural language descriptions, achieve cross-modal semantic alignment between language and 3D vision, and provide core technical support for applications such as robot navigation and intelligent interaction.

[0053] Adaptive-View Fusion (AVF) refers to a technique that dynamically evaluates the reliability of multi-view features by adaptively fusing complementary information from different perspectives through attention entropy quantization and visibility weighting to generate a unified and robust 3D feature representation.

[0054] Hierarchical 2D-Assisted 3D Alignment (H2A3) refers to a technique that leverages the powerful semantic understanding capabilities of pre-trained 2D visual language models (VLMs) to map 3D point cloud features to a 2D feature space through panoramic rendering and perspective adaptation. This enables the transfer of 2D language reasoning capabilities to 3D tasks and improves cross-modal alignment accuracy.

[0055] Visual-Language Models (VLMs): These are pre-trained models (such as the ViT model) that can simultaneously process visual information (such as images and views) and semantic information (such as natural language descriptions) to achieve visual-semantic cross-modal alignment and reasoning.

[0056] Attention Entropy: A metric used to quantify the redundancy of feature information in multiple views. The lower the entropy value, the richer and more reliable the effective information of the view features, providing a basis for view weight allocation.

[0057] Multi-View Feature Fusion: This refers to the joint modeling of 3D scene features (geometric features and visual features) collected from different perspectives and sensors, fusing complementary information to solve the problem of insufficient features from a single view and improve the integrity and robustness of feature representation.

[0058] Existing 3D visual localization methods suffer from two major shortcomings: First, the multi-view feature fusion method is unreasonable. Existing methods mostly use a single view or fixed weights to fuse multi-view features, which cannot dynamically adapt to differences in perspective and object occlusion in different 3D scenes. This leads to deviations in the features extracted from different views, thus affecting the accuracy of target localization. Second, the cross-modal alignment accuracy is low. 3D point cloud features are mainly geometric information, while natural language descriptions are mainly semantic information. There are inherent differences between the feature spaces of the two. Existing alignment methods do not fully utilize the capabilities of mature 2D visual language models, making it difficult to achieve fine-grained semantic matching between 3D features and natural language, which easily leads to localization deviations or mislocalization.

[0059] Against this backdrop, 3D visual positioning technology based on multi-view fusion and cross-modal alignment has emerged, aiming to solve the positioning pain points of existing methods by optimizing feature fusion methods and improving cross-modal alignment accuracy, and to provide more accurate and robust 3D positioning capabilities for related applications.

[0060] In some related technologies, multi-view feature data is acquired using 3D LiDAR and cameras. These multi-view features are then fused using fixed weights to generate a unified 3D feature. Simple semantic mapping aligns the 3D features with natural language, thereby controlling the execution module to complete target localization. In other related technologies, a 3D vision acquisition module acquires point cloud bird's-eye view (BEV), perspective view (RV), and 2D rendered view. The multi-view features are then transmitted to a fixed-weight fusion module. This module performs a weighted summation of the multi-view features according to preset weight ratios (e.g., BEV view weight 0.4, RV view weight 0.3, 2D rendered view weight 0.3), outputting a unified 3D feature. A language alignment module performs simple parsing of the natural language description, extracts keyword semantic features, and performs a rough match with the unified 3D feature. Upon successful matching, a positioning control command is generated to control the execution module to complete target localization. This solution can achieve basic positioning functionality in simple 3D scenes, but in complex scenes (such as object occlusion or changing viewpoints), positioning accuracy and robustness significantly decrease.

[0061] This reveals that existing 3D localization schemes employ fixed-weight fusion of multi-view features, failing to dynamically assess the reliability of each view's features. When a view is occluded or subject to noise interference, it still participates in the fusion process with fixed weights, leading to redundant information interference or missing effective information, resulting in inconsistencies in multi-view features and impacting localization accuracy. The language alignment module in existing schemes only performs simple keyword semantic matching, failing to fully utilize the semantic understanding capabilities of mature 2D visual language models (VLMs) and failing to achieve precise adaptation between 3D features and the 2D VLMs feature space. This fails to bridge the inherent gap between 3D geometric features and natural language semantic features, making fine-grained semantic alignment difficult and prone to mislocalization. Existing schemes lack effective feature filtering mechanisms, failing to effectively filter noisy data (such as sensor errors and environmental interference) in multi-view features, resulting in insufficient reliability of the fused 3D features and a significant drop in localization performance in complex 3D scenes (such as densely packed objects or severe occlusion). Furthermore, the fixed fusion strategies and alignment methods of existing schemes cannot be dynamically adjusted according to the complexity of the 3D scene and the distribution of objects, limiting adaptability and making it difficult to meet the localization needs of diverse scenarios such as robot navigation and industrial inspection.

[0062] To address the aforementioned issues, this invention provides a 3D visual localization method and system based on adaptive view fusion and 2D auxiliary language alignment. It utilizes the attention entropy and visibility weighting in the Adaptive View Fusion (AVF) module to dynamically evaluate the reliability of features from each view, adaptively assigning weights and fusing complementary information from different views to generate a unified and robust 3D feature representation, thus resolving the inconsistency problem of multi-view features. Through the powerful semantic understanding capabilities of pre-trained 2D VLMs in the 2D Auxiliary Language Alignment (H2A3) module, as well as panoramic rendering and perspective adaptation, 3D point cloud features are mapped to the 2D feature space, enabling the transfer of 2D language reasoning capabilities to 3D tasks and improving the cross-modal alignment accuracy between 3D features and natural language. By filtering noisy data from multi-view features, the reliability of the fused 3D features is improved, enhancing the method's localization robustness in complex scenes. The method provided by this invention enables multi-module collaborative work, improving the method's scene adaptability and making it widely applicable to various scenarios such as robot navigation, industrial inspection, and intelligent scene interaction, meeting diverse localization needs.

[0063] The following description, in conjunction with the accompanying drawings, introduces the 3D visual positioning method and system based on adaptive view fusion and 2D auxiliary language alignment provided by embodiments of the present invention.

[0064] like Figure 1 As shown, this embodiment of the invention provides a 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment, which includes the following steps:

[0065] S101 extracts target bounding box information (BoxInfo), spatial location information, and 3D point cloud features from 3D point cloud data, renders the 3D point cloud data into image data, extracts image features from the image data, and extracts language features from user-provided text data.

[0066] Among them, the text data corresponds to the 3D point cloud data, that is, the text data is a textual description of the target corresponding to the 3D point cloud data.

[0067] In some examples, point-based deep learning algorithms are used to extract features from 3D point cloud data, learning semantic or geometric features to obtain 3D point cloud features. Illustratively, a point-based deep learning algorithm could be the PointNet++ network.

[0068] By performing geometric, detection, or coordinate calculations on 3D point cloud data, target bounding box information (BoxInfo) and spatial location information are obtained. The target bounding box information (BoxInfo) includes 3D bounding box parameters and coordinates, while the spatial location information includes spatial relationships.

[0069] The rendered image data is input into the ViT (Visual Transformer) model to extract image features.

[0070] The text data input by the user is fed into the BERT model to extract language features.

[0071] S102, the target bounding box information is processed using a box encoder to obtain target bounding box features, and the spatial location information is processed using a spatial encoder to obtain spatial features.

[0072] S103 uses the 2D auxiliary language alignment module (H2A3 module) to perform multimodal fusion of image features, target box features and language features to obtain initial fused features; and then fuses the initial fused features with 3D point cloud features to obtain multimodal fused features.

[0073] In other words, image features, bounding box features, and language features are input into the H2A3 module. Through the internal semantic alignment mechanism of the H2A3 module, the inherent gap between 3D geometric features and language semantics is bridged, and the initial fusion of multimodal features is achieved.

[0074] S104, through the adaptive view fusion module, performs view fusion of multimodal fusion features and spatial features to obtain the fusion features of the target; and performs localization scoring based on the fusion features of the target to obtain 3D localization results.

[0075] The 3D visual localization method based on adaptive view fusion and 2D auxiliary language alignment provided in this invention embodiment performs multimodal fusion of image features, target bounding box features, and language features through a 2D auxiliary language alignment module to obtain initial fused features. These initial fused features are then fused with 3D point cloud features to obtain multimodal fused features, thereby enabling the transfer of 2D language reasoning capabilities to 3D tasks and improving the cross-modal alignment accuracy between 3D features and natural language. The adaptive view fusion module performs view fusion of the multimodal fused features and spatial features to obtain the fused features of the target. The adaptive view fusion module dynamically evaluates the reliability of each view feature using attention entropy and visibility weighting, adaptively assigns weights, and fuses complementary information from different views to generate a unified and robust 3D feature representation, solving the problem of inconsistency among multi-view features. The method provided in this invention embodiment enables multi-module collaborative work, improves the method's scene adaptability, and can be widely applied to various scenarios such as robot navigation, industrial inspection, and intelligent scene interaction, meeting diverse localization needs.

[0076] In some embodiments of the present invention, the 2D auxiliary language alignment module includes a Language-Visual CA module, an Object-Language-Enhance CA module, and an Object-Language CA module.

[0077] The 2D-assisted language alignment module performs multimodal fusion of image features, bounding box features, and language features to obtain initial fused features, including:

[0078] Using linguistic features as the first query and image features as the first key and first value, a language-visual cross-module is used to perform cross-attention calculation on the first query, first key, and first value to obtain language-enhanced visual features. The language-visual cross-module highlights the visual blocks that are semantically most consistent with the linguistic description, thus generating language-enhanced visual features.

[0079] In some examples, language-enhanced visual features Obtained through the following formula:

[0080] ;

[0081]

[0082] In the formula, As a linguistic feature, For the first query, As the first key, As the first value, For image features, It is the dimension of the first key (i.e., the key vector).

[0083] The target bounding box features are used as the second query, and the language-enhanced visual features are used as the second key and the second value. The object-language enhancement cross module is used to perform cross attention calculation on the second query, the second key, and the second value to obtain the updated target bounding box features.

[0084] In some examples, the object-language enhanced cross-module is used to perform cross-attention computation on the second query, the second key, and the second value to obtain updated target box features, including:

[0085] The object-language enhanced cross-module is used to perform cross-attention calculation on the second query, the second key, and the second value to obtain the initial updated target box features.

[0086] Schematic representation of the initially updated target bounding box features It can be obtained through the following formula:

[0087] ;

[0088] ;

[0089] In the formula, This is the second query. It's the second key. It is the second value. It is the target bounding box feature.

[0090] The initially updated target bounding box features are adaptively fused with the target bounding box features to obtain the updated target bounding box features.

[0091] Indicatively, the updated target bounding box features It can be obtained through the following formula:

[0092] ;

[0093] In the formula, Learnable coefficients It is initialized to 0.5.

[0094] Through the aforementioned object-language enhancement cross-module and adaptive fusion, object features are selectively integrated with the most relevant visual cues to the text, enhancing the semantic alignment between object geometry and the linguistic context.

[0095] Using language features as the third query and the updated target box features as the third key and third value, the object-language cross-module is used to perform cross-attention calculation on the third query, third key, and third value to obtain the initial fused features.

[0096] Schematic representation of features after initial fusion It can be obtained through the following formula:

[0097] ;

[0098] ;

[0099] In the formula, For the third query, It's the third key. It is the third value.

[0100] The object-language cross-processing module handles linguistic features and updated bounding box features, ensuring global semantic alignment between the linguistic description and the 3D spatial structure. If the length of the fused F sequence does not match the number of target points N, linear interpolation is used to maintain size consistency.

[0101] In some embodiments of the present invention, the initial fused features are fused with 3D point cloud features to obtain multimodal fused features, including:

[0102] The initial fused features are used as the fourth key, the 3D point cloud features as the fourth value, and the language features as the fourth query. The attention mechanism in the Transformer module is used to perform cross-attention calculation on the fourth key, the fourth value, and the fourth query to obtain multimodal fused features.

[0103] In some embodiments of the present invention, the adaptive view fusion module includes a feature enhancement module, a feature rearrangement module, a weight estimation network, and a weighted aggregation layer.

[0104] The adaptive view fusion module performs view fusion on multimodal fusion features and spatial features to obtain the fused features of the target, including:

[0105] The feature enhancement module enhances the multimodal fusion features to obtain enhanced features.

[0106] In some examples, the feature enhancement module includes cross-attention layers, Dropout, layer normalization, and feedforward layers. The multimodal fusion features are sequentially processed using cross-attention, Dropout, layer normalization, and feedforward layers to obtain the enhanced features.

[0107] The enhanced features are concatenated with the spatial features to obtain the concatenated features x, which are then input into the feature rearrangement module to obtain the rearranged features.

[0108] In some examples, within the feature rearrangement module, given a set of target features extracted from different views, the input tensor can be represented as: x∈ Where x represents the concatenated features input to the feature rearrangement module, B is the batch size, V is the number of views, N is the number of targets in each scene, and D is the feature dimension; each feature vector x b,v,n Used to encode the appearance, spatial and contextual cues of the nth target from the vth viewpoint.

[0109] To facilitate view-level computation, the tensor x is first reshaped into x′∈ It aligns targets in all views into a unified mini-batch structure, enabling efficient parallel processing and learning of cross-view relationships.

[0110] A weight estimation network is used to evaluate the relative importance of each view, thus obtaining attention weights. It is understandable that the purpose of feature reordering is to explicitly preserve the structural correspondences between different views. In the reordered tensor x′, each view feature still maintains an independent and separable representation. Based on this, the view weight estimation network can directly model each view feature, thereby learning its relative importance in cross-view fusion without compromising semantic consistency between views.

[0111] In some examples, the weight estimation network includes a first fully connected layer. and the second fully connected layer .

[0112] The relative importance of each view is evaluated using a weight estimation network to obtain attention weights, including:

[0113] View Input sequentially to the first fully connected layer and the second fully connected layer Obtain relative importance values :

[0114] ;

[0115] In the view dimension, relative importance values Perform a Softmax operation to obtain a normalized attention weight αᵥ, which reflects the relative contribution of each view, highlighting effective views while suppressing redundant or noisy views.

[0116] Based on attention weights, and by using a weighted aggregation layer to sum all views in a weighted manner, the fusion features of the target are obtained.

[0117] Schematic representation of the target's fusion features It can be obtained through the following formula:

[0118] ;

[0119] In the formula, V is the number of views.

[0120] In some embodiments of the present invention, after processing spatial location information using a spatial encoder to obtain spatial features, the spatial features are sequentially input into a cross-attention layer, Dropout, layer normalization layer, and feed-forward layer. The spatial features are further optimized by the cross-attention layer, Dropout, layer normalization layer, and feed-forward layer, and the optimized spatial features are then input into the AVF module.

[0121] In some embodiments of the present invention, there are a total of T AVF modules, which are connected in sequence. The output of AVF-1 is the input of AVF-2, and so on, with the output of AVK-T-1 being the input of AVK-T.

[0122] The adaptive view fusion and 2D auxiliary language alignment 3D visual localization method provided in this invention offers a multi-view feature adaptive fusion strategy based on attention entropy and visibility weighting. This strategy dynamically evaluates the reliability of each view feature, rationally allocates weights, and fuses complementary information from different views. It avoids missing features or interference from redundant information in a single view, generating a unified and robust 3D feature representation. This mechanism is the core of solving the problem of inconsistent multi-view features. The process of using a weighted aggregation layer to perform a weighted summation of all views retains the complementary information of effective views and suppresses redundant or noisy cues. This results in fused features that possess both rich semantic information and geometric consistency, significantly improving the performance of downstream inference and localization tasks.

[0123] Finally, the 3D visual localization method based on adaptive view fusion and 2D auxiliary language alignment provided in this embodiment of the invention has improved the localization accuracy on Sr3D and Nr3D datasets compared with the prior art, and reduced the mislocalization rate in complex occlusion scenes. The overall performance is better than the prior art.

[0124] On the other hand, embodiments of the present invention also provide a 3D visual positioning system based on adaptive view fusion and 2D auxiliary language alignment, the system comprising:

[0125] The first feature extraction module is used to extract target bounding box information, spatial location information and 3D point cloud features from 3D point cloud data, render the 3D point cloud data into image data, extract image features from the image data, and extract language features from user-provided text data, with the text data corresponding to the 3D point cloud data.

[0126] The second feature extraction module is used to process the target box information using a box encoder to obtain target box features, and to process the spatial location information using a spatial encoder to obtain spatial features.

[0127] The multimodal fusion module is used to perform multimodal fusion of image features, bounding box features, and language features through the 2D auxiliary language alignment module to obtain initial fused features; and then fuses the initial fused features with 3D point cloud features to obtain multimodal fused features.

[0128] The view fusion and localization module is used to perform view fusion of multimodal fusion features and spatial features through the adaptive view fusion module to obtain the fusion features of the target; and to perform localization scoring based on the fusion features of the target to obtain 3D localization results.

[0129] The 3D visual positioning system based on adaptive view fusion and 2D auxiliary language alignment provided in this embodiment of the invention corresponds to the 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment provided in any of the above embodiments, and will not be described in detail here.

[0130] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0132] The foregoing has provided a detailed description of a 3D visual positioning method and system based on adaptive view fusion and 2D auxiliary language alignment provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment, characterized in that, The method includes: Extract target bounding box information, spatial location information, and 3D point cloud features from 3D point cloud data, render the 3D point cloud data into image data, and extract image features from the image data; extract language features from user-provided text data, wherein the text data corresponds to the 3D point cloud data. The target bounding box information is processed using a bounding box encoder to obtain target bounding box features, and the spatial location information is processed using a spatial encoder to obtain spatial features. The image features, the target bounding box features, and the language features are fused using a 2D auxiliary language alignment module to obtain initial fused features; and the initial fused features are then fused with the 3D point cloud features to obtain multimodal fused features. The adaptive view fusion module performs view fusion of the multimodal fusion features and the spatial features to obtain the target's fusion features; and based on the target's fusion features, a positioning score is performed to obtain a 3D positioning result.

2. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to claim 1, characterized in that, The 2D auxiliary language alignment module includes a language-visual cross module, an object-language enhanced cross module, and an object-language cross module; The step of using a 2D-assisted language alignment module to perform multimodal fusion of the image features, the target bounding box features, and the language features to obtain initial fused features includes: Using the language features as the first query, the image features as the first key and the first value, and the language-visual cross-module to perform cross-attention calculation on the first query, the first key and the first value, language-enhanced visual features are obtained. The target bounding box features are used as the second query, and the language-enhanced visual features are used as the second key and the second value. The object-language enhancement cross-module is used to perform cross-attention calculation on the second query, the second key, and the second value to obtain the updated target bounding box features. Using the language features as the third query, the updated target box features as the third key and the third value, the object-language cross-interaction module is used to perform cross-attention calculation on the third query, the third key, and the third value to obtain the initial fused features.

3. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to claim 2, characterized in that, The step of using the object-language enhanced cross-payment module to perform cross-attention calculation on the second query, the second key, and the second value to obtain updated target box features includes: The object-language enhanced cross-payment module is used to perform cross-attention calculation on the second query, the second key, and the second value to obtain the initial updated target box features; The initially updated target bounding box features are adaptively fused with the target bounding box features to obtain the updated target bounding box features.

4. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to claim 1, characterized in that, The process of fusing the initial fused features with the 3D point cloud features to obtain multimodal fused features includes: The initial fused features are used as the fourth key, the 3D point cloud features as the fourth value, and the language features as the fourth query. The attention mechanism in the Transformer module is used to perform cross-attention calculation on the fourth key, the fourth value, and the fourth query to obtain the multimodal fused features.

5. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to claim 1, characterized in that, The adaptive view fusion module includes a feature enhancement module, a feature rearrangement module, a weight estimation network, and a weighted aggregation layer; The step of using an adaptive view fusion module to perform view fusion of the multimodal fusion features and the spatial features to obtain the fusion features of the target includes: The feature enhancement module enhances the multimodal fusion features to obtain enhanced features. The enhanced features are concatenated with the spatial features to obtain concatenated features, which are then input into the feature rearrangement module to obtain rearranged features. The relative importance of each view is evaluated using the weight estimation network to obtain attention weights; Based on the attention weights, and by using the weighted aggregation layer to sum all the views in a weighted manner, the fusion features of the target are obtained.

6. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to claim 5, characterized in that, The weight estimation network includes a first fully connected layer. and the second fully connected layer ; The process of using the weight estimation network to evaluate the relative importance of each view and obtain attention weights includes: View The data is sequentially input into the first fully connected layer. and the second fully connected layer Obtain relative importance values : ； In the view dimension, the relative importance value Perform a Softmax operation to obtain the normalized attention weights.

7. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to any one of claims 1 to 6, characterized in that, Extracting 3D point cloud features from 3D point cloud data, including: The 3D point cloud features are extracted from the 3D point cloud data using the PointNet++ network.

8. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to any one of claims 1 to 6, characterized in that, The step of extracting image features from the image data includes: The image features are extracted from the image data using the ViT model.

9. The 3D visual positioning method based on adaptive view fusion and 2D auxiliary language alignment according to any one of claims 1 to 6, characterized in that, The extraction of language features from user-provided text data includes: The BERT model is used to extract the language features from the text data.

10. A 3D visual positioning system based on adaptive view fusion and 2D auxiliary language alignment, characterized in that, The system includes: The first feature extraction module is used to extract target bounding box information, spatial location information, and 3D point cloud features from 3D point cloud data, render the 3D point cloud data into image data, extract image features from the image data, and extract language features from user-provided text data, wherein the text data corresponds to the 3D point cloud data. The second feature extraction module is used to process the target box information using a box encoder to obtain target box features, and to process the spatial location information using a spatial encoder to obtain spatial features. The multimodal fusion module is used to perform multimodal fusion on the image features, the target box features, and the language features through the 2D auxiliary language alignment module to obtain initial fused features; and to fuse the initial fused features with the 3D point cloud features to obtain multimodal fused features. The view fusion and localization module is used to perform view fusion of the multimodal fusion features and the spatial features through the adaptive view fusion module to obtain the fusion features of the target; and to perform localization scoring based on the fusion features of the target to obtain 3D localization results.

Images