Image retrieval method based on geometric perception attention

Abstract

The present application relates to the technical field of computer digital data processing, in particular to an image retrieval method based on geometric perception attention, comprising: inputting a two-dimensional image to be retrieved and target three-dimensional point cloud data into a bidirectional healing feature extraction channel; extracting a two-dimensional image scene level geometric priori using a visual geometry model, injecting an instance mask feature, and mapping into a pseudo three-dimensional manifold feature compatible with the underlying structure of the point cloud; inputting the pseudo three-dimensional manifold and the point cloud sparse feature into a cross-domain comparison model, calculating a geometric guide attention loss, focusing on the geometric salient region of the physical scene, and generating a cross-modal filtering mask; using the mask to drive an external parameter perception cross-attention mechanism, explicitly injecting camera external parameter hypothesis parameters, so that two-dimensional and three-dimensional data feature blocks interact with each other in their respective original domains; finally, calculating the similarity based on the interaction result and outputting the result in order. The present application avoids format conversion information loss and improves cross-modal retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer digital data processing technology, specifically to an image retrieval method based on geometric perception attention. Background Technology

[0002] Cross-modal image retrieval, especially cross-modal retrieval from 2D images to 3D point clouds, has crucial application value in fields such as high-precision assisted positioning for autonomous vehicles, autonomous navigation for robots, and augmented reality. It is one of the core technologies for realizing spatial environment perception in complex scenarios.

[0003] Existing cross-modal retrieval methods from 2D images to 3D point clouds rely on projecting the 3D point cloud into a 2D view, or upscaling or mapping the 2D image to a shared embedding space for feature extraction, followed by feature alignment using conventional attention mechanisms. However, these traditional methods often forcibly unify the data format, disrupting the original pixel topology of the 2D image and the underlying native structure of the sparse point distribution in the 3D point cloud. Furthermore, existing multimodal feature interaction mechanisms suffer from weak geometric constraints, making it difficult to explicitly utilize spatial extrinsic parameters such as camera relative pose and their errors, and lacking effective fusion of scene-level geometric priors such as physical scene depth. In addition, when dealing with complex scenes, there is the problem of background interference in fine-tuning; existing models are easily affected by semantic interference from backgrounds lacking geometric discriminativeness, such as the sky and dynamic clouds, making it difficult to effectively focus attention.

[0004] In summary, existing technologies face the following pressing technical challenges: First, the significant gap between two-dimensional and three-dimensional modalities means that traditional format conversions can easily lead to the loss of underlying structural information, making it impossible to achieve accurate feature interaction within the native domain. Second, existing feature extraction models lack explicit constraints on physical geometric priors and spatial extrinsic parameters, resulting in low cross-domain feature alignment accuracy. Finally, a large amount of background semantics lacking geometric discriminativeness severely interferes with feature extraction from topologically salient regions, leading to insufficient robustness and accuracy in the final cross-modal retrieval.

[0005] To address this, an image retrieval method based on geometric perception attention is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide an image retrieval method based on geometric perception attention, which achieves high-precision cross-modal retrieval from two-dimensional to three-dimensional by integrating scene geometric priors and extrinsic parameter perception interaction.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An image retrieval method based on geometrical awareness attention includes: The two-dimensional image data to be retrieved and the target three-dimensional point cloud data are input into the bidirectional bridged feature extraction channel; the scene-level geometric prior of the two-dimensional image data is extracted using the visual geometric basic model to obtain the global features of the two-dimensional image; instance mask features are injected during the descriptor generation process to map the global features of the two-dimensional image into pseudo-three-dimensional manifold features that are compatible with the underlying structure of the target three-dimensional point cloud data. The pseudo-3D manifold features and the sparse features of the target 3D point cloud data are input into the cross-domain contrast model to calculate the geometry-guided attention loss. Based on the feature fusion process constrained by the geometry-guided attention loss, the attention weights are focused on the geometrically salient regions in the physical scene to generate a cross-modal filtering mask. By utilizing a cross-modal filtering mask-driven extrinsic perception cross-attention mechanism, the camera extrinsic hypothesis parameters are explicitly injected, enabling feature blocks of 2D image data and feature groups of target 3D point cloud data to interact within their respective native domains. Based on the output of feature interaction, the similarity between two-dimensional image data and target three-dimensional point cloud data is calculated, and the final retrieval results are output based on the similarity ranking.

[0008] Preferably, the step of inputting the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridged feature extraction channel includes: acquiring the two-dimensional image data to be retrieved and the target three-dimensional point cloud data respectively; performing a preliminary visual feature extraction step on the two-dimensional image data to obtain initial features of the two-dimensional image; and performing independent point cloud feature extraction steps on the target three-dimensional point cloud data to obtain the underlying topological structure features of the target three-dimensional point cloud data.

[0009] Preferably, the step of extracting scene-level geometric priors from 2D image data using a visual geometric foundation model, injecting instance mask features during descriptor generation, and mapping the extracted 2D image global features to pseudo-3D manifold features compatible with the underlying structure of the target 3D point cloud data includes: extracting scene-level geometric priors from the 2D image data using a pre-trained visual geometric foundation model, performing instance segmentation processing on the 2D image data to obtain target instance mask features; fusing the scene-level geometric priors and the target instance mask features to generate 2D image global features containing joint priors of spatial topology and target instances; and performing a cross-modal manifold mapping step to project the 2D image global features onto a shared embedded manifold space compatible with the underlying topological structure features of the target 3D point cloud data, thereby transforming them into the pseudo-3D manifold features.

[0010] Preferably, the cross-domain comparison model includes: The cross-domain association construction unit receives the pseudo-3D manifold features and the sparse features of the target 3D point cloud data, calculates the cross-domain feature affinity between the two, and constructs an initial cross-modal association feature map. The geometric loss calculation unit is used to perform regularization calculation on the initial cross-modal association feature map using a preset geometric guided attention loss function, and outputs the geometric guided attention loss used to constrain network training. The saliency weight focusing unit is connected to the geometric loss calculation unit. Based on the geometrically guided attention loss constraint bidirectional feature fusion process, it dynamically adjusts the cross-attention distribution and forces the attention weight to focus on the geometric region with topological saliency in the physical scene, thus maintaining the spatial distinguishability of multimodal features. The mask adaptive generation unit, connected to the saliency weight focusing unit, performs threshold filtering on the focused attention weight distribution, removes background semantics that do not meet the geometric saliency condition, and generates and outputs the cross-modal filtering mask.

[0011] Preferably, the extrinsic parameter sensing cross-attention mechanism includes: obtaining the relative pose transformation matrix between the camera coordinate system and the target 3D point cloud map coordinate system, as a candidate pose parameter for retrieval; The relative pose transformation matrix is ​​encoded into a spatial position embedding vector and explicitly injected into the query or key feature matrix of the cross-attention calculation model to establish a mathematical mapping between two-dimensional pixel coordinates and three-dimensional spatial coordinates. While keeping the pixel topology format of the two-dimensional image data and the sparse point distribution format of the target three-dimensional point cloud data unchanged, the attention weight matrix between the feature block and the feature group is directly calculated by combining the cross-modal filtering mask and the spatial location embedding vector, thus completing the cross-modal native domain feature aggregation.

[0012] Preferably, calculating the similarity between two-dimensional image data and target three-dimensional point cloud data includes: performing a geometric coherence verification step on the native domain feature aggregation result output by the extrinsic perception cross-attention mechanism; aggregating matching relationships and extracting classification vectors through a multi-layer transformer; and mapping the classification vectors to a one-dimensional rearranged scalar score as the similarity between the two-dimensional image data and the target three-dimensional point cloud data.

[0013] Preferably, before inputting the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridging feature extraction channel, the method further includes: extracting a global coarse descriptor of the two-dimensional image data to be retrieved; performing a global cosine similarity calculation on the global coarse descriptor and multiple candidate point cloud features in a pre-constructed three-dimensional point cloud map database; based on the comparison result of the global cosine similarity, recalling the top N candidate point cloud sub-images in terms of similarity from the three-dimensional point cloud map database as the target three-dimensional point cloud data, where N is a natural number greater than 1; correspondingly, the step of sorting and outputting the final retrieval result based on similarity includes: using the similarity output by the feature interaction, performing a local precise rearrangement step on the recalled N target three-dimensional point cloud data, and outputting the rearranged final cross-modal retrieval result.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention injects scene-level geometric priors and instance masks into 2D image features and maps them to pseudo-3D manifold features. This constrained feature enables high compatibility between 2D images and 3D point clouds with significant differences at the underlying structure, effectively bridging the data gap in traditional cross-modal retrieval and laying a solid manifold space foundation for subsequent high-precision feature matching.

[0015] 2. This invention designs a geometrically guided attention loss constraint feature fusion process, dynamically focusing attention weights on geometrically salient regions in the physical scene. This constraint feature drives the generation of cross-modal filtering masks, eliminating useless background semantic interference lacking 3D correspondence, enhancing the spatial discriminative power of multimodal features, and improving the robustness of retrieval in complex scenes.

[0016] 3. This invention utilizes a cross-modal filtering mask-driven extrinsic parameter sensing cross-attention mechanism to establish a mathematical mapping between 2D pixels and 3D space by explicitly injecting camera extrinsic parameters. This constrained feature allows 2D and 3D features to interact directly within their respective native domains, avoiding spatial information loss caused by forced data format conversion and improving cross-modal feature alignment accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart of an image retrieval method based on geometric perception attention proposed in this invention; Figure 2 This is a flowchart illustrating the steps of an image retrieval method based on geometric perception attention proposed in this invention. Figure 3 This is a diagram showing the structure of the cross-domain comparison model proposed in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 Please see Figures 1 to 3 This invention provides an image retrieval method based on geometric perception attention, the technical solution of which is as follows: An image retrieval method based on geometrical awareness attention, such as Figures 1-2 As shown, it includes: The two-dimensional image data to be retrieved and the target three-dimensional point cloud data are input into the bidirectional bridged feature extraction channel; the scene-level geometric prior of the two-dimensional image data is extracted using the visual geometric basic model to obtain the global features of the two-dimensional image; instance mask features are injected during the descriptor generation process to map the global features of the two-dimensional image into pseudo-three-dimensional manifold features that are compatible with the underlying structure of the target three-dimensional point cloud data. The pseudo-3D manifold features and the sparse features of the target 3D point cloud data are input into the cross-domain contrast model to calculate the geometry-guided attention loss. Based on the feature fusion process constrained by the geometry-guided attention loss, the attention weights are focused on the geometrically salient regions in the physical scene to generate a cross-modal filtering mask. By utilizing a cross-modal filtering mask-driven extrinsic perception cross-attention mechanism, the camera extrinsic hypothesis parameters are explicitly injected, enabling feature blocks of 2D image data and feature groups of target 3D point cloud data to interact within their respective native domains. Based on the output of feature interaction, the similarity between two-dimensional image data and target three-dimensional point cloud data is calculated, and the final retrieval results are output based on the similarity ranking.

[0020] The overall technical solution described above is divided into four core phases, and the inputs, outputs, and data dimensions of each phase are explained below: Phase 1 (Two-way bridged feature extraction): Input is the two-dimensional image to be retrieved. (in , These represent the image height and width (3 indicates the RGB channels) and the target 3D point cloud after coarse recall filtering. (Where N is the number of points in the point cloud); the output is the initial features of the two-dimensional image. and the underlying topological features of 3D point clouds and the subsequently generated pseudo-3D manifold features ,in , This represents the size of the downsampled feature map. M represents the number of two-dimensional feature channels, and M represents the number of points after sparse sampling of the point cloud. The number of three-dimensional feature channels. To share the embedded spatial dimension.

[0021] Phase Two (Geometric Prior Extraction and Manifold Mapping): Extracting scene-level geometric prior vectors using a visual geometric foundation model. Obtain the target instance mask through instance segmentation. (K is the number of detected instances), and the fusion generates global features of the two-dimensional image. Transformed into pseudo-3D manifold features via cross-modal manifold mapping .

[0022] Phase Three (Cross-Domain Comparison and Attention Focus): Input is... and Calculate geometrically guided attention loss using a cross-domain contrast model Output cross-modal filtering mask .

[0023] Phase 4 (External Parameter Perception Attention and Similarity Calculation): Combining Combined with extrinsic parameters, perform native domain feature aggregation and output aggregated features. The one-dimensional rearranged scalar score is output after similarity calculation. .

[0024] Furthermore, the step of inputting the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridged feature extraction channel includes: acquiring the two-dimensional image data to be retrieved and the target three-dimensional point cloud data respectively; performing a preliminary visual feature extraction step on the two-dimensional image data to obtain initial features of the two-dimensional image; and performing independent point cloud feature extraction steps on the target three-dimensional point cloud data to obtain the underlying topological structure features of the target three-dimensional point cloud data.

[0025] The preliminary visual feature extraction step employs a ConvNeXt-Base model pre-trained on a large-scale visual dataset (ImageNet-22K) as the backbone network for two-dimensional image feature extraction. Specifically, the two-dimensional image to be retrieved... After scaling to a uniform resolution of 224×224 pixels, the image is input into the ConvNeXt-Base network, and the feature map output from the fourth stage of the network is used as the initial feature map of the two-dimensional image. Then, a 1×1 convolutional layer is used to compress the number of channels to... ,get The independent point cloud feature extraction steps employ a PointNet++ model pre-trained on the ShapeNet and KITTI datasets as the backbone network for 3D point cloud feature extraction. Specifically, for the input target 3D point cloud data... First, the points are downsampled to M=2048 points using farthest point sampling. Then, the data is input into the PointNet++ network, where local geometric features are extracted step by step through an ensemble abstraction layer. The point-by-point features output by the network are then used as the underlying topological features of the 3D point cloud. Here, the two-dimensional feature extraction channel and the three-dimensional feature extraction channel operate independently and do not share parameters, each maintaining its own modal data characteristics. Therefore, this is called "bidirectional bridging"—the two channels are bridged and aligned in the subsequent shared embedding manifold space.

[0026] This invention constructs a bidirectional, non-shared feature extraction channel to independently extract features from 2D images and 3D point clouds. This design preserves the inherent visual semantics and underlying topology of each modality to the greatest extent possible, avoiding cross-interference in the early stages of feature extraction. This constrained feature effectively bridges the data gap between 2D and 3D, laying the foundation for subsequent high-precision cross-modal alignment within a shared space.

[0027] Furthermore, the step of extracting scene-level geometric priors from 2D image data using a visual geometric foundation model, injecting instance mask features during descriptor generation, and mapping the extracted 2D image global features to pseudo-3D manifold features compatible with the underlying structure of the target 3D point cloud data includes: extracting scene-level geometric priors from the 2D image data using a pre-trained visual geometric foundation model, performing instance segmentation processing on the 2D image data to obtain target instance mask features; fusing the scene-level geometric priors and the target instance mask features to generate 2D image global features containing joint priors of spatial topology and target instances; and performing a cross-modal manifold mapping step to project the 2D image global features onto a shared embedded manifold space compatible with the underlying topological structure features of the target 3D point cloud data, thereby transforming them into the pseudo-3D manifold features.

[0028] The specific visual geometry foundation model adopts the DINOv2-ViT-Large model (based on the VisionTransformer architecture, containing 24 Transformer layers, with a latent space dimension of 1024, and self-supervised pre-trained on the LVD-142M dataset containing 142 million images) as the backbone network, and combines it with a geometry prior extraction network composed of depth / normal vector prediction heads, utilizing its scene understanding and geometry perception capabilities to extract scene-level geometry prior vectors. .

[0029] The instance segmentation process specifically employs the ViT-Huge version of the Segment Anything Model (SAM) to perform zero-shot instance segmentation on the two-dimensional image data. The SAM model receives the two-dimensional image... As input, the system outputs a set of instance segmentation masks through three sub-modules: an image encoder, a cue encoder, and a mask decoder. Each of them Let be the binary mask of the k-th instance. Take the union of all instance masks and binarize it (with a threshold of 0.5) to obtain the target instance mask features. .

[0030] The fusion calculation employs a combination of channel-dimensional feature stitching and linear projection. First, the scene geometric prior vectors are... Extended to the initial features of the two-dimensional image through broadcasting operations Feature map with consistent spatial dimensions Then, the target instance mask features are... Downsampling to 7×7 resolution .Will and (After expanding to a single channel) the features are concatenated along the channel dimension to obtain joint features. Then, a 1×1 convolutional projection layer and a global average pooling operation are used to generate two-dimensional image global features that include joint priors of spatial topology and target instances. .

[0031] The purpose of the cross-modal manifold mapping step is to map features from the two-dimensional image domain to a shared manifold space compatible with three-dimensional point cloud features. The specific process is as follows: Let the global features of the two-dimensional image be The target shared embedded manifold space is Its dimensions are Cross-modal manifold mapping function Defined as a three-layer multilayer perceptron combined with unit hyperspherical normalization and inverse reconstruction regularization constraints: First layer: ,in , Second layer: ,in , Third layer: ,in , Apply L2 normalization constraints to the mapping result to ensure that it lies on the unit hypersphere: Further introduce manifold-preserving regularity terms Through a learnable inverse mapping matrix Reconstructing pseudo-3D features back into the original 2D feature space while constraining the mapping process to preserve the manifold geometry: The first item is reconstruction losses, ensuring... Able to recover the original two-dimensional global features through linear mapping Key information in the text; the second item is the approximate orthogonal constraint ( The column vectors of the inverse mapping matrix are constrained to be approximately orthogonal to maintain the geometric stability of the manifold projection. The learnable inverse mapping matrix is ​​randomly initialized using a normal distribution, and end-to-end parameter updates are performed during the pre-training and fine-tuning phases using the backpropagation gradient of the geometrically guided attention loss function to ensure that the generated pseudo-3D manifold features are consistent with the global features of the 2D image.

[0032] Will As a basis vector, in By replicating and fine-tuning at various spatial locations, pseudo-3D manifold features are obtained. .in The pseudopoints are generated by the following method: [using the two-dimensional feature map] Each spatial grid location is uniformly sampled along the camera's optical axis. Several hypothetical depth values ​​(depth range 1m~80m, logarithmic interval sampling) are back-projected into 3D space using camera intrinsic parameters, totaling... 10 candidate 3D reference points were subsequently truncated to a total of 1000 points by farthest point sampling or confidence ranking. There are three-dimensional reference points. The features at each reference point are defined by basis vectors. The positional encoding offset of the 3D coordinates of this point is added to obtain the feature of the underlying topology of the 3D point cloud. It features a compatible manifold structure and corresponding number of points. This mapping ensures that the transformed 2D features are compatible with the underlying topological features of the 3D point cloud in the embedding space. It has a compatible manifold structure, which facilitates subsequent cross-domain comparative learning.

[0033] This invention introduces a visual geometric foundation model and an instance segmentation mask to deeply integrate 2D global features with scene geometry and instance topological priors. This constrained feature, combined with a mapping mechanism featuring manifold-preserving regularization, reduces the loss of key information and maintains the stability of the spatial geometry when generating pseudo-3D manifold features. This effectively eliminates the modal barrier between 2D and 3D, improving the accuracy of cross-modal feature alignment.

[0034] Furthermore, such as Figure 3 As shown, the cross-domain comparison model includes: The cross-domain association construction unit receives the pseudo-3D manifold features and the sparse features of the target 3D point cloud data, calculates the cross-domain feature affinity between the two, and constructs an initial cross-modal association feature map. The geometric loss calculation unit is used to perform regularization calculation on the initial cross-modal association feature map using a preset geometric guided attention loss function, and outputs the geometric guided attention loss used to constrain network training. The saliency weight focusing unit is connected to the geometric loss calculation unit. Based on the geometrically guided attention loss constraint bidirectional feature fusion process, it dynamically adjusts the cross-attention distribution and forces the attention weight to focus on the geometric region with topological saliency in the physical scene, thus maintaining the spatial distinguishability of multimodal features. The mask adaptive generation unit, connected to the saliency weight focusing unit, performs threshold filtering on the focused attention weight distribution, removes background semantics that do not meet the geometric saliency condition, and generates and outputs the cross-modal filtering mask.

[0035] The specific implementations of each unit in the cross-domain comparison model are as follows: (a) Cross-domain association building unit This unit receives pseudo-3D manifold features. sparse features of 3D point clouds First, a linear projection layer is used to... Mapping to shared dimensions ,in .

[0036] Cross-domain feature affinity (i.e., 2048×2048) Calculated using scaled cosine similarity: in, for The i-th eigenvector, for The j-th eigenvector, The temperature coefficient is set to 0.07. This represents the vector dot product.

[0037] Apply softmax normalization to each row of the affinity matrix to construct an initial cross-modal association feature map. : (ii) Geometric Loss Calculation Unit This unit utilizes the following geometry-guided attention loss function to process the initial cross-modal association feature map. Perform regularization calculation: Define the geometric guided attention loss function The weighted combination of the three losses: Among them: First, the pseudo-3D manifold features are... and the features of the 3D point cloud after projection Global descriptors are obtained by global average pooling respectively. and Then, compare them at the batch level: In the formula, The training batch size is set to 64, meaning each batch contains 64 pairs of paired samples (two-dimensional image, three-dimensional point cloud sub-image). Let i be the global pseudo-3D descriptor for the i-th image sample. It is the global descriptor of the positive sample 3D point cloud corresponding to the i-th image sample (i.e., the point cloud sub-image from the same physical scene). This serves as the global descriptor for other point cloud samples within the batch (as a negative sample). The cosine similarity function is used. For comparison, the temperature coefficient is set to 0.07.

[0038] Geometric consistency regularization term Then, the constraint ensures that the cross-modal correlation feature map maintains topological consistency in space: In the formula, Let K be the set of indices of the K-nearest neighbors (K=8) of the i-th pseudopoint in 3D space. This loss term forces neighboring points to have similar association patterns in the cross-modal association space.

[0039] Attention sparsity regularization Constrain the distribution of attention weights: In the formula, the first term is the L1 norm constraint that promotes sparsity, and the second term is the entropy-based constraint: It is the entropy function. is the entropy regularization coefficient (set to 0.01).

[0040] Geometric significance supervision term Explicit spatial structure constraints are introduced to force the cross-modal feature fusion process to align with the topological boundaries of the physical space: In the formula, The geometric saliency pseudo-label is obtained by jointly calculating the depth gradient edges of the 2D image and the local normal curvature of the 3D point cloud. This constrained feature provides the network with a realistic geometric prior of the physical scene, thus constraining the attention weights. Actively focus on topologically significant regions with high curvature or deep transitions.

[0041] The weighting coefficients are empirically set as follows: , , , .

[0042] (III) Significance Weighting Focus Unit This unit is based on geometrically guided attention loss. The backpropagation gradient is used to dynamically adjust the cross-attention distribution. Specifically, during training, the backpropagation gradient is used to dynamically adjust the cross-attention distribution. As a supervisory signal, the linear projection layer parameters in the cross-domain correlation building blocks and the temperature parameters in the affinity calculation are updated via gradient descent.

[0043] Attention weight matrix after focusing The calculation process is as follows: (1) Calculate the basic score of cross-attention: in For learnable projection matrices, For attention head dimension.

[0044] Introducing geometric prior bias terms to link the initial cross-modal associated feature maps (Taking the logarithmic field) as a geometrically guided bias for attention-injected computation: in The geometric guiding strength coefficient is set to 0.3. A very small constant to prevent numerical overflow. Each element in The geometric affinity between the i-th pseudo 3D feature and the j-th 3D point cloud feature is characterized and injected into the softmax calculation in the form of a logarithmic bias. This can make the attention weights biased towards geometrically consistent cross-modal matching pairs while keeping the gradient differentiable.

[0045] Through the above mechanism The attention weights are forced to focus on geometric regions with topological saliency in the physical scene (such as building edges, road markings, lampposts, etc.), while suppressing regions lacking geometric distinctiveness, such as flat roads and the sky.

[0046] (iv) Mask Adaptive Generation Unit This unit focuses on the distribution of attention weights after focusing. Perform adaptive threshold filtering. Threshold The calculation method is as follows: in, The global mean of the attention weights. The standard deviation of attention weights This is an adjustable coefficient (set to 0.5).

[0047] Threshold binarization is performed on the attention weights: The binary mask is maximized along the column direction of the 3D point cloud to obtain the geometric saliency score vector corresponding to each pseudo-3D point. .because The pseudopoints are arranged according to 49 spatial grid locations × 42 depth hypotheses. The average significance score of the 42 depth hypotheses for each spatial grid location is taken to obtain the spatial significance map. .Will Perform bilinear interpolation upsampling to the original image resolution The cross-modal filtering mask is obtained. .

[0048] This mask effectively eliminates background semantics in background regions (such as the sky, distant vegetation, etc.) that do not meet the geometric saliency condition (i.e., lack pixel regions corresponding to a clear 3D geometric structure), and only retains the foreground regions that have a clear geometric structure in the 3D point cloud, which are used to drive subsequent extrinsic parameter perception cross-attention calculation.

[0049] This invention constructs a cross-domain contrast model that utilizes geometrically guided attention loss to constrain feature fusion, dynamically focusing attention weights on topologically significant geometric regions to effectively suppress interference from indistinguishable backgrounds. Combined with adaptively generated cross-modal filtering masks, the influence of background regions on attention distribution is further suppressed. This constrained feature enhances the spatial discriminative power of multimodal features, eliminates invalid information for subsequent feature interactions, and improves the accuracy and robustness of cross-modal retrieval.

[0050] Furthermore, the extrinsic parameter perception cross-attention mechanism includes: obtaining the relative pose transformation matrix between the camera coordinate system and the target 3D point cloud map coordinate system, as a candidate pose parameter for retrieval; The relative pose transformation matrix is ​​encoded into a spatial position embedding vector and explicitly injected into the query or key feature matrix of the cross-attention calculation model to establish a mathematical mapping between two-dimensional pixel coordinates and three-dimensional spatial coordinates. While keeping the pixel topology format of the two-dimensional image data and the sparse point distribution format of the target three-dimensional point cloud data unchanged, the attention weight matrix between the feature block and the feature group is directly calculated by combining the cross-modal filtering mask and the spatial location embedding vector, thus completing the cross-modal native domain feature aggregation.

[0051] The key design of the extrinsic parameter-aware cross-attention mechanism lies in explicitly encoding the camera extrinsic parameters into learnable position embedding vectors, enabling the cross-attention calculation process to have cross-modal alignment capabilities. The specific implementation is as follows: The relative pose transformation matrix It is a homogeneous transformation matrix containing rotation components. Translation components : Specifically, R is a 3×3 orthogonal rotation matrix ( ), describes the rotational transformation from the camera coordinate system to the target 3D point cloud map coordinate system; It is a three-dimensional translation vector, representing the translation transformation of the camera in the coordinate system of the target's three-dimensional point cloud map.

[0052] In practical application scenarios where the extrinsic parameters are unknown or contain noise, this invention adopts the following two extrinsic parameter assumption strategies: (1) Cascaded Discrete Sampling Strategy: To address the real-time requirements of high-dynamic scenarios such as autonomous driving, this strategy avoids computational overload caused by blind global sampling. First, based on the initial pose prior provided by the vehicle's inertial navigation system or coarse GPS, in locally constrained... First, candidate poses are generated within the space to narrow the search range. Second, a coarse-to-fine cascaded selection mechanism is adopted, using lightweight global feature cosine similarity as a surrogate evaluation to quickly select the top candidate poses from the local space. One (e.g.) High-confidence pose; finally, only this Each high-confidence pose is used as a hypothetical parameter of the camera extrinsic parameters. These are then input into the complete extrinsic perception cross-attention model for forward computation, and the result corresponding to the candidate pose with the highest similarity score is taken.

[0053] (2) Learning estimation strategy: using a lightweight extrinsic estimation network ,by and As input, direct regression predicts the relative pose transformation matrix. .

[0054] Transform the extrinsic parameter matrix Encoded as spatial location embedding vector The process is as follows: For the location in the two-dimensional feature map The pixel, and its corresponding three-dimensional spatial reference point. (In the normalized camera coordinate system) is: in For the camera intrinsic parameter matrix, This is the estimated depth value for this pixel inferred from the visual geometric fundamental model. Transform this reference point to the target 3D point cloud map coordinate system: (Taking the first three dimensions) for the transformed coordinates The three coordinate dimensions are respectively encoded using sine-cosine position coding. : For each coordinate dimension and frequency index : The three-dimensional encoded vectors are concatenated to obtain a dimension of The encoded vector is then processed through a two-layer MLP (input dimension...) The process involves 128 hidden layers and 64 output layers to obtain the final spatial location embedding vector. .all The positions of the pseudopoints are embedded to form a matrix. .

[0055] Define the query matrix Q, the key matrix K, and the value matrix V: (External parameter injection query) (3D point coordinate position encoding injection key) in , All are learnable projection matrices; The spatial position is embedded by directly encoding the coordinates of the three-dimensional point cloud using sine-cosine encoding.

[0056] Extrinsic parameter perception cross-attention weight matrix (Preserving the row dimension of the pixel topology format of a two-dimensional image) (The column dimension M of the sparse point distribution format of the 3D point cloud is calculated directly without format conversion). This is a geometrically perceptual additive masking matrix. Its attention domain extension matrix is ​​based on cross-modal filtering masks. Build, for The geometric salient position, corresponding to ;for The background position, corresponding By injecting minimal negative values ​​before weight normalization, active suppression of background semantics is achieved while maintaining the sum of attention weights in the current row at 1. Specifically, if the attention domain expansion matrix... If an entire row in the algorithm is set to 0 (meaning all 3D candidate points corresponding to the current pixel block are considered non-geometrically salient background), then the backoff mechanism is activated for that row, automatically retaining the row's base cross-attention score. One (e.g.) =8) The highest affinity position is not masked to prevent numerical computation failure and to ensure the continuity and interpretability of multimodal feature space aggregation.

[0057] use By performing weighted aggregation on the value matrix V, we obtain the native domain aggregation features of the two-dimensional image domain: Simultaneously, calculate the reverse aggregation features in the 3D point cloud domain: Concatenate the aggregated features from two directions along the feature dimension: Then, a linear projection layer is used to unify the feature dimensions to... , Here, the native domain means that two-dimensional image features retain their pixel topology (i.e., spatial arrangement with pixels as basic units), and three-dimensional point cloud features retain their sparse point distribution (i.e., an unordered set with points as basic units). The two interact within their respective native data domains, avoiding information loss caused by format conversion.

[0058] This invention constructs an extrinsic parameter-aware cross-attention mechanism, explicitly encoding the camera's relative pose as a position embedding and injecting it into the interaction model, establishing a precise mathematical mapping between 2D pixels and 3D coordinates. This constrained feature allows for direct feature aggregation of multimodal data while preserving their original formats, effectively avoiding information loss caused by format conversion. Combined with a filtering mask to accurately shield against background interference, it improves the accuracy of cross-modal feature alignment and interaction efficiency.

[0059] Further, calculating the similarity between the two-dimensional image data and the target three-dimensional point cloud data includes: performing a geometric coherence verification step on the native domain feature aggregation result output by the extrinsic perception cross-attention mechanism; aggregating matching relationships and extracting classification vectors through a multi-layer transformer; and mapping the classification vectors to a one-dimensional rearranged scalar score as the similarity between the two-dimensional image data and the target three-dimensional point cloud data.

[0060] The specific implementation of the similarity calculation stage is as follows: Aggregate native domain features output by the extrinsic parameter-aware cross-attention mechanism As input, the purpose of geometric coherence verification is to examine whether the aggregated cross-modal features satisfy the geometric consistency constraint, that is, whether the matched features have spatial coherence in three-dimensional space.

[0061] right Each pair of adjacent pseudopoints in (Based on the definition of K-nearest neighbor relationship in three-dimensional space, K=8), calculate its feature consistency score. : in This represents vector concatenation. For learnable parameters, This is the Sigmoid activation function. For all adjacent pseudopoint pairs... Take the average value; if the average value is lower than the threshold... If the consistency test fails, the candidate search is discarded or its final similarity score is reduced.

[0062] A standard 2-layer Transformer encoder is used (without position encoding, because...). (Already includes spatial location information) Perform global matching relationship aggregation: The Transformer encoder's parameter configuration for each layer is as follows: 8 multi-head self-attention heads (h=8), attention dimension 512 (64 dimensions per head), feedforward hidden layer dimension 1024, GELU activation function used, and Dropout rate = 0.1. No. The output of the Transformer layer is: in For standard multi-head self-attention mechanisms, It is a two-layer MLP.

[0063] Take the global average pooled features from the last layer of the Transformer as the classification vector. : ( (Total number of layers) will be used to classify the vectors. The similarity is mapped to a one-dimensional scalar score through a two-layer MLP: First layer: ,in , Second layer (output layer): ,in , , It is an unnormalized scalar fraction.

[0064] right Apply the Sigmoid activation function and map it to Interval: This is the final similarity score between the two-dimensional image data and the target three-dimensional point cloud data. The closer the score is to 1, the higher the retrieval matching degree.

[0065] This invention effectively verifies the spatial consistency of cross-modal features by introducing geometric coherence verification, eliminating or reducing the weight of incorrect matches, and improving retrieval reliability. By combining a multi-layer transformer to aggregate global matching relationships, it mines deep contextual information between features and ultimately maps it to a high-precision similarity scalar score. This constrained feature achieves an organic combination of local geometric verification and global semantic measurement, improving the accuracy of cross-modal retrieval of two-dimensional and three-dimensional data.

[0066] Furthermore, before inputting the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridging feature extraction channel, the method further includes: extracting a global coarse descriptor of the two-dimensional image data to be retrieved; performing a global cosine similarity calculation between the global coarse descriptor and multiple candidate point cloud features in a pre-constructed three-dimensional point cloud map database; based on the comparison result of the global cosine similarity, recalling the top N candidate point cloud sub-images in terms of similarity from the three-dimensional point cloud map database as the target three-dimensional point cloud data, where N is a natural number greater than 1; correspondingly, the step of sorting and outputting the final retrieval result based on similarity includes: using the similarity output by the feature interaction, performing a local precise rearrangement step on the recalled N target three-dimensional point cloud data, and outputting the rearranged final cross-modal retrieval result.

[0067] For the two-dimensional image to be retrieved This paper employs a widely used approach in image retrieval tasks to extract global coarse descriptors: a pre-trained ResNet-50 backbone network (pre-trained on ImageNet) is used, and the 2048-dimensional feature vector after the network's Global Average Pooling (GAP) layer is taken as the initial global features. Based on this, Generalized Mean Pooling (GeM) is introduced to replace the traditional GAP operation to enhance the discriminative power of the features. in, This refers to the specific value in the k-th dimension (or k-th feature channel) of the final generated global coarse descriptor vector. The feature map spatial domain (7×7) is the output of the last convolutional layer of ResNet-50. For the first The activation value of each channel at position x. Set the GeM pooling parameters (to 3.0). The final output is a global coarse descriptor. .

[0068] The construction process of the three-dimensional point cloud map database is as follows: (1) The target scene is scanned by LiDAR to obtain a three-dimensional point cloud map of the complete scene; (2) The complete point cloud map is divided into multiple candidate point cloud sub-maps using a sliding window strategy. Each sub-map covers a spatial area of ​​approximately 20 meters × 20 meters, and there is 50% spatial overlap between adjacent sub-maps to cover the boundary conditions; (3) For each candidate point cloud sub-map, the global point cloud features are extracted using the PointNet++ network. (and (Dimensional alignment), stored in the database.

[0069] Global coarse descriptor for the image to be retrieved Features of all candidate point clouds in the database Perform cosine similarity calculation: In the formula, .

[0070] After calculation, all candidate point cloud sub-images are sorted from highest to lowest cosine similarity. The top N candidate point cloud sub-images are selected as the target 3D point cloud data for subsequent fine ranking. The value of N needs to strike a balance between recall and computational efficiency; a recommended range is [range missing]. In a specific embodiment, the specific value of N can be determined by the validation set; for example, N=100 can be selected when the candidate set size, map density and recall curve meet preset conditions, so as to control the fine ranking calculation overhead within a reasonable range.

[0071] For the N target 3D point cloud data points retrieved in coarse recall, the following steps are performed: bidirectional seamless feature extraction → geometric prior extraction and manifold mapping → cross-domain comparison and attention focusing → extrinsic parameter perception attention and similarity calculation, resulting in N similarity scores. Reorder the N candidates from highest to lowest similarity score, and output the final cross-modal retrieval result list.

[0072] Similarity scores used in the rearrangement stage The computation not only utilizes geometric coherence verification and multi-layer transformer aggregation, but also introduces global cosine similarity as a prior bias term for fusion: in Set the fusion weight (to 0.8). This refers to the similarity score output during the fine-ranking stage. This refers to the global cosine similarity during the coarse recall phase. This fusion strategy combines the coarse-grained discrimination capability of global descriptors with the fine-grained matching capability of geometric perception attention, further improving retrieval accuracy.

[0073] The deep neural network model involved in this invention employs a two-stage training strategy: Phase 1 (Pre-training): Before pre-training, 2D images and corresponding global 3D point clouds are extracted from the dataset. Using the camera pose at the time of 2D image acquisition as the origin, 3D point clouds are cropped within a preset spatial distance range as positive sample point cloud sub-images. Simultaneously, other point cloud sub-images are randomly selected outside this spatial distance range as negative sample point cloud sub-images to construct a paired dataset for pre-training. Pre-training is performed on large-scale 2D-3D paired datasets (such as KITTI-Odometry and NuScenes) using InfoNCE contrastive loss to learn the basic alignment capability of cross-modal feature representations. The optimizer uses AdamW with an initial learning rate of... The batch size is 64, and the training lasts for 100 epochs.

[0074] Phase Two (Fine-tuning): Apply the full geometry-guided attention loss function to the target scene data. End-to-end fine-tuning was performed. The optimizer used SGD with Momentum (momentum=0.9), with an initial learning rate of... The batch size is 16, and the training lasts for 50 epochs. Cosine annealing is used for learning rate scheduling, with the minimum learning rate reduced to [missing value]. .

[0075] This invention constructs a two-stage retrieval framework of "coarse recall - fine re-ranking". It utilizes global descriptors to quickly filter candidate point clouds, effectively balancing high recall rate with computational overhead. The re-ranking stage deeply integrates global coarse-grained priors with local fine-grained geometric similarity, improving the final matching accuracy. Simultaneously, combined with a two-stage training strategy, it further enhances the model's fundamental ability for cross-modal feature alignment and its generalization performance across target scenes.

[0076] This invention effectively bridges the cross-modal data gap by fusing 2D image features with geometric priors and mapping them to pseudo-3D manifold features. These constrained features, combined with geometrically guided attention loss, generate cross-modal filtering masks, enabling the model to accurately focus on salient regions and eliminate background interference. Furthermore, an extrinsic-aware cross-attention mechanism is introduced to achieve feature interaction between multimodal data within their respective native domains, avoiding information loss caused by format conversion. The synergistic effect of these constrained features improves the overall accuracy and robustness of cross-modal retrieval.

[0077] Example 2 Example 2 provides a specific execution process for applying an image retrieval method based on geometric perception attention in an urban canyon positioning scenario for autonomous vehicles. In this scenario, the vehicle's GPS signal is weak or briefly lost due to obstruction by tall buildings, making accurate positioning impossible. Therefore, the retrieval system needs to input the two-dimensional street view images collected in real time by the onboard visual sensor. The system then retrieves the most matching three-dimensional point cloud sub-image from a pre-constructed high-precision three-dimensional point cloud map of the city to achieve high-precision cross-modal assisted positioning.

[0078] First, the system extracts a global coarse descriptor from the real-time 2D street view image of the current intersection captured by the vehicle-mounted camera. A pre-trained ResNet-50 backbone network combined with generalized mean pooling is used to extract an initial 2048-dimensional global feature. This global coarse descriptor is then compared with all candidate point cloud features in a pre-built 3D point cloud map database within the vehicle system to calculate global cosine similarity. Based on the global cosine similarity comparison results, the top 100 candidate point cloud sub-images (each sub-image representing a 20m × 20m local 3D spatial region at a possible location around the vehicle) are quickly retrieved from the massive urban point cloud database and used as target 3D point cloud data for the fine-ranking stage.

[0079] Subsequently, the 2D street view image to be retrieved and the coarsely recalled 3D point cloud data of 100 targets are sequentially input into the bidirectional bridged feature extraction channel. The system acquires the current 2D street view image and each candidate point cloud sub-image respectively. For the 2D street view image, a ConvNeXt-Base model is used to perform a preliminary visual feature extraction step to obtain the initial features of the 2D image. At the same time, for each target 3D point cloud data, a PointNet++ model is used to perform independent point cloud feature extraction steps. By downsampling to 2048 points through farthest point sampling and extracting local geometric features, the underlying topological structure features of the target 3D point cloud data are obtained.

[0080] Next, a pre-trained visual geometry foundation model is used to extract scene-level geometric priors (e.g., perception of intersection depth gradients) from the 2D street view image. A zero-shot instance segmentation model is then used to perform instance segmentation on the 2D street view image, obtaining target instance mask features such as traffic lights, lampposts, and surrounding buildings. The scene-level geometric priors and target instance mask features are then concatenated and fused along the channel dimension to generate a 2D image global feature containing a joint prior of spatial topology and target instances. A cross-modal manifold mapping step is then performed, projecting the 2D image global feature onto a shared embedded manifold space compatible with the underlying topological features of the target 3D point cloud data, transforming it into pseudo-3D manifold features. Through this constrained feature design, combined with manifold-preserving regularization, the mapping process ensures that the stability of the physical spatial geometry is maintained while minimizing the loss of key information.

[0081] Then, the generated pseudo-3D manifold features and the sparse features of the target 3D point cloud data are input into the cross-domain contrast model. An initial cross-modal association feature map is constructed by calculating the cross-domain feature affinity between the two, and regularization is performed using the geometry-guided attention loss function. Based on the backpropagation gradient constraint of the geometry-guided attention loss, the bidirectional feature fusion process dynamically adjusts the cross-attention distribution, forcing the attention weights to focus on geometrically significant regions in the current street scene (such as rigid building edges, clear lane edges, etc.), and performs threshold filtering on the focused attention weight distribution to remove background semantics that do not meet the geometric saliency condition (such as large areas of dynamic clouds, distant tree canopies, or moving irrelevant vehicles), generating and outputting a cross-modal filter mask.

[0082] Next, the generated cross-modal filtering mask is used to drive the extrinsic perception cross-attention mechanism. The relative pose transformation matrix between the vehicle camera coordinate system and the target city's high-precision 3D point cloud map coordinate system is obtained (i.e., candidate poses are retrieved, such as by generating and selecting the optimal candidate pose parameters in the local space through a discrete sampling strategy). This relative pose transformation matrix is ​​encoded as a spatial location embedding vector and explicitly injected into the query or key feature matrix of the cross-attention calculation model. While maintaining the pixel topology of the 2D street view image and the sparse point distribution of the target 3D point cloud data, the attention weight matrix between the 2D feature block and the 3D feature group is directly calculated by combining the cross-modal filtering mask and the spatial location embedding vector, completing the cross-modal native domain feature aggregation.

[0083] Finally, a geometric coherence verification step is performed on the native domain feature aggregation results to check whether the matched cross-modal features have spatial coherence in the 3D physical space (e.g., whether the identified continuous road segment features remain continuous in the point cloud space). Incorrect matches that fail the coherence check are discarded. Subsequently, a two-layer standard Transformer encoder aggregates global matching relationships and extracts classification vectors, which are then mapped to a one-dimensional rearranged scalar score via a multilayer perceptron. Using the similarity output from this feature interaction, combined with global cosine similarity as a prior bias term, a weighted fusion is performed on the recalled 100 target 3D point cloud data to perform a local precise rearrangement step. The 3D point cloud sub-image with the highest similarity score is output as the final retrieval result, thus providing accurate 3D reference coordinates for the current position of autonomous vehicles.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image retrieval method based on geometrical awareness attention, characterized in that, include: Input the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridge feature extraction channel; Scene-level geometric priors of 2D image data are extracted using a visual geometric foundation model to obtain global features of the 2D image; During the descriptor generation process, instance mask features are injected to map the global features of the two-dimensional image into pseudo-three-dimensional manifold features that are compatible with the underlying structure of the target three-dimensional point cloud data. The pseudo-3D manifold features and the sparse features of the target 3D point cloud data are input into the cross-domain contrast model to calculate the geometry-guided attention loss. Based on the feature fusion process constrained by the geometry-guided attention loss, the attention weights are focused on the geometrically salient regions in the physical scene to generate a cross-modal filtering mask. By utilizing a cross-modal filtering mask-driven extrinsic perception cross-attention mechanism, the camera extrinsic hypothesis parameters are explicitly injected, enabling feature blocks of 2D image data and feature groups of target 3D point cloud data to interact within their respective native domains. Based on the output of feature interaction, the similarity between two-dimensional image data and target three-dimensional point cloud data is calculated, and the final retrieval results are output based on the similarity ranking.

2. The image retrieval method based on geometrical awareness attention according to claim 1, characterized in that, The step of inputting the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridged feature extraction channel includes: acquiring the two-dimensional image data to be retrieved and the target three-dimensional point cloud data respectively; performing a preliminary visual feature extraction step on the two-dimensional image data to obtain the initial features of the two-dimensional image; and performing independent point cloud feature extraction steps on the target three-dimensional point cloud data to obtain the underlying topological structure features of the target three-dimensional point cloud data.

3. The image retrieval method based on geometrical awareness attention according to claim 1, characterized in that, The method of extracting scene-level geometric priors from 2D image data using a pre-trained visual geometric foundation model, injecting instance mask features during descriptor generation, and mapping the extracted 2D image global features into pseudo-3D manifold features compatible with the underlying structure of the target 3D point cloud data includes: extracting scene-level geometric priors from the 2D image data using a pre-trained visual geometric foundation model, performing instance segmentation processing on the 2D image data to obtain target instance mask features; fusing the scene-level geometric priors and the target instance mask features to generate 2D image global features containing joint priors of spatial topology and target instances; and performing a cross-modal manifold mapping step to project the 2D image global features onto a shared embedded manifold space compatible with the underlying topological structure features of the target 3D point cloud data, thereby transforming them into the pseudo-3D manifold features.

4. The image retrieval method based on geometrical awareness attention according to claim 1, characterized in that, The cross-domain comparison model includes: The cross-domain association construction unit receives the pseudo-3D manifold features and the sparse features of the target 3D point cloud data, calculates the cross-domain feature affinity between the two, and constructs an initial cross-modal association feature map. The geometric loss calculation unit is used to perform regularization calculation on the initial cross-modal association feature map using a preset geometric guided attention loss function, and outputs the geometric guided attention loss used to constrain network training. The saliency weight focusing unit is connected to the geometric loss calculation unit. Based on the geometrically guided attention loss constraint bidirectional feature fusion process, it dynamically adjusts the cross-attention distribution and forces the attention weight to focus on the geometric region with topological saliency in the physical scene, thus maintaining the spatial distinguishability of multimodal features. The mask adaptive generation unit, connected to the saliency weight focusing unit, performs threshold filtering on the focused attention weight distribution, removes background semantics that do not meet the geometric saliency condition, and generates and outputs the cross-modal filtering mask.

5. The image retrieval method based on geometrical awareness attention according to claim 1, characterized in that, The extrinsic parameter perception cross-attention mechanism includes: obtaining the relative pose transformation matrix between the camera coordinate system and the target 3D point cloud map coordinate system, as a candidate pose parameter for retrieval; The relative pose transformation matrix is ​​encoded into a spatial position embedding vector and explicitly injected into the query or key feature matrix of the cross-attention calculation model to establish a mathematical mapping between two-dimensional pixel coordinates and three-dimensional spatial coordinates. While keeping the pixel topology format of the two-dimensional image data and the sparse point distribution format of the target three-dimensional point cloud data unchanged, the attention weight matrix between the feature block and the feature group is directly calculated by combining the cross-modal filtering mask and the spatial location embedding vector, thus completing the cross-modal native domain feature aggregation.

6. The image retrieval method based on geometrical awareness attention according to claim 5, characterized in that, Calculating the similarity between two-dimensional image data and target three-dimensional point cloud data includes: performing a geometric coherence verification step on the native domain feature aggregation result output by the extrinsic perceptual cross-attention mechanism; aggregating matching relationships and extracting classification vectors through a multi-layer transformer; and mapping the classification vectors to a one-dimensional rearranged scalar score as the similarity between the two-dimensional image data and the target three-dimensional point cloud data.

7. The image retrieval method based on geometrical awareness attention according to claim 1, characterized in that, Before inputting the two-dimensional image data to be retrieved and the target three-dimensional point cloud data into the bidirectional bridged feature extraction channel, the method further includes: extracting a global coarse descriptor of the two-dimensional image data to be retrieved; performing a global cosine similarity calculation on the global coarse descriptor and multiple candidate point cloud features in a pre-constructed three-dimensional point cloud map database; based on the comparison result of the global cosine similarity, recalling the top N candidate point cloud sub-images with the highest similarity from the three-dimensional point cloud map database as the target three-dimensional point cloud data, where N is a natural number greater than 1; correspondingly, the step of sorting and outputting the final retrieval result based on similarity includes: using the similarity output by the feature interaction, performing a local precise rearrangement step on the recalled N target three-dimensional point cloud data, and outputting the rearranged final cross-modal retrieval result.

Images