Cross-view localization method for autonomous driving based on cascaded mask guidance

Abstract

The application discloses a kind of automatic driving cross-view positioning methods based on cascading mask guide, comprising: obtaining ground image and satellite map slice covering corresponding position, respectively extracting coarse-grained features and fine-grained features of ground image and satellite map, predicting the probability distribution that vehicle position can appear in multiple coarse-grained grid areas of satellite map, and at least one high-confidence grid area is filtered out accordingly;According to high-confidence grid area, the corresponding binary spatial mask is generated, and the binary spatial mask is converted into attention bias matrix, which is used to constrain the search range in feature matching;Fine-grained features and attention bias matrix are input into fine-grained regression network, attention bias matrix is used to adjust the attention calculation of satellite map fine-grained features, and fine-grained features of ground image are fused to regress the fine offset of the position of vehicle in the high-confidence grid area.The application improves the accuracy and stability of visual matching positioning.

Description

Technical Field

[0001] This application relates to a cross-view localization method for autonomous driving based on cascaded mask guidance, which belongs to the field of autonomous driving localization. Background Technology

[0002] Autonomous driving systems place high demands on vehicle positioning accuracy and stability. In complex urban road environments, GNSS signals are susceptible to building obstruction and multipath effects, leading to positioning drift or failure, which fails to meet the continuous high-precision positioning requirements of autonomous driving. Existing methods increasingly employ LiDAR and high-precision map matching for positioning, but these solutions heavily rely on costly sensors and the construction and maintenance of high-precision maps, hindering their widespread application in large-scale autonomous driving scenarios. In contrast, vision-based cross-view matching positioning methods rely solely on low-cost sensors such as cameras, offering advantages such as flexible deployment and strong environmental adaptability, and are gradually developing into an independent source of positioning information used in parallel with GNSS, IMU, and LiDAR. This vision-based method uses a deep learning model to perform cross-view feature matching between ground-view images of the vehicle and satellite overhead maps, thereby estimating the vehicle's position in the satellite map.

[0003] In practical applications, due to the wide coverage and vertical downward perspective of satellite maps, a large number of background areas exist that are irrelevant to the vehicle's horizontal perspective. These include large building rooftops, enclosed open areas, tree canopies, and distant landmarks. Existing end-to-end cross-view localization models often lack this refined ability to discriminate between complex scenes and background interference areas. This makes cross-view matching localization models susceptible to interference from visually similar but geographically inconsistent features, especially in the absence of effective region constraints and attention filtering mechanisms. Consequently, the vehicle may be incorrectly localized to irrelevant background areas, impacting the reliability and robustness of autonomous driving systems in complex urban environments. Summary of the Invention

[0004] According to one aspect of this application, a cross-view localization method for autonomous driving based on cascaded mask guidance is provided, which improves the accuracy and stability of visual matching localization.

[0005] The cross-view localization method for autonomous driving based on cascaded mask guidance includes the following steps: S1. Semantic Feature Context Extraction and Coarse-grained Feature Matching: Obtain ground images captured by vehicle-mounted cameras and satellite map slices covering the corresponding locations. Extract coarse-grained and fine-grained features from the ground images and satellite maps respectively. Based on the coarse-grained features, perform cross-view feature interaction and fusion through a network including a dual-stream encoder and decoder to predict the probability distribution of the vehicle's location in multiple coarse-grained grid regions in the satellite map, and thereby select at least one high-confidence grid region. S2. Spatial Mask Generation and Geometric Constraint Construction: Based on the selected high-confidence grid regions, a corresponding binary spatial mask is generated in the satellite map feature space. The binary spatial mask is used to identify the high-confidence grid regions and the background regions to be masked. The binary spatial mask is then converted into an attention bias matrix to constrain the search range in subsequent feature matching. S3. Mask-guided fine-grained attitude regression: The fine-grained features extracted in S1 and the attention bias matrix obtained in S2 are input into the fine-grained regression network. The attention calculation of the fine-grained features of the satellite map is adjusted using the attention bias matrix and fused with the fine-grained features of the ground image to regress the position offset of the vehicle in the high-confidence grid area.

[0006] Furthermore, in step S1, extracting coarse-grained and fine-grained features from the ground image and satellite map includes: The coarse-grained features are derived from the output of the last layer of the visual backbone network; The fine-grained features are formed by superimposing multiple intermediate layer features of the visual backbone network in the channel dimension.

[0007] Furthermore, in S1, the process of cross-view feature interaction through the dual-stream encoder includes: In each layer of the dual-stream encoder, fixed two-dimensional position codes are added to the coarse-grained features of the ground image and the satellite map, respectively; The enhancement process of coarse-grained features in satellite maps, which enhances their respective features through a self-attention mechanism, can be represented as follows:

[0008] in, Representation layer normalization; This indicates a multi-head self-attention mechanism; For position encoding; These are intermediate feature variables representing the coarse-grained features of the l-th layer satellite map; represents the coarse-grained features of the satellite map at layer l-1; l represents the layer number of the two-stream encoder. L represents the total number of layers in the dual-stream encoder. The bidirectional interaction between the two feature streams is achieved through a cross-attention mechanism. The update process, where coarse-grained features from the satellite map are used as the query and coarse-grained features from the ground image are used as the key and value, is represented as follows:

[0009] in, This is a cross-attention mechanism; ; For querying coarse-grained features of satellite maps; The key to coarse-grained features of ground images; The value of the coarse-grained feature of the ground image; This represents coarse-grained features of satellite maps. Ground image coarse-grained features are updated using satellite map coarse-grained features as keys and values.

[0010] Furthermore, in S1, the fusion process of the decoder specifically includes: Using the coarse-grained features of the satellite map after interaction with the L-layer dual-stream encoder as the query, and the coarse-grained features of the ground image as the key and value, the aggregated features are calculated through a cross-attention mechanism, as follows:

[0011] in, This is an aggregation feature; This represents the coarse-grained features of the Lth layer satellite map; This represents the coarse-grained features of the ground image at layer L; The aggregated features predict the probability distribution of vehicles in multiple coarse-grained grid regions of the satellite map using a classification head.

[0012] Furthermore, in step S2, generating a corresponding binary space mask in the satellite map feature space includes: Based on the probability distribution, the K grid regions with the highest probabilities are selected as the effective regions; In the mask matrix, the elements corresponding to the valid regions are assigned a value of 1, and the elements corresponding to the remaining regions are assigned a value of 0, as follows:

[0013] in, Indicates the first The mask value corresponding to each grid; This represents the output probability value of the coarse-grained classifier. Sort the above probability values ​​in descending order.

[0014] Furthermore, the binary space mask is converted into an attention bias matrix. The rules are:

[0015] in, Let be the binary space mask for the j-th coarse-grained grid.

[0016] Furthermore, the attention calculation process is adjusted using the aforementioned attention bias matrix: Attention score of fine-grained features in satellite maps :

[0017] in, Query the weight matrix; The key weight matrix is ​​a learnable matrix. For feature dimensions; For fine-grained features of satellite maps; Adjusted fine-grained features of satellite maps for:

[0018] in, is a learnable value weight matrix.

[0019] Furthermore, in S3, the fine-grained feature fusion process includes: The query uses the sum of the adjusted satellite map fine-grained features and the location code, the key uses the sum of the ground image fine-grained features and the location code, and the value uses the ground image fine-grained features. Fusion features are obtained through cross-attention mechanism. , is represented as:

[0020] in, For querying fine-grained features of satellite maps; The key to fine-grained features of ground images; The value of the fine-grained feature of the ground image; The fused features, after passing through the feedforward network FFN and residual connections, output the final fine-grained fused features, which are used to regress the refined position offset.

[0021] The beneficial effects that this application can produce include: The cascaded mask-guided cross-view localization method for autonomous driving provided in this application effectively solves the localization drift and multimodal ambiguity problems caused by background noise interference in traditional end-to-end methods. Simultaneously, it employs a coarse-to-fine cascaded mechanism, utilizing both global semantic information for rapid initial screening and local geometric constraints for accurate regression. This approach significantly reduces computational redundancy in non-target regions, improving system efficiency. This method can be used as an independent visual localization tool in autonomous driving systems, integrated with existing GNSS, IMU, and LiDAR localization methods to enhance the reliability and safety of autonomous driving localization systems. Attached Figure Description

[0022] Figure 1 This is a structural block diagram of a cross-view positioning method guided by cascaded masking in one embodiment of this application; Figure 2 This is a flowchart illustrating the implementation of a cross-view positioning method guided by cascaded masks in one embodiment of this application. Figure 3 This is a schematic diagram of a coarse-grained feature matching module in one embodiment of this application; Figure 4 This is a schematic diagram of a fine-grained attitude regression module in one embodiment of this application. Detailed Implementation

[0023] The present application is described in detail below with reference to the embodiments, but the present application is not limited to these embodiments.

[0024] See Figure 1-4 A cross-view localization method for autonomous driving based on cascaded mask guidance is characterized by the following steps: S1. Semantic Feature Context Extraction and Coarse-grained Feature Matching: Obtain ground images captured by vehicle-mounted cameras and satellite map slices covering the corresponding locations. Extract coarse-grained and fine-grained features from the ground images and satellite maps respectively. Based on the coarse-grained features, perform cross-view feature interaction and fusion through a network including a dual-stream encoder and decoder to predict the probability distribution of the vehicle's location in multiple coarse-grained grid regions in the satellite map, and thereby select at least one high-confidence grid region. S2. Spatial Mask Generation and Geometric Constraint Construction: Based on the selected high-confidence grid regions, a corresponding binary spatial mask is generated in the satellite map feature space. The binary spatial mask is used to identify the high-confidence grid regions and the background regions to be masked. The binary spatial mask is then converted into an attention bias matrix to constrain the search range in subsequent feature matching. S3. Mask-guided fine-grained attitude regression: The fine-grained features extracted in S1 and the attention bias matrix obtained in S2 are input into the fine-grained regression network. The attention calculation of the fine-grained features of the satellite map is adjusted using the attention bias matrix and fused with the fine-grained features of the ground image to regress the position offset of the vehicle in the high-confidence grid area.

[0025] Specifically, semantic feature context extraction and coarse-grained feature matching are used to quickly narrow down the vehicle's location range in satellite maps, reducing the complexity of subsequent fine-grained calculations. First, real-time ground images of the vehicle's surroundings, including scene information such as roads, buildings, and signs, are acquired from the vehicle's onboard camera, along with satellite map tiles of the corresponding geographic area provided by a geographic information system. These satellite map tiles include extensive geographic topological information. Then, feature extraction is performed on both types of images, simultaneously acquiring coarse-grained and fine-grained features. Coarse-grained features are used for rapid global matching, while fine-grained features are used for subsequent precise positioning. Based on the extracted coarse-grained features, a Tranformer structure containing a dual-stream encoder and decoder is used to complete cross-view feature interaction and fusion. The dual-stream encoder processes coarse-grained features from ground images and satellite maps respectively, enabling bidirectional information interaction between the two types of features and overcoming the limitations of single-view features. The decoder then fuses the interacted features and outputs the probability distribution of the vehicle in multiple pre-defined coarse-grained grid areas on the satellite map. Based on this probability distribution, at least one high-confidence grid area is selected as the target area for subsequent fine-grained positioning, thus achieving initial convergence of the positioning range.

[0026] Furthermore, spatial masks are used to shield irrelevant background interference, providing clear geometric constraints for subsequent feature matching and improving matching accuracy and efficiency. Based on the selected high-confidence grid regions, corresponding binary spatial masks are generated in the feature space of the satellite map. The binary spatial mask is a matrix that matches the feature dimensions of the satellite map, clearly distinguishing between high-confidence grid regions where vehicles may exist and background regions that do not require attention, thus achieving preliminary filtering of background interference. This binary spatial mask is further converted into an attention bias matrix, which is embedded in the subsequent fine-grained feature matching process. By setting bias parameters to constrain the search range of feature matching, the model is forced to focus its attention on the high-confidence grid regions corresponding to the high-confidence grids, avoiding ineffective searches in broad background regions, reducing the complexity of feature matching, and minimizing the impact of background noise on positioning accuracy.

[0027] Furthermore, under the constructed geometric constraints, the vehicle's specific positional offset within the high-confidence grid area is regressed through deep fusion of fine-grained features, completing the final localization. First, the extracted fine-grained features from the ground image and satellite map are input into the fine-grained regression network along with an attention bias matrix. Then, the network adjusts the attention calculation logic of the satellite map's fine-grained features using the attention bias matrix, assigning higher attention weights to features corresponding to the high-confidence grid area and suppressing features corresponding to background areas, ensuring that feature interactions focus on the effective area. Next, the adjusted satellite map fine-grained features are deeply fused with the ground image fine-grained features, fully utilizing the detailed features of the ground image and the global topological features of the satellite map to compensate for the deficiencies of single-view features. Finally, the regression network outputs the vehicle's positional offset within the high-confidence grid area, which, combined with the grid's geographic coordinates, yields the vehicle's precise absolute coordinates in the satellite map, completing the entire cross-view localization process.

[0028] In step S1, the coarse-grained and fine-grained features of the ground image and satellite map are extracted, including: The coarse-grained features are derived from the output of the last layer of the visual backbone network; The fine-grained features are formed by superimposing multiple intermediate layer features of the visual backbone network in the channel dimension.

[0029] Specifically, both coarse-grained and fine-grained features originate from the same visual backbone network, such as a CNN (Convolutional Neural Network) or a Transformer network. The appropriate network type can be selected based on the actual positioning accuracy requirements. The extraction logic for both is based on the outputs of different network layers. Coarse-grained features are derived from the output of the last layer of the visual backbone network. The deeper the layer of the visual backbone network, the higher the level of abstraction of the output features and the larger the receptive field, effectively capturing the global semantic information of the image while ignoring irrelevant details and noise. Selecting the output of the last layer as coarse-grained features allows for the rapid acquisition of global feature representations for both types of images, meeting the need for rapid narrowing of the positioning range in coarse-grained matching and ensuring the efficiency of global matching. Fine-grained features are formed by superimposing features from multiple intermediate layers of the visual backbone network along the channel dimension. The features output from the intermediate layers of the visual backbone network have a lower level of abstraction and a smaller receptive field, preserving the detailed information of the image. This detailed information is crucial for achieving precise fine-grained positioning. Overlaying multiple intermediate layer features along the channel dimension can integrate detailed features from different levels, enrich the dimensionality and information content of fine-grained features, avoid the problem of insufficient detail in a single intermediate layer feature, provide sufficient detail support for subsequent fine-grained pose regression, and ensure the accuracy of localization.

[0030] In step S1, the process of cross-view feature interaction through the dual-stream encoder includes: In each layer of the dual-stream encoder, fixed two-dimensional position codes are added to the coarse-grained features of the ground image and the satellite map, respectively; The enhancement process of coarse-grained features in satellite maps, which enhances their respective features through a self-attention mechanism, can be represented as follows:

[0031] in, Representation layer normalization; This indicates a multi-head self-attention mechanism; For position encoding; These are intermediate feature variables representing the coarse-grained features of the l-th layer satellite map; represents the coarse-grained features of the satellite map at layer l-1; l represents the layer number of the two-stream encoder. L represents the total number of layers in the dual-stream encoder. The bidirectional interaction between the two feature streams is achieved through a cross-attention mechanism. The update process, where coarse-grained features from the satellite map are used as the query and coarse-grained features from the ground image are used as the key and value, is represented as follows:

[0032] in, This is a cross-attention mechanism; ; For querying coarse-grained features of satellite maps; The key to coarse-grained features of ground images; The value of the coarse-grained feature of the ground image; This represents coarse-grained features of satellite maps. Ground image coarse-grained features are updated using satellite map coarse-grained features as keys and values.

[0033] Specifically, the dual-stream encoder processes coarse-grained features from terrestrial images and satellite maps separately. Through hierarchical feature enhancement and bidirectional interaction, it achieves information complementarity between the two types of features, reducing feature mismatch issues caused by viewpoint differences. In each layer of the dual-stream encoder, fixed two-dimensional positional codes are added to the coarse-grained features of both the terrestrial and satellite maps. These positional codes inject spatial location information into the features. Since the features output by the visual backbone network themselves do not contain location information, and cross-viewpoint localization is based on spatial location matching, adding positional codes clarifies the spatial distribution of features in the image. This ensures that subsequent attention mechanisms can interact with features based on spatial location, avoiding spatial misalignment in feature matching and providing spatial constraints for cross-viewpoint feature matching. After adding positional codes, the multi-head self-attention mechanism (MHSA) individually enhances the coarse-grained features of both the terrestrial and satellite images, improving the expressive power of single-viewpoint features. Taking the satellite map feature enhancement process as an example, the intermediate feature variables of the coarse-grained satellite map features in layer l are obtained by performing multi-head self-attention calculation on the coarse-grained satellite map features in layer l-1, then performing residual connections with the original coarse-grained satellite map features in layer l-1, and finally performing layer normalization (LN) operation. The purpose is to use the self-attention mechanism to allow each feature point of the satellite feature to pay attention to other relevant feature points in the feature map, strengthening the global correlation of the features and filtering noise in the features themselves; layer normalization is used to stabilize network training and avoid feature value shifts; residual connections are used to alleviate the gradient vanishing problem in deep networks and ensure the stability of the feature enhancement process. The self-attention enhancement process of ground image features is completely consistent with that of satellite features, achieving optimization of the ground features themselves. After completing the self-attention enhancement of their respective features, a cross-attention mechanism is used to achieve bidirectional interaction between the ground feature stream and the satellite feature stream, completing the information complementarity of the two types of perspective features. Using one type of feature as the query and another type of feature as the key and value, this application achieves targeted information extraction from both types of features. Specifically, it clarifies the satellite feature update and ground feature update using symmetrical logic: coarse-grained features of the satellite map with added location encoding are used as the query, and coarse-grained features of the ground image with added location encoding are used as the key and value. Interactive features are calculated using a multi-head cross-attention mechanism (MHCA), then residually connected with the original coarse-grained features of the satellite map, and finally updated coarse-grained features of the satellite map are obtained through layer normalization (LN). The definitions of query, key, and value clearly define the source of the features, all based on the superposition of features from layer (l-1) and location encoding, ensuring that the interaction process incorporates spatial location information. Cross-attention allows satellite features to extract information relevant to themselves from ground features, such as the correspondence between ground road layouts and satellite region outlines, compensating for the lack of detail in satellite features.Employing logic symmetrical to satellite feature updates, this method uses coarse-grained features of the ground image as the query and coarse-grained features of the satellite map as the key and value. Through MHCA mechanism, residual connection, and layer normalization, updated coarse-grained features of the ground image are obtained, enabling the extraction of global topological information from satellite features and compensating for the lack of global perspective of ground features.

[0034] Through hierarchical iteration, deep bidirectional interaction of coarse-grained features between ground and satellite maps can be achieved, gradually reducing the domain differences between the two types of perspective features, laying the foundation for feature fusion and probability prediction in subsequent decoders.

[0035] In step S1, the fusion process of the decoder is specifically as follows: Using satellite features obtained through interaction with an L-layer dual-stream encoder as the query and ground features as the key and value, the aggregated features are calculated using a cross-attention mechanism, as follows:

[0036] in, This is an aggregation feature; This represents the coarse-grained features of the Lth layer satellite map; This represents the coarse-grained features of the ground image at layer L; The aggregated features predict the probability distribution of vehicles in multiple coarse-grained grid regions of the satellite map using a classification head.

[0037] Specifically, the decoder further aggregates the coarse-grained features of the ground image and satellite map after the interaction of the L-layer dual-stream encoder to generate aggregated features that can characterize the probability of vehicle location. Finally, the coarse-grained grid region where the vehicle may exist is determined by probability prediction. The coarse-grained features of the satellite map after the interaction of the L-layer dual-stream encoder are used as queries, and the coarse-grained features of the ground image after the interaction of the L-layer are used as keys and values. Feature calculation is performed through a multi-head cross-attention mechanism. Then, the calculation results are residually connected with the original coarse-grained features of the satellite map. Finally, the aggregated features are obtained through layer normalization.

[0038] The calculation of aggregated features relies on coarse-grained features from satellite and ground images in the final L layer, both of which incorporate location encoding to ensure that the aggregation process incorporates spatial location information. The application of a cross-attention mechanism allows satellite features to fully absorb effective detail information from ground features while retaining their own global topological advantages, achieving deep fusion of the two types of features. Compared to single features, aggregated features can more comprehensively and accurately represent vehicle location information, reducing positioning errors caused by viewpoint differences. Furthermore, the design of residual connections and layer normalization, consistent with the encoder, stabilizes the feature aggregation process, preventing feature distortion and training instability.

[0039] After obtaining the aggregated features, they are input into the classification head, which is the output layer of the neural network. Typically, this is done using... The activation function, calculated by the classification head, predicts the probability distribution of a vehicle within multiple pre-defined coarse-grained grid regions on a satellite map. These coarse-grained grid regions are areas pre-divided by gridding satellite map tiles. The grid size can be adjusted according to actual positioning needs; coarse-grained grids are typically larger to quickly narrow down the positioning range. The probability distribution represents the likelihood of a vehicle existing within each grid region.

[0040] It should be noted that the probability distribution is the basis for screening high-confidence grid regions in S1. Subsequently, by sorting and screening the probability distribution, the high-confidence regions where vehicles may exist can be obtained, thus completing the core objective of coarse-grained matching and providing a foundation for spatial mask generation.

[0041] In step S2, generating a corresponding binary space mask in the satellite map feature space includes: Based on the probability distribution, the K grid regions with the highest probabilities are selected as the effective regions; In the mask matrix, the elements corresponding to the valid regions are assigned a value of 1, and the elements corresponding to the remaining regions are assigned a value of 0, as follows:

[0042] in, Indicates the first The mask value corresponding to each grid; This represents the output probability value of the coarse-grained classifier. Sort the above probability values ​​in descending order.

[0043] Specifically, the binary space mask distinguishes between effective and background regions in the feature space of the satellite map. The generation process relies on the vehicle location probability distribution output by the decoder. The probability distribution output by the classification head is sorted according to probability values ​​from high to low. The K grid regions with the highest probabilities after sorting are selected as the Top-K effective regions, where K is a preset hyperparameter that can be adjusted according to actual positioning accuracy and computational complexity requirements. A larger K value results in more effective regions and higher computational complexity, but a lower false negative rate. A smaller K value results in fewer effective regions and lower computational complexity, but may pose a risk of false negatives. Filtering effective regions further narrows down the possible range of vehicles, ensuring the targeted nature of subsequent mask guidance. A mask matrix with the same dimension as the satellite map feature space is constructed. Each element in the mask matrix corresponds to a coarse-grained grid region in the satellite map, and its value is determined by a preset rule, explicitly stated mathematically: if a grid region i belongs to the Top-K effective regions selected above (i∈Top-K), then the mask element corresponding to that grid... A value of 1 is assigned to indicate that the area is a high-confidence grid area that needs to be focused on; if a certain grid area i does not belong to the Top-K valid area, the mask element Mi corresponding to the grid is assigned a value of 0, indicating that the area is a background area that needs to be masked.

[0044] The binary space mask generated in this way can clearly and intuitively divide the effective and invalid regions in the feature space of satellite maps, achieving preliminary filtering of background interference. At the same time, its simple binary values ​​facilitate subsequent conversion into attention bias matrices, reducing computational complexity.

[0045] The binary space mask is converted into an attention bias matrix. The rules are:

[0046] in, For the binary mask of the j-th coarse-grained grid, it is worth noting that, although in the aforementioned steps... Indicates the first A mask for each grid, but in the attention calculation of this step, in order to index the satellite features. Keep it in correspondence, remember For the first The mask value of each satellite grid.

[0047] Specifically, the attention bias matrix transforms the filtering constraints of the binary space mask into weight constraints for attention calculation, guiding the subsequent fine-grained regression network to focus attention on high-confidence grid regions. The elements in the attention bias matrix correspond to the attention bias value of the j-th feature point of the satellite map's fine-grained features during the fine-grained feature matching process, and its value is determined by the elements of the binary space mask. The value of is determined by , if If the j-th grid region is a high-confidence grid region and belongs to the Top-K effective regions, then the corresponding attention bias value is... The value is assigned to 0. Therefore, no suppression is imposed on feature interactions in high-confidence grid regions, allowing the first feature of fine-grained features in high-confidence grid regions of satellite maps to be enabled. Normal attention calculations are performed on each feature point to ensure that features in high-confidence grid regions can fully interact and merge. If That is, the first If a grid region is considered a background region and does not belong to the Top-K valid regions, then the corresponding attention bias value is... Assigned value Therefore, strong suppression is applied to feature interactions in the background region, and this is incorporated into the subsequent attention score calculation. After the bias value, The activation function calculation causes the attention weights corresponding to the background region to approach 0, thereby forcing the model to ignore the features of the background region, avoiding interference from background noise on fine-grained feature matching, and ensuring that the model's attention is always focused on the high-confidence grid region.

[0048] By transforming the rules, the filtering logic of the binary space mask was successfully converted into the constraint logic of attention computation, realizing the connection from coarse-grained region filtering to fine-grained feature guidance. This not only reduced the computational complexity of fine-grained feature matching but also improved the accuracy of feature matching, providing key constraints for fine-grained pose regression.

[0049] The process of adjusting attention calculation using the aforementioned attention bias matrix: Attention score of fine-grained features in satellite maps :

[0050] in, Query the weight matrix; The key weight matrix is ​​a learnable matrix. For feature dimensions; For fine-grained features of satellite maps; Adjusted fine-grained features of satellite maps for:

[0051] in, is a learnable value weight matrix.

[0052] Specifically, the attention bias matrix adjusts the attention calculation of fine-grained features in satellite maps. By setting the bias value, it optimizes the distribution of attention weights within the fine-grained features, focusing on high-confidence grid areas and suppressing background areas. The attention score measures the correlation between different feature points in the fine-grained features of the satellite map. Its calculation logic is clearly defined through mathematical expressions. The fine-grained features of the satellite map are multiplied by the learnable query weight matrix and key weight matrix to obtain the query features and key features of the fine-grained features. The weight matrix and key matrix are adaptively learnable parameters during model training, used to project the fine-grained features of the satellite map onto a feature space suitable for attention calculation, improving the effectiveness of attention calculation. Matrix multiplication is then performed on the transposes of the query features and key features to obtain the original similarity between feature points. Subsequently, this original similarity is divided by the square root of the feature dimension for scaling, mitigating the problem of excessively high feature dimensions leading to excessively large original similarity values. To address the vanishing gradient problem after activation, the stability of attention score calculation is ensured. The scaled similarity is added to the attention bias matrix to inject the attention bias; finally, through... The activation function normalizes the addition result to obtain the final attention score. At this point, the attention score corresponding to the background region will be affected by... The bias value approaches 0, and the attention score corresponding to the high-confidence grid area is retained and normalized to optimize the attention weight. The calculated attention score is multiplied by the product of the satellite map fine-grained feature and the learnable value weight matrix to obtain the adjusted satellite map fine-grained feature. The value weight matrix is ​​consistent with the query weight matrix and the key weight matrix, and are all learnable parameters used to optimize the projection of the value features of the satellite map fine-grained feature. The multiplication operation of the attention score and the value feature distributes the optimized attention weight to each feature point of the satellite map fine-grained feature, strengthens the expression of the features of the high-confidence grid area, suppresses the interference of the background area features, and makes the adjusted satellite map fine-grained feature more targeted, laying the foundation for subsequent fusion with the ground image fine-grained feature and position offset regression.

[0053] In S3, the fine-grained feature fusion process includes: The query uses the sum of the adjusted satellite map fine-grained features and the location code, the key uses the sum of the ground image fine-grained features and the location code, and the value uses the ground image fine-grained features. Fusion features are obtained through cross-attention mechanism. , is represented as:

[0054] in, For querying fine-grained features of satellite maps; The key to fine-grained features of ground images; The value of the fine-grained feature of the ground image; The fused features, after passing through the feedforward network FFN and residual connections, output the final fine-grained fused features, which are used to regress the refined position offset.

[0055] Specifically, fine-grained feature fusion deeply integrates adjusted satellite map fine-grained features with ground image fine-grained features, combining the global topological advantages of satellite features with the detailed advantages of ground features to generate fused features that accurately represent vehicle positions, providing support for position offset regression. Specifically, the sources of the query, key, and value in the cross-attention mechanism are clearly defined to ensure that the fusion process incorporates spatial location information: the query consists of the sum of adjusted satellite map fine-grained features and location codes; the purpose of adding location codes is to inject spatial location information, ensuring that feature fusion can match based on spatial location and avoiding spatial misalignment during the fusion process. The key consists of the sum of ground image fine-grained features and location codes, consistent with the query's location code, ensuring spatial consistency of key-value matching. The value directly uses ground image fine-grained features; the core purpose is to inject the detailed information from ground image fine-grained features into the fused features, compensating for the lack of detail in satellite map fine-grained features. The prepared queries, keys, and values ​​are input into a multi-head cross-attention mechanism for feature interaction calculation. Subsequently, the calculated interaction features are residually connected with the adjusted fine-grained features of the satellite map. Finally, layer normalization (LN) is applied to obtain the preliminary fused features. This application clarifies the correlation of each parameter and the design of residual connections and layer normalization to stabilize the fusion process, avoid feature distortion, and ensure that the fused features simultaneously retain the global advantages of satellite features and the detailed advantages of ground features. The preliminary fused features are input into a feedforward network, and the nonlinear transformation of FFN further optimizes the expressive power of the fused features and filters out noise generated during the fusion process. Then, the output of FFN is residually connected with the preliminary fused features to complete the final feature optimization and output the final fine-grained fused features. The final fine-grained fused features are input into a fine-grained regression network to regress the refined positional offset of the vehicle within the selected high-confidence grid area. Combined with the geographic coordinates of the high-confidence grid area itself, the precise absolute coordinates of the vehicle in the satellite map can be calculated, completing the entire cross-view positioning process and ensuring that the positioning accuracy meets the actual needs of autonomous driving.

[0056] Example: In one embodiment of the present invention, vehicle localization can be achieved in complex urban environments based on an autonomous driving test vehicle equipped with a high-performance computing platform, combined with the proposed cascaded mask-guided cross-view localization method.

[0057] In this embodiment, the vehicle-mounted visual image and satellite map of a real urban canyon driving scenario are selected as input. The specific scenario is as follows: the vehicle is driving on a main urban road with tall buildings on both sides. The image captured by the vehicle-mounted forward-looking camera includes the road extension line, the facades of commercial buildings on both sides and the roadside trees. Due to the multipath effect, the vehicle-mounted GNSS signal has severe drift, with an error of more than 10m. The system simultaneously acquires a satellite map within a 91.2×91.2m range around the current location. The resolution of this satellite map is 800×800 pixels.

[0058] This embodiment first divides the satellite map into a 19×19 coarse-grained grid (361 grids in total), with each coarse-grained grid corresponding to a 14×14 pixel resolution area in the satellite map. Then, a pre-trained DINOv2 (ViT-B) is used as the feature extraction model to extract semantic features and contextual information from the vehicle-mounted visual image (ground image) and the satellite map. The coarse-grained features are taken from the output of the last layer of the DINOv2 (ViT-B) backbone network; the fine-grained features are taken from the output features of the middle layers (layers 5, 6, 7, and 8) of the model, and these four layers of features are then concatenated. Based on the method described in step S1, the ground image features and satellite map features are fused to calculate the positioning probability of each grid in the coarse positioning stage. Finally, the coordinates of the top 64 grids by probability (e.g., [Row: 4, Col: 3], [Row: 5, Col: 6], [Row: 8, Col: 10], etc.) are output. These grid coordinates are the target areas for subsequent fine-grained matching.

[0059] Based on the coarse localization results described above, this embodiment constructs explicit spatial geometric constraints to generate a corresponding binary space mask. The specific parameters of this mask matrix are as follows: the mask dimension is 19×19, consistent with the size of the satellite feature map; the active region (value=1) corresponds to the pixel area of ​​the first 64 grids (such as [Row: 4, Col: 3], [Row: 5, Col: 6], etc.) of the coarse localization output mapped onto the satellite feature map, and this region covers the high probability range of the vehicle's true physical location; the masked region (value=0) consists of all pixels outside the active region, including interference areas such as parallel auxiliary roads that are highly similar to the target road texture but have incorrect geographical locations.

[0060] This embodiment injects the generated binary space mask into a fine-grained regression network. Through a mask-guided cross-attention mechanism, it achieves deep interaction and computation of features between ground images and satellite maps. During this process, relying on the constraints of the network used in step S3, the model is forced to ignore background noise masked by the mask and focus only on effective features within the activation region. Finally, the network outputs refined positioning parameters, such as local normalized offsets: ∆x=0.42, ∇y=0.68 (meaning the vehicle is located 42% to the right and 68% down from the top left corner of the target grid). Combining the coarse positioning grid coordinates with this local normalized offset, the vehicle's pixel position in the satellite map can be calculated. An example calculation is as follows: x,y=(4+0.42×14, 3+0.68×14); then, by converting the satellite map pixel coordinates to the actual GPS coordinates, the vehicle's precise latitude and longitude position in the world coordinate system can be obtained.

[0061] The cascaded mask-guided cross-view localization method proposed in this application effectively solves the interference of background noise on visual feature matching in complex urban environments. Simultaneously, it overcomes the spatial ambiguity problem in decentralized scenarios through a coarse-to-fine cascaded strategy. This invention generates a spatial mask to explicitly shield background areas unrelated to vehicle position, forcing the model to interact with features only within a high-confidence grid range, significantly improving the accuracy of visual localization. Furthermore, this mask-guided mechanism transforms the global search problem into a fine-grained calibration of local regions, greatly reducing redundant computation in invalid areas and achieving sub-meter-level vehicle localization while reducing computational power consumption. This method significantly enhances the perception robustness of autonomous driving systems in satellite signal rejection scenarios (such as urban canyons and tunnel entrances), providing solid technical support for the safe navigation of autonomous vehicles.

[0062] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A cross-view localization method for autonomous driving based on cascaded mask guidance, characterized in that, Includes the following steps: S1. Semantic Feature Context Extraction and Coarse-grained Feature Matching: Obtain ground images captured by vehicle-mounted cameras and satellite map slices covering the corresponding locations. Extract coarse-grained and fine-grained features from the ground images and satellite maps respectively. Based on the coarse-grained features, perform cross-view feature interaction and fusion through a network including a dual-stream encoder and decoder to predict the probability distribution of the vehicle's location in multiple coarse-grained grid regions in the satellite map, and thereby select at least one high-confidence grid region. S2. Spatial Mask Generation and Geometric Constraint Construction: Based on the selected high-confidence grid regions, a corresponding binary spatial mask is generated in the satellite map feature space. The binary spatial mask is used to identify the high-confidence grid regions and the background regions to be masked. The binary spatial mask is then converted into an attention bias matrix to constrain the search range in subsequent feature matching. S3. Mask-guided fine-grained attitude regression: The fine-grained features extracted in S1 and the attention bias matrix obtained in S2 are input into the fine-grained regression network. The attention calculation of the fine-grained features of the satellite map is adjusted using the attention bias matrix and fused with the fine-grained features of the ground image to regress the position offset of the vehicle in the high-confidence grid area.

2. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 1, characterized in that, In step S1, the coarse-grained and fine-grained features of the ground image and satellite map are extracted, including: The coarse-grained features are derived from the output of the last layer of the visual backbone network; The fine-grained features are formed by superimposing multiple intermediate layer features of the visual backbone network in the channel dimension.

3. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 1, characterized in that, In step S1, the process of cross-view feature interaction through the dual-stream encoder includes: In each layer of the dual-stream encoder, fixed two-dimensional position codes are added to the coarse-grained features of the ground image and the satellite map, respectively; The coarse-grained feature enhancement process of satellite maps, which enhances each feature through a self-attention mechanism, can be represented as follows: ； in, Representation layer normalization; This indicates a multi-head self-attention mechanism; For position encoding; These are intermediate feature variables representing the coarse-grained features of the l-th layer satellite map; represents the coarse-grained features of the satellite map at layer l-1; l represents the layer number of the two-stream encoder. L represents the total number of layers in the dual-stream encoder. The bidirectional interaction between the two feature streams is achieved through a cross-attention mechanism. The update process, where coarse-grained features from the satellite map are used as the query and coarse-grained features from the ground image are used as the key and value, is represented as follows: ； in, This is a cross-attention mechanism; ; For querying coarse-grained features of satellite maps; The key to the granular features of the ground image; These are coarse-grained feature values ​​for ground images; This represents the coarse-grained features of the satellite map at layer l. Ground image coarse-grained features are updated using satellite map coarse-grained features as keys and values.

4. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 3, characterized in that, In step S1, the fusion process of the dual-stream decoder is specifically as follows: Using the coarse-grained features of the satellite map after interaction with the L-layer dual-stream encoder as the query, and the coarse-grained features of the ground image as the key and value, the aggregated features are calculated through a cross-attention mechanism, as follows: ； in, This is an aggregation feature; This represents the coarse-grained features of the Lth layer satellite map; This represents the coarse-grained features of the ground image at layer L; The aggregated features predict the probability distribution of vehicles in multiple coarse-grained grid regions of the satellite map using a classification head.

5. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 1, characterized in that, In step S2, generating a corresponding binary space mask in the satellite map feature space includes: Based on the probability distribution, the K grid regions with the highest probabilities are selected as the effective regions; In the mask matrix, the elements corresponding to the valid regions are assigned a value of 1, and the elements corresponding to the remaining regions are assigned a value of 0, as follows: ； in, Indicates the first The mask value corresponding to each grid; This represents the output probability value of the coarse-grained classifier. Sort the above probability values ​​in descending order.

6. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 5, characterized in that, The binary space mask is converted into an attention bias matrix. The rules are: ； in, Let be the binary space mask for the j-th coarse-grained grid.

7. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 1, characterized in that, The process of adjusting attention calculation using the aforementioned attention bias matrix: Attention score of fine-grained features in satellite maps : ； in, This is a learnable query weight matrix; The key weight matrix is ​​a learnable matrix. For feature dimensions; For fine-grained features of satellite maps; Adjusted fine-grained features of satellite maps for: ； in, is a learnable value weight matrix.

8. The autonomous driving cross-view localization method based on cascaded mask guidance according to claim 7, characterized in that, In S3, the fine-grained feature fusion process includes: The query uses the sum of the adjusted satellite map fine-grained features and the location code, the key uses the sum of the ground image fine-grained features and the location code, and the value uses the ground image fine-grained features. Fusion features are obtained through cross-attention mechanism. , is represented as: ； in, For querying fine-grained features of satellite maps; The key to fine-grained features of ground images; The value of the fine-grained feature of the ground image; The fused features, after passing through the feedforward network FFN and residual connections, output the final fine-grained fused features, which are used to regress the refined position offset.

Images