Air-ground collaborative topology understanding method and device based on latent world model

Abstract

The application provides a kind of air-ground cooperation topology understanding method and device based on latent world model, it is related to the technical field of computer vision, the method generates purified prior BEV feature by comparing prior and real-time BEV feature, and generates time series enhanced query parameters and features using latent world model combined with time series information;Then, the multi-source BEV feature is fused through the online topology reasoning model, and the time series or single-frame topology reasoning is carried out, and the final lane-level topology relationship is selected according to the pose reliability, which solves the technical problems of high misjudgment rate, low accuracy and low efficiency of the prior art, and achieves the technical effect of improving the accuracy and robustness of topology understanding in complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular, to an air-ground collaborative topology understanding method and device based on a latent world model. BACKGROUND

[0002] With the rapid development of computer vision and robotics, map topology understanding plays an increasingly important role in autonomous driving, indoor navigation, augmented reality and other fields. Existing map topology understanding methods are mainly based on environmental information obtained by visual sensors, which detect and analyze key points, line segments or regions in the scene, and then infer the topology relationship in space. In the prior art, monocular vision-based methods reconstruct the environment structure through 2D image sequences, but are easily limited by changes in viewing angle; stereo vision or multi-sensor fusion-based methods can provide more rich depth information, but still have difficulty in dealing with occlusion problems in complex scenes. In addition, deep learning-based methods predict topology relationships by training a large amount of labeled data, which improves accuracy to some extent, but has high computational complexity and is difficult to meet real-time requirements.

[0003] Current mainstream map topology understanding systems usually adopt a hierarchical processing architecture: feature extraction and matching at the bottom layer, local topology relationship construction at the middle layer, and global map integration at the high layer. However, this processing method is prone to misjudgment in dynamic environments or partial occlusion, leading to deviations in topology relationship reasoning. At the same time, the system often needs to balance between processing efficiency and reasoning accuracy, making it difficult for existing methods to meet real-time application requirements and accuracy requirements in complex scenarios at the same time. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an air-ground collaborative topology understanding method and device based on a latent world model, to solve the technical problems of high misjudgment rate, low accuracy and low efficiency in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for understanding air-ground cooperative topology based on a latent world model, the method comprising: obtaining prior BEV features generated based on air-ground cooperative information, and real-time BEV features generated based on multi-view images; generating purified prior BEV features based on the prior BEV features and the real-time BEV features; generating time-series enhanced query parameters and time-series enhanced BEV features of a current frame by using the latent world model based on query parameters of a neighboring previous frame, the real-time BEV features, and relative poses between adjacent frames; fusing the time-series enhanced BEV features, the real-time BEV features, and the purified prior BEV features through an encoder of an online topology reasoning model to obtain fused BEV features; performing topology reasoning with time-series information based on the time-series enhanced query parameters, initialization query parameters of the model, and the fused BEV features by using a decoder of the online topology reasoning model to generate a first reasoning result; or performing single-frame topology reasoning based on the initialization query parameters, the real-time BEV features, and the purified prior BEV features to generate a second reasoning result; and selecting between the first reasoning result and the second reasoning result according to reliability of current pose information to determine a final lane-level topology relationship.

[0006] In some optional implementations, the air-ground cooperative information comprises a standard resolution map and aerial photographs, and obtaining the prior BEV features generated based on the air-ground cooperative information comprises: obtaining the standard resolution map and the aerial photographs of a target region, and generating the prior BEV features by using a map prior extraction model.

[0007] In some optional implementations, the encoder of the online topology reasoning model comprises a first fast system and a first slow system, the first fast system is configured to process the multi-view images collected in real time to generate the real-time BEV features, and the first slow system is configured to generate the time-series enhanced query parameters and the time-series enhanced BEV features of the current frame.

[0008] In some optional implementations, the latent world model comprises a query world model and a BEV world model, and generating the time-series enhanced query parameters and the time-series enhanced BEV features of the current frame by using the latent world model comprises: the first slow system generates an action latent variable based on the relative poses between adjacent frames; the query world model is used to generate the time-series enhanced query parameters of the current frame based on the query parameters of the neighboring previous frame and the action latent variable; and the BEV world model is used to generate the time-series enhanced BEV features based on the real-time BEV features of the neighboring previous frame and the action latent variable.

[0009] In some optional implementations, based on the prior BEV feature and the real-time BEV feature, the step of generating the purified prior BEV feature includes: processing the prior BEV feature and the real-time BEV feature through a dynamic prior weight network to generate the purified prior BEV feature; the dynamic prior weight network includes: a feature splicing module, a conflict area detection module, a dynamic weight calculation module, and a weighted fusion module; the feature splicing module is configured to splice the prior BEV feature and the real-time BEV feature to generate a feature with fusion conflict information; the conflict area detection module is configured to extract and reduce dimensions of the feature with fusion conflict information to generate a conflict confidence map; the dynamic weight calculation module is configured to determine a dynamic prior weight map according to the conflict confidence map; and the weighted fusion module is configured to multiply the prior BEV feature and the dynamic prior weight map element by element to generate the purified prior BEV feature.

[0010] In some optional implementations, the decoder of the online topology inference model includes a first TransFormer layer, a second slow system, and a second fast system; the first TransFormer layer is configured to generate a current query parameter based on the initial query parameter, the real-time BEV feature, and the purified prior BEV feature; the second slow system includes a query parameter update module, a plurality of second TransFormer layers, and a first topology inference head; the query parameter update module is configured to update the current query parameter to generate a timing-enhanced query parameter; the plurality of second TransFormer layers are configured to generate a first target query parameter based on the timing-enhanced query parameter and the fused BEV feature; and the first topology inference head is configured to generate a first inference result based on the first target query parameter and model parameters shared with the second fast system.

[0011] In some optional implementations, the second fast system includes a plurality of third TransFormer layers and a second topology inference head; the plurality of third TransFormer layers are configured to generate a second target query parameter based on the initial query parameter and the fused BEV feature; the second topology inference head is configured to generate a second inference result based on the second target query parameter and model parameters shared with the second slow system; and the selection between the first inference result and the second inference result according to the reliability of the current pose information includes: when the current pose information exists and the reliability is greater than a reliability threshold, selecting the first inference result as the final lane-level topology relationship; and when the current pose information exists and the reliability is less than the reliability threshold, or is missing, selecting the second inference result as the final lane-level topology relationship.

[0012] Secondly, embodiments of the present invention provide an air-ground cooperative topology understanding device based on a latent world model. The device includes: a feature acquisition module, configured to acquire prior BEV features generated based on air-ground cooperative information and real-time BEV features generated based on multi-view images; and to generate purified prior BEV features based on the prior BEV features and the real-time BEV features; a first generation module, configured to generate temporally enhanced query parameters and temporally enhanced BEV features for the current frame using a latent world model, based on query parameters of the adjacent previous frame, the real-time BEV features, and the relative pose between adjacent frames; and a feature fusion module, configured to fuse the temporally enhanced BEV features and the real-time BEV features with... The aforementioned prior BEV features are fused using the encoder of the online topology inference model to obtain fused BEV features. The second generation module is used to perform topology inference with time-series information introduced by the decoder of the online topology inference model, based on the aforementioned time-series enhanced query parameters, the model's initialization query parameters, and the aforementioned fused BEV features, to generate a first inference result; or, based on the aforementioned initialization query parameters, the aforementioned real-time BEV features, and the aforementioned prior BEV features, to perform single-frame topology inference and generate a second inference result. The selection module is used to select from the first and second inference results based on the reliability of the current pose information to determine the final lane-level topology relationship.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0015] This invention provides a method and apparatus for air-ground cooperative topology understanding based on a latent world model, relating to the technical field of computer vision. The method includes: first, acquiring prior BEV features generated based on air-ground cooperative information and real-time BEV features generated based on multi-view images; generating cleaned-up prior BEV features based on the prior BEV features and real-time BEV features; then, generating temporally enhanced query parameters and temporally enhanced BEV features for the current frame using a latent world model based on query parameters from the previous frame, real-time BEV features, and the relative pose between adjacent frames; fusing the temporally enhanced BEV features, real-time BEV features, and cleaned-up prior BEV features through an encoder of an online topology inference model to obtain fused BEV features; performing topology inference incorporating temporal information based on the temporally enhanced query parameters, the model's initialization query parameters, and the fused BEV features through a decoder of the online topology inference model to generate a first inference result; or, performing single-frame topology inference based on the initialization query parameters, real-time BEV features, and cleaned-up prior BEV features to generate a second inference result; and selecting from the first and second inference results based on the reliability of the current pose information to determine the final lane-level topology relationship. The above method solves the technical problems of high misjudgment rate, low accuracy and efficiency of existing technologies, and achieves the technical effect of improving the accuracy and robustness of topology understanding in complex environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an air-ground collaborative topology understanding method based on a latent world model, provided as an embodiment of the present invention;

[0018] Figure 2 A schematic diagram illustrating the application of an air-ground collaborative topology understanding method based on a latent world model, provided in an embodiment of the present invention.

[0019] Figure 3 A schematic diagram of the structure of an encoder for an online topology inference model provided in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the structure of a decoder for an online topological inference model provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] In practical applications of map topology understanding, existing technologies primarily employ the following methods: First, trajectory data of target vehicles within an intersection area is acquired, comprising a set of trajectory points generated during vehicle movement. Then, based on the directional characteristics of each lane's trajectory lines, cluster analysis is performed on trajectory points with similar characteristics within the vehicle trajectory point set to extract intermediate trajectory data representing the center lines of each lane. Next, the boundary line positions of each lane are deduced based on the lane center line data. Finally, using the lane boundary lines as constraints, the vehicle trajectories of each lane are processed according to standard driving behavior guidelines to generate lane driving paths that conform to actual driving habits, thereby establishing the topological relationship between lane lines and driving paths in the intersection area. However, this method still has significant technical limitations in key performance indicators such as false positive rate, accuracy, and computational efficiency.

[0023] Based on this, the present invention provides a method and apparatus for air-ground cooperative topology understanding based on a potential world model to solve the above problems.

[0024] To facilitate understanding of this embodiment, a detailed description of the air-ground cooperative topology understanding method based on a latent world model disclosed in this invention will be provided first. (See [link to relevant documentation]). Figure 1 The diagram shows a flowchart of an air-ground cooperative topology understanding method based on a latent world model. This method can be executed by an electronic device and mainly includes the following steps S102 to S110:

[0025] Step S102: Obtain prior BEV features generated based on air-ground collaborative information and real-time BEV features generated based on multi-view images; generate cleaned prior BEV features based on prior BEV features and real-time BEV features.

[0026] In one embodiment, the air-ground cooperative information may include: a standard resolution map and drone aerial images; further, obtaining prior BEV features generated based on the air-ground cooperative information may include: obtaining a standard resolution map and drone aerial images of the target area, and generating prior BEV features through a map prior extraction model.

[0027] The aforementioned map prior extraction model can take a standard resolution map (SD map) and drone aerial images as input, and output map prior BEV features from the drone's perspective. These features include both the high-precision topological structure of the SD map and the fine-grained texture and boundary information of the drone imagery, thereby compensating for the perception blind spots when the field of view of the vehicle-mounted sensors is limited or GPS fails.

[0028] The prior BEV features output by the aforementioned map prior extraction model This is a form of static knowledge. However, the real road environment is dynamically changing, and static prior maps inevitably conflict with the dynamic scene perceived in real time. For example, the prior map shows that the road is unobstructed, but real-time perception reveals that the road is closed by traffic cones due to construction. To solve this problem, this embodiment of the invention introduces a dynamic prior weight network based on conflict perception.

[0029] In one embodiment, the step of generating cleaned prior BEV features based on prior BEV features and real-time BEV features includes: processing the prior BEV features and real-time BEV features through a dynamic prior weight network to generate cleaned prior BEV features.

[0030] The dynamic prior weight network includes: a feature concatenation module, a conflict region detection module, a dynamic weight calculation module, and a weighted fusion module. The feature concatenation module is used to concatenate prior BEV features and real-time BEV features to generate features that fuse conflict information. The conflict region detection module is used to extract features and reduce the dimensionality of the features that fuse conflict information to generate a conflict confidence map. The dynamic weight calculation module is used to determine the dynamic prior weight map based on the conflict confidence map. The weighted fusion module is used to multiply the prior BEV features and the dynamic prior weight map element-wise to generate cleaned prior BEV features.

[0031] This dynamic prior weight network uses real-time BEV features and prior BEV features As input, its core task is to automatically evaluate the consistency between prior information and real-time sensed information at each spatial location (BEV grid cell) and generate a dynamic prior weight map. This is used to determine the degree of prior trust or real-time trust awareness in subsequent fusion processes.

[0032] The specific structure and workflow of the network are as follows:

[0033] (1) Input and feature concatenation: real-time BEV features and prior BEV features By splicing the data along the channel dimension, features that integrate conflicting information are obtained.

[0034]

[0035] (2) Conflict area detection: Features that integrate conflict information The input is a lightweight convolutional encoder (consisting of several convolutional layers and a ReLU activation function), which progressively extracts features and reduces dimensionality, and finally outputs an initial conflict confidence map through a Sigmoid activation function. Conflict confidence plot The value of each pixel is between 0 and 1. The closer the value is to 1, the greater the conflict between the prior knowledge and real-time perception at that location.

[0036] (3) Dynamic weight calculation: conflict confidence plot It cannot be directly used as a weight; instead, its opposite—the weight of trusting priors—is used. Therefore, it is calculated... The final dynamic prior weight graph is obtained. The closer the value is to 1, the more the system trusts the prior map; the closer the value is to 0, the more the system trusts real-time perception.

[0037] (4) Weighted fusion: The "purified" prior BEV features that are finally sent to the subsequent fusion module. Prior BEV features output by the aforementioned map prior extraction model from the original prior features. It is obtained by multiplying the dynamic prior weight graph element by element.

[0038]

[0039] in This indicates element-wise multiplication. This operation is equivalent to multiplying elements in the conflict region (low-order multiplication). The value suppresses or even erases the influence of prior features, forcing the model to rely more on real-time perception; in consistent regions (high... The value retains strong prior knowledge to complete the occlusion and improve accuracy.

[0040] This network requires no additional manually labeled data and is trained through a self-supervised approach. During training, "conflicts" are intentionally created in the input data; for example, a subset of training samples is randomly selected and their prior features are... The prior features are replaced with those of a completely different path, thus creating a "pseudo-conflict." The training objective is to make the network output... The value is high (close to 1) in regions where conflict occurs, and low (close to 0) in regions where no conflict occurs. The loss function used is the binary cross-entropy loss, which is mathematically defined as follows:

[0041]

[0042] in, Confidence plot for conflict The total number of pixels, where P is the pixel index. This is the network's predicted value at pixel P. This is the true binary collision label corresponding to this pixel.

[0043] Real Conflict Tags The data can be obtained without manual annotation. During training, it can be automatically generated based on the following rules: For training samples selected to generate "pseudo-conflicts," the true labels of their corresponding regions are obtained. Set the label to 1 (indicating conflict), and set the remaining regions to 0 (indicating consistency); for normal training samples that do not generate conflicts, set the label to 1 for all locations. Set to 0 (meaning that prior knowledge and perception should be consistent).

[0044] Next, based on the features obtained in the above steps, topological reasoning is performed using an online topological reasoning model to determine the final lane-level topological relationships.

[0045] In one embodiment, the encoder of the online topology inference model includes a first fast system and a first slow system; the first fast system is used to process multi-view images acquired in real time to generate real-time BEV features; the first slow system is used to generate temporal enhancement query parameters and temporal enhancement BEV features for the current frame.

[0046] The decoder of the online topological inference model consists of a second slow system and a second fast system. In other words, the online topological inference model comprises two parts: an encoder and a decoder, both of which are composed of fast and slow dual systems. The slow system uses historical information and is driven by the world model; the fast system performs single-frame inference, ensuring robustness when there is no historical input.

[0047] Step S104: Based on the query parameters of the adjacent previous frame, the real-time BEV features, and the relative pose between adjacent frames, the latent world model is used to generate the temporal augmentation query parameters and temporal augmentation BEV features of the current frame.

[0048] In one embodiment, the latent world model includes a query world model and a BEV world model; generating temporally enhanced query parameters and temporally enhanced BEV features for the current frame using the latent world model includes: a first slow system generating action latent variables based on the relative poses between adjacent frames; generating temporally enhanced query parameters for the current frame using the query world model based on the query parameters and action latent variables of the adjacent previous frame; and generating temporally enhanced BEV features using the BEV world model based on the real-time BEV features and action latent variables of the adjacent previous frame.

[0049] As a concrete example, the encoder of an online topological inference model consists of a fast and a slow dual system. (Combined)Figure 3 As shown, the input to the encoder fast system is multi-view images collected in real time by the vehicle-mounted camera (without incorporating historical information). After processing by a PV-to-BEV encoder, the corresponding real-time BEV features are output, i.e., the BEV features of the current frame (T). Here, H, W, and C represent height, width, and number of channels, respectively. In addition, the encoder's fast system includes an initialization query from the learnable embedding space. N represents the number of queries.

[0050] The encoder's slow system is activated when a reliable pose is available. It uses the detection results and relative pose of the previous frame (T-1) to transfer the historical state to the current frame (T) through the "latent world model", thereby generating more accurate temporal enhancement queries and temporal enhancement BEV features for the current frame, which are then used by the subsequent decoder.

[0051] Specifically, the relative pose (translation t, rotation) between adjacent frames After being flattened, the action latent variables are obtained through MLP processing. :

[0052]

[0053] Query the world model in action latent variables Under the condition of querying historical data, such as the query of the previous frame (T-1), Perform the transformation to obtain the current frame (T) time-series enhancement query. BEV world model in action latent variables BEV features of the previous frame (T-1) under the condition Perform the transformation to obtain the temporal enhanced BEV features of the current frame (T). ,in The world model is composed of Transformer blocks, which are copied and concatenated along the channel dimension with the query and BEV features. These blocks contain self-focused modules and feedforward modules.

[0054]

[0055] Furthermore, the encoder uses the BEV feature fusion module to integrate the cleaned prior BEV features (corresponding to) generated in step S102 above. Figure 3 The prior BEV features of the clean map (in the clean map) and the features output by the fast system and the slow system are fused to generate fused BEV features. The fusion process will be described in detail below.

[0056] Step S106: The time-enhanced BEV features, real-time BEV features, and cleaned-up prior BEV features are fused by the encoder of the online topology inference model to obtain fused BEV features.

[0057] In addition to the fast and slow dual systems, the encoder of the online topology inference model also includes a BEV feature fusion module. This BEV feature fusion module can use a gated recursive unit to temporally enhance the BEV features of the current frame (T) output by the slow system. The current frame (T) BEV features output in real time by the aforementioned fast system And the cleaned map prior BEV features output after processing by a conflict-aware dynamic prior weight network from the map prior BEV features. By performing fusion, the fused BEV features are obtained. This is for use by subsequent decoders.

[0058] Step S108: Using the decoder of the online topology reasoning model, perform topology reasoning that incorporates time-series information based on the time-enhanced query parameters, the model's initial query parameters, and fused BEV features to generate a first reasoning result; or, perform single-frame topology reasoning based on the initial query parameters, real-time BEV features, and cleaned-up prior BEV features to generate a second reasoning result.

[0059] In one embodiment, the decoder of the online topology inference model includes a first Transformer layer, a second slow system, and a second fast system. The first Transformer layer generates current query parameters based on initial query parameters, real-time BEV features, and cleaned-up prior BEV features. The second slow system includes a query parameter update module, several second Transformer layers, and a first topology inference head. The query parameter update module updates the current query parameters to generate time-enhanced query parameters. Several second Transformer layers generate a first target query parameter based on the time-enhanced query parameter and fused BEV features. The first topology inference head generates a first inference result based on the first target query parameter and model parameters shared with the second fast system.

[0060] In one embodiment, the second fast system includes several third Transformer layers and a second topology inference head; the several third Transformer layers are used to generate second target query parameters based on initialization query parameters and fused BEV features; the second topology inference head is used to generate a second inference result based on the second target query parameters and model parameters shared with the second slow system.

[0061] As a concrete example, see Figure 4As shown, the decoder of the online topology inference model consists of a fast and a slow dual system. The decoder structure comprises several Transformer layers and a topology inference head. The Transformer layers consist of self-attention, lane attention, and a feedforward network. The input to the first Transformer layer is the initialization query from the fast system of the decoder. Real-time output of the current frame (T) BEV features and purification prior BEV characteristics Without the participation of time-series augmented queries, the processing at this layer is shared by both fast and slow systems, as expressed in the formula:

[0062]

[0063] For slow systems using decoders, according to The confidence level is classified, and the M initial queries with the lowest confidence are replaced with time-series augmentation queries. The remaining NM queries are still reserved for initialization. In the second and subsequent Transformer layers, the input query contains M time-series enhancement queries and NM initialization queries. The BEV features utilize fused BEV features. That is, subsequent slow system processing follows:

[0064]

[0065] For fast decoder systems, temporal augmentation queries are not required in the second and subsequent Transformer layers, but fused BEV features are still necessary. Therefore, subsequent fast system processing follows:

[0066]

[0067] The second and subsequent Transformer layers accept parallel inputs from both the slow and fast systems, which share parameters. After processing by the final Transformer layer, the fast and slow systems output topological relationships using their shared parameter topology inference heads. The slow system contains temporal information, while the fast system does not. Therefore, when pose information is available during inference, the inference result from the slow system is used as the final output; when pose information is missing or inaccurate, the inference result from the fast system is used as the final output. This design allows the model to fully utilize temporal information under certain conditions, significantly improving its robustness.

[0068] Step S110: Select from the first inference result and the second inference result based on the reliability of the current pose information to determine the final lane-level topology relationship.

[0069] In one embodiment, the conditions for determining the final lane-level topology relationship by selecting from the first inference result and the second inference result based on the reliability of the current pose information include: when the current pose information exists and its reliability is greater than a reliability threshold, the first inference result is selected as the final lane-level topology relationship; when the current pose information exists and its reliability is less than the reliability threshold, or is missing, the second inference result is selected as the final lane-level topology relationship.

[0070] The reliability of current pose information is typically determined by multi-sensor fusion and consistency evaluation mechanisms. This can be achieved by comprehensively evaluating the state output and data quality of multiple sensors. The criterion for this judgment is whether continuous and accurate relative pose transformations can be provided, specifically determined by the availability and accuracy of sensors such as GPS and IMU. For example, this can be achieved through the following methods:

[0071] Sensor confidence weighting: Using multi-source data such as inertial measurement unit (IMU), global navigation satellite system (GNSS), visual odometry (VO) or lidar (LiDAR), weights are dynamically assigned according to the working status of each sensor in the current environment (such as GNSS signal strength, number of visual feature points) to determine the overall confidence of pose estimation.

[0072] Residual analysis and consistency test: The residual between the observed and predicted values ​​is calculated by filtering algorithms (such as extended Kalman filter EKF or particle filter). If the residual exceeds the preset threshold, it is considered that the current pose estimation may be abnormal and has low reliability.

[0073] Motion continuity detection: Check whether the pose changes at adjacent moments conform to physical motion constraints (such as acceleration and angular velocity limits) to avoid jumps or jitters, thereby judging the rationality of the pose output.

[0074] Map matching verification: The current estimated pose is matched with a known high-precision map. If the projected position deviates significantly from observable landmarks (such as lane lines and signs), the pose is determined to be unreliable.

[0075] The above methods for judging pose reliability can often be combined in various ways to improve the accuracy of pose estimation in complex environments.

[0076] For ease of understanding, see [link to relevant documentation]. Figure 2The diagram illustrates an application of an air-ground cooperative topology understanding method based on a latent world model. This method first generates real-time BEV features from multi-view images collected in real-time by an onboard camera; then, it uses the acquired SD map and UAV aerial images (air-ground cooperative information) to generate map prior BEV features through a map prior extraction model; next, it inputs the real-time BEV features and map prior BEV features into a conflict-aware dynamic prior weight network for processing, generating purified map prior BEV features; finally, it processes these features using an online topology inference model to generate a topology inference result. The embodiments of this invention further provide preferred examples of the specific implementation of the above key steps.

[0077] As a preferred example, a method for generating SD map features corresponding to a standard resolution map may include:

[0078] First, extract the SD map from the Open Street Map (OSM), which means extracting a polyline containing m roads given a specific GPS location and orientation. A local SD map, where each polyline represents an ordered set of two-dimensional points, including k related attributes. For example, the number of broken lines and their respective lengths may vary depending on the road type and number of lanes.

[0079] Next, the extracted SD map is rotated and cropped to generate a view centered on the drone's location that covers the required spatial area. The core purpose of this step is to achieve spatial alignment with the drone aerial image, establishing a unified geometric basis for subsequent feature fusion. Specifically, the rotation operation aims to eliminate directional deviations between the global coordinate system of the SD map and the local coordinate system of the drone, ensuring that the map orientation (such as road direction) is consistent with the drone's shooting perspective; the cropping operation is used to define a unified physical spatial range (a square area with fixed side length centered on the drone), ensuring that the actual geographical area covered by the two maps is completely consistent. Without this step, a point on the SD map will not correspond to the same physical point on the drone image, making subsequent feature fusion difficult.

[0080] The aforementioned UAV location can be provided by the UAV's onboard multi-sensor fusion positioning system. This system uses high-precision latitude and longitude coordinates provided by the Real-time Dynamic Differential (RTK) Global Navigation Satellite System (GNSS) as its core, and integrates inertial measurement unit (IMU) data to perform dead reckoning during brief interruptions in satellite signals, thereby ensuring that the UAV's own three-dimensional position and attitude information with centimeter-level accuracy can be obtained in real time, continuously, and stably.

[0081] This location information serves as a reference point for spatial registration and can be used to calculate the rotation and cropping parameters required for the SD map. Specifically, N points can be sampled uniformly along each polyline, and then the number of polylines can be filled with... To maintain consistent map counts across each training batch, each SD map is reformatted as follows: and ,in Using sine coding Will Transform from 2D coordinates to the corresponding sinusoidal embedding Its dimension is d, that is:

[0082]

[0083] Based on the above road polygon lines and related attributes SD map features can be generated using linear layers and Transformer encoders. :

[0084] ;

[0085] Where Transformer represents the encoder, Linear represents the linear layer, Concat represents the connection structure, and M is the binary mask of the effective polyline.

[0086] As a preferred example, a method for generating features corresponding to drone aerial images may include: collecting aerial images using a drone, and generating orthophotos aligned with specific locations, orientations, and spatial extents through map stitching, rotation, and cropping processes. Image features are extracted using ResNet-50, i.e.:

[0087] flatten it to Dimensions.

[0088] As a preferred example, in order to encode SD maps and drone images into unified prior features, features extracted from SD maps and drone images are... and The sequential crossover into the BEV feature map encoded with positional embeddings is represented as follows: This step first requires initializing an empty BEV feature map covering the target space. Each pixel location in this feature map is assigned a high-dimensional location vector through sine-cosine position encoding. This location vector uniquely encodes the real-world geographic coordinates (X, Y) of the pixel in the BEV space, providing a common geometrically based embedding benchmark for subsequent features from different sources. Subsequently, feature vectors from different modalities are mapped to their corresponding locations in the BEV feature map according to their geographic coordinates. Sequential crossover refers to combining two types of features... and The data is sequentially injected into the BEV mesh that has already encoded location information. Specifically, the fusion method involves concatenating channels followed by a convolutional layer for fusion. Therefore, the final prior BEV features... It can be obtained in the following ways:

[0089]

[0090] As a preferred example, the overall loss of an online topological inference model can be composed of a weighted average of the slow system loss and the fast system loss:

[0091]

[0092] Fast system loss Equivalent to lane segment detection loss Slow system loss Includes lane segment detection loss and world model loss Through weight and Balance, that is:

[0093]

[0094] Lane segment detection loss The goal is to supervise the geometry, category, and topological relationships of lane segments (centerline, left and right boundaries). It is based on Hungarian matching, calculating the loss after optimally matching the predicted results with the ground truth (GT).

[0095] World model loss Divided into BEV feature loss and query loss :

[0096]

[0097] BEV Feature Loss Supervised temporal enhancement of BEV features using mean squared error (MSE) loss. and the current frame BEV feature output in real time Alignment, that is:

[0098]

[0099] Query loss Ensure that predictions are made after historical queries are propagated through the world model. Consistent with the current frame's ground truth (GT), including coordinate loss. Classification loss Mask loss Three sub-loss functions. Coordinate loss. Lane point coordinates are monitored using the L1 loss function; classification loss. Supervision category and boundary type; mask loss Supervise semantic segmentation masks in the BEV space.

[0100]

[0101] in, , , and Representing the coordinates of lane segments, the class of center lines, the class of boundary lines, and time-series enhanced queries, respectively. Predicted semantic BEV mask. , , and This represents the true value after the conversion from frame T-1 to frame T.

[0102] Based on the same inventive concept, this invention also provides an air-ground cooperative topology understanding device based on a latent world model, which mainly includes the following parts:

[0103] The feature acquisition module is used to acquire prior BEV features generated based on air-ground coordination information and real-time BEV features generated based on multi-view images; and to generate cleaned prior BEV features based on prior BEV features and real-time BEV features.

[0104] The first generation module is used to generate temporally enhanced query parameters and temporally enhanced BEV features for the current frame based on the query parameters of the adjacent previous frame, real-time BEV features, and relative poses between adjacent frames, using the latent world model.

[0105] The feature fusion module is used to fuse time-series enhanced BEV features, real-time BEV features, and cleaned-up prior BEV features through the encoder of the online topology inference model to obtain fused BEV features.

[0106] The second generation module is used to perform topological reasoning with time-series information introduced by the decoder of the online topological reasoning model, based on the time-series enhanced query parameters, the model's initial query parameters, and fused BEV features, to generate a first reasoning result; or, based on the initial query parameters, real-time BEV features, and cleaned-up prior BEV features, to perform single-frame topological reasoning and generate a second reasoning result.

[0107] The selection module is used to determine the reliability of the current pose information in the first inference result.

[0108] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0109] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the method described above.

[0110] Specifically, the aforementioned memory and processor can be general-purpose memory and processor, without any specific limitations. When the processor runs a computer program stored in the memory, it can execute the above method.

[0111] The processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0112] Corresponding to the above method, this embodiment of the invention also provides a computer-readable storage medium storing machine-executable instructions. When the computer-executable instructions are called and run by a processor, the computer-executable instructions cause the processor to perform the steps of the above method.

[0113] The apparatus provided in this embodiment of the invention can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the apparatus provided in this embodiment of the invention are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the apparatus embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0114] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0115] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0118] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0120] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0121] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0122] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for air-ground collaborative topology understanding based on a latent world model, characterized in that, The method includes: Acquire prior BEV features generated based on air-ground collaborative information and real-time BEV features generated based on multi-view images; based on the prior BEV features and the real-time BEV features, generate purified prior BEV features through a dynamic prior weight network. The dynamic prior weight network includes: a feature concatenation module, a conflict region detection module, a dynamic weight calculation module, and a weighted fusion module. The feature concatenation module concatenates the prior BEV features and the real-time BEV features to generate features that fuse conflict information. The conflict region detection module extracts and reduces the dimensionality of the features that fuse conflict information to generate a conflict confidence map. The dynamic weight calculation module determines a dynamic prior weight map based on the conflict confidence map. The weighted fusion module multiplies the prior BEV features and the dynamic prior weight map element-wise to generate cleaned prior BEV features. Based on the query parameters of the adjacent previous frame, real-time BEV features, and relative pose between adjacent frames, the latent world model is used to generate the temporal augmented query parameters and temporal augmented BEV features of the current frame. The time-enhanced BEV features, the real-time BEV features, and the cleanup prior BEV features are fused through the encoder of an online topological inference model to obtain fused BEV features. The decoder of the online topology inference model performs topology inference with time-series information based on the time-series enhanced query parameters, the model's initial query parameters, and the fused BEV features to generate a first inference result; or, it performs single-frame topology inference based on the initial query parameters, the real-time BEV features, and the cleaned-up prior BEV features to generate a second inference result. Based on the reliability of the current pose information, a selection is made between the first inference result and the second inference result to determine the final lane-level topology.

2. The method according to claim 1, characterized in that, The air-to-ground cooperative information includes: standard resolution maps and aerial photographs; prior BEV features generated based on the air-to-ground cooperative information are obtained, including: Obtain standard resolution maps and aerial images of the target area, and generate prior BEV features through map prior extraction models.

3. The method according to claim 1, characterized in that, The encoder of the online topological inference model includes a first fast system and a first slow system; The first fast system is used to process the multi-view images acquired in real time to generate the real-time BEV features; the first slow system is used to generate the temporal enhancement query parameters and temporal enhancement BEV features of the current frame.

4. The method according to claim 3, characterized in that, The latent world model includes a query world model and a BEV world model; the latent world model is used to generate temporally enhanced query parameters and temporally enhanced BEV features for the current frame, including: The first slow system generates motion latent variables based on the relative pose between adjacent frames; Based on the query parameters of the adjacent previous frame and the action latent variables, the temporal enhanced query parameters of the current frame are generated using the query world model. Based on the real-time BEV features of the adjacent previous frame and the action latent variables, the BEV world model is used to generate temporally enhanced BEV features.

5. The method according to claim 3, characterized in that, The decoder of the online topological inference model includes a first TransFormer layer, a second slow system, and a second fast system; The first TransFormer layer is used to generate the current query parameters based on the initial query parameters, the real-time BEV features, and the cleaned-up prior BEV features; The second slow system includes a query parameter update module, several second TransFormer layers, and a first topology inference head; The query parameter update module is used to update the current query parameters and generate time-series enhanced query parameters; the plurality of second TransFormer layers are used to generate first target query parameters based on the time-series enhanced query parameters and the fused BEV features; the first topology inference head is used to generate a first inference result based on the first target query parameters and model parameters shared with the second fast system.

6. The method according to claim 5, characterized in that, The second fast system includes several third TransFormer layers and a second topology inference head; The plurality of third TransFormer layers are used to generate second target query parameters based on the initialization query parameters and the fused BEV features; the second topology inference head is used to generate a second inference result based on the second target query parameters and model parameters shared with the second slow system; Based on the reliability of the current pose information, the selection between the first and second inference results is made to determine the final lane-level topology relationship. The conditions include: When the current pose information exists and its reliability is greater than the reliability threshold, the first inference result is selected as the final lane-level topology relationship; If the current pose information exists but its reliability is less than the reliability threshold, or if it is missing, the second inference result is selected as the final lane-level topology relationship.

7. A space-ground cooperative topology understanding device based on a latent world model, characterized in that, include: The feature acquisition module is used to acquire prior BEV features generated based on air-ground cooperative information, as well as real-time BEV features generated based on multi-view images. Based on the prior BEV features and the real-time BEV features, a cleaned prior BEV feature is generated through a dynamic prior weight network. The dynamic prior weight network includes: a feature concatenation module, a conflict region detection module, a dynamic weight calculation module, and a weighted fusion module. The feature concatenation module concatenates the prior BEV features and the real-time BEV features to generate features that fuse conflict information. The conflict region detection module extracts and reduces the dimensionality of the features that fuse conflict information to generate a conflict confidence map. The dynamic weight calculation module determines a dynamic prior weight map based on the conflict confidence map. The weighted fusion module multiplies the prior BEV features and the dynamic prior weight map element-wise to generate cleaned prior BEV features. The first generation module is used to generate temporally enhanced query parameters and temporally enhanced BEV features for the current frame based on the query parameters of the adjacent previous frame, real-time BEV features, and relative poses between adjacent frames, using the latent world model. The feature fusion module is used to fuse the time-series enhanced BEV features, the real-time BEV features, and the cleaned-up prior BEV features through the encoder of the online topology inference model to obtain fused BEV features. The second generation module is used to generate a first inference result by performing topological inference with time-series information introduced based on the time-series enhanced query parameters, the model's initialization query parameters, and the fused BEV features through the decoder of the online topological inference model; or, to perform single-frame topological inference based on the initialization query parameters, the real-time BEV features, and the cleaned-up prior BEV features. The selection module is used to select from the first inference result and the second inference result based on the reliability of the current pose information to determine the final lane-level topology relationship.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images