Scene topology understanding method and device, storage medium and program product

Abstract

The invention discloses a scene topology understanding method and device, a storage medium and a program product, and relates to the field of computer systems based on a specific calculation model, and the method comprises the steps: inputting a multi-view environment image set into a backbone network, and generating bird's-eye view features corresponding to the environment image set; calculating a spatial transformation matrix of the aerial view features at the current moment and the aerial view features of the previous K frames, and performing space-time alignment on the obtained K + 1 frames of aerial view features to obtain multi-frame fusion features; inputting the multi-frame fusion features into a map prior model to obtain aerial view correction features; decoding the aerial view correction features based on a topological decoder, and generating a lane topological graph; comparing the lane topological graph with the annotation data, calculating an error loss function, and optimizing network parameters based on the error loss function; and generating an optimized topological graph, and determining a scene topology result based on the optimized topological graph. By implementing the method, the environment topology understanding capability in a complex scene can be improved, and the generation precision of the lane topological graph is optimized.

Description

Technical Field

[0001] The present application relates to the field of computer systems based on specific computing models, and in particular to a scene topology understanding method, device, storage medium and program product. Background Art

[0002] With the advancement of autonomous driving technology, the demand for smart cars to navigate complex road conditions is increasing. Accurately understanding road topology and lane layout is crucial for safe navigation, especially in scenarios like urban intersections. This places higher demands on the system's environmental perception capabilities in dynamic scenarios.

[0003] Navigation perception systems in related technologies primarily employ two approaches: one relies on offline high-precision map matching and positioning, completing navigation by comparing real-time data collected with a pre-built map database; the other employs online mapping, constructing a 3D structural model of the local environment in real time using multi-sensor data. Both approaches rely primarily on direct feature extraction and mapping when processing environmental information.

[0004] However, under complex road conditions, relying solely on single feature extraction and mapping makes it difficult to simultaneously take into account road structure information at different distance scales, which affects the accuracy of understanding the overall road network topology. Summary of the Invention

[0005] This application provides a scene topology understanding method, device, storage medium and program product for improving the ability to understand environmental topology in complex scenarios and optimizing the generation accuracy of lane topology maps.

[0006] In the first aspect, the present application provides a scene topology understanding method, which is applied to a computing device, and the method includes: inputting a set of environmental images taken by cameras from multiple perspectives at the same time into a backbone network to generate a bird's-eye view feature corresponding to the environmental image set; calculating the spatial transformation matrix of the bird's-eye view feature at the current moment and the bird's-eye view feature of the previous K frames based on the vehicle motion parameters, and performing spatiotemporal alignment on the obtained K+1 frame bird's-eye view features to obtain multi-frame fusion features; inputting the multi-frame fusion features into a map prior model to obtain a bird's-eye view correction feature; decoding the bird's-eye view correction feature based on a topology decoder to generate a lane topology map; comparing the lane topology map with the labeled data, calculating the error loss function, and optimizing the network parameters of the backbone network and the topology decoder based on the error loss function; generating an optimized topology map based on the optimized backbone network and topology decoder, and determining the scene topology result based on the optimized topology map.

[0007] In the above embodiment, the computing device first uses the backbone network to generate bird's-eye view features from multi-view camera images, then realizes multi-frame feature fusion through spatiotemporal alignment, and performs feature correction in combination with map prior knowledge. Finally, it generates a lane topology map based on the topology decoder, and optimizes the network parameters through the error loss function, so that the system can accurately understand the road structure in complex scenes, realize end-to-end learning from multi-source data to scene topology, and improve the accuracy and robustness of environmental perception.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of calculating the spatial transformation matrix of the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames based on the vehicle motion parameters, performing spatiotemporal alignment on the obtained K+1 frame bird's-eye view features, and obtaining multi-frame fusion features specifically includes: obtaining the bird's-eye view features at the current moment as query features, and determining that the query features correspond to the historical bird's-eye view features of the previous K frames; determining the spatial transformation matrix based on the translational motion parameters and rotational motion parameters of the vehicle; spatially aligning the historical bird's-eye view features with the query features based on the spatial transformation matrix to obtain an associated feature sequence; and fusing the associated feature sequence and the query features to obtain a multi-frame fusion feature.

[0009] In the above embodiment, the computing device calculates the spatial transformation matrix based on the vehicle motion parameters to achieve precise alignment of the historical frame and the current frame features. Through the correlation analysis and fusion of the query features and the historical features, the system can make full use of the temporal information, improve the feature expression capability, and effectively solve the feature inconsistency problem caused by vehicle motion.

[0010] In combination with some embodiments of the first aspect, in some embodiments, before the step of inputting the multi-frame fusion features into the map prior model to obtain the bird's-eye view correction features, the method further includes: performing equally spaced sampling on each broken line in the standard definition map to obtain multiple sampling points, and making the number of sampling points of all broken lines equal by zero padding to generate a road broken line sequence; performing sinusoidal function mapping on the two-dimensional coordinates of the road broken line sequence to generate a multidimensional embedding vector; splicing the multidimensional embedding vector with attribute information including road type and number of lanes, and inputting the splicing result into a Transformer encoder to generate a first feature vector; extracting image features of the satellite map based on the ResNet-50 network, and reconstructing the image features into a two-dimensional matrix to obtain a second feature vector; and constructing a map prior model based on the first feature vector and the second feature vector.

[0011] In the above embodiment, the computing device performs structured processing on the standard map, including line sampling, coordinate mapping, and attribute information fusion, to generate rich semantic features. Combined with the satellite image features extracted by the ResNet-50 network, a map prior model with both local details and global structure is constructed.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the topology decoder decoding includes a lane segment conversion module and a topology sequence decoder; the step of decoding the bird's-eye view correction features based on the topology decoder to generate a lane topology map specifically includes: processing the bird's-eye view correction features through the lane segment conversion module, extracting the lane key point coordinates and connection relationships, and generating a lane adjacency matrix; filtering redundant feature points based on the lane adjacency matrix to obtain a key point set containing starting point information and end point information; inputting the key point set into the topology sequence decoder to obtain the connection relationship between all key points in the key point set; connecting all point data in the key point set according to the connection relationship to obtain a lane topology map.

[0013] In the above embodiment, the computing device completes the two key steps of lane segment conversion and sequence decoding based on the topology decoder, extracts the coordinates and connection relationships of key points, generates an adjacency matrix and filters redundant feature points, and then analyzes the connection relationships between key points. Finally, it generates an accurate lane topology map, realizing efficient topological structure analysis.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after the step of decoding the bird's-eye view correction features based on the topology decoder to generate a lane topology map, the method also includes: extracting the spatial position features of the lane nodes in the optimized topology map to generate initial node features; aggregating the initial node features based on the graph neural network to obtain enhanced node features; predicting the attribute label of each lane node based on the enhanced node features; the attribute label includes lane type, driving direction and priority level; and adjusting the priority of the lane connection relationship in the optimized topology map based on the attribute label to generate a scene topology result with traffic rule constraints.

[0015] In the above embodiment, the computing device uses a graph neural network to aggregate and enhance node features, predict the attribute labels of lane nodes, including lane type, driving direction and priority level, and optimizes and adjusts the topology map based on this attribute information so that the generated scene topology results conform to the actual traffic rules and constraints.

[0016] In combination with some embodiments of the first aspect, in some embodiments, before the step of inputting the multi-frame fusion features into the map prior model to obtain the bird's-eye view correction features, the method also includes: calculating the correlation coefficient between the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames; generating feature fusion weights based on the correlation coefficients; performing weighted calculation on the multi-frame fusion features based on the feature fusion weights to obtain time series features; and correcting the map prior model based on the time series features.

[0017] In the above embodiment, the computing device calculates the correlation coefficient between the bird's-eye view features at different times, generates feature fusion weights, and realizes weighted fusion of multi-frame features, which can effectively improve the quality of temporal features and optimize the performance of the map prior model.

[0018] In combination with some embodiments of the first aspect, in some embodiments, the step of calculating the correlation coefficient between the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames specifically includes: performing a feature decomposition operation on the bird's-eye view features of each frame to obtain feature components; calculating the distance matrix between the feature components corresponding to the bird's-eye view features of the current frame and the feature components corresponding to the bird's-eye view features of the previous K frames; performing a normalization operation on the distance matrix to obtain the correlation coefficient.

[0019] In the above embodiment, the computing device decomposes the bird's-eye view features to obtain feature components, calculates the distance matrix between the feature components and normalizes them to obtain an accurate correlation coefficient, providing a reliable weight basis for feature fusion.

[0020] In a second aspect, an embodiment of the present application provides a computing device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, the one or more processors calling the computer instructions to cause the computing device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computing device, enables the computing device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] It is understandable that the computing device provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can be referenced to the beneficial effects of the corresponding methods and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By adopting a multi-view image input and multi-frame feature fusion solution, combined with map prior knowledge and a topology decoder, and optimizing network parameters through an error loss function, the system can fully perceive environmental information from different angles and time sequences. At the same time, it uses prior knowledge to provide structured guidance, effectively solving the problem in existing technologies that single feature extraction cannot take into account road structure information at different scales. It then achieves accurate understanding and reconstruction of environmental topology in complex scenarios, improving the environmental perception capability of the automatic navigation system.

[0025] 2. By adopting a processing method that combines standard map line sampling, coordinate mapping, and attribute information fusion, combined with the ResNet-50 network to extract satellite image features, and constructing a map prior model containing rich semantic information, the system can simultaneously obtain the geometric and attribute characteristics of roads, effectively solving the problem of insufficient utilization of map information in existing technologies, and thus achieving a comprehensive understanding of the road network structure, providing reliable prior knowledge support for subsequent topological analysis.

[0026] 3. By adopting a feature fusion weight calculation method based on correlation coefficients and optimizing multi-frame fusion features through weighted operations, the system can adaptively adjust the fusion ratio of features at different times, effectively solving the problem of simple superposition of multi-frame features in existing technologies that easily introduces noise, thereby achieving high-quality temporal feature extraction and improving the performance and robustness of the map prior model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a scene topology understanding method according to an embodiment of the present application; Figure 2 This is another flowchart of the scene topology understanding method in an embodiment of the present application; Figure 3 It is a schematic diagram of the physical device structure of the computing device in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, the singular expressions "a", "an", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations of one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0030] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.

[0031] In increasingly complex urban road environments, autonomous vehicles need to accurately understand the topological structure of road scenes. This is especially true in complex scenarios such as multi-lane intersections and roundabouts, where the road structure contains numerous topological relationships such as forks and merges, and is subject to multiple levels of traffic regulations. For example, a six-lane intersection includes a through lane, a left-turn lane, and a right-turn lane. Different lanes have different priorities, and some lanes are controlled by traffic lights. Traditional environmental perception methods often struggle to accurately understand such complex road topologies, and are prone to misjudging lane connections and inaccurate priority recognition, posing safety risks to the decision-making and planning of autonomous driving systems.

[0032] In related technologies, topological understanding of road scenes can be achieved by combining feature matching based on pre-stored maps with single-view real-time detection. The following describes scenarios where the scene topology understanding method in related technologies is used.

[0033] The current mainstream solution combines preloaded high-precision maps with real-time sensor detection. For example, at a multi-lane intersection, the system first loads pre-stored road topology data. It then uses a forward-facing camera to detect lane markings and road markings, aligning the real-time detection results with the map data through simple feature matching. However, this approach has significant drawbacks: First, map data may be outdated or incomplete, failing to reflect conditions such as road construction and temporary diversions. Second, feature detection from a single perspective is easily affected by factors such as occlusion and lighting, leading to inaccurate matching. When the road structure changes, the system is unable to adapt and adjust its cognitive results in a timely manner.

[0034] By integrating spatiotemporal alignment and map prior knowledge, we achieve accurate understanding of complex scenes, which not only improves the accuracy of topology recognition but also enhances the system's adaptability to scene changes. The following describes scenarios where the scene topology understanding method described in this application is used.

[0035] After adopting this solution, the system can effectively handle complex road scenarios. Taking a multi-lane Y-shaped intersection as an example, the system first obtains omnidirectional environmental images from multiple cameras and extracts bird's-eye view features through the backbone network. Then, combined with vehicle motion information, the system performs spatiotemporal alignment and fusion of consecutive multi-frame features, overcoming the problem of incomplete single-frame image information. At the same time, by integrating map prior knowledge and real-time perception results, the system can accurately understand the relationship between lane bifurcations and merging, and correctly identify the attributes and priority levels of different lanes. Even in the case of road diversion construction, the system can promptly adjust its cognitive results based on the dynamic update of multi-frame features.

[0036] It can be seen that the scene topology understanding method in the embodiment of the present application can not only achieve accurate road structure recognition, but also effectively solve the problems of incomplete features and poor environmental adaptability in the existing technology, thereby realizing reliable navigation of the intelligent driving system.

[0037] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , which is a flow chart of the scene topology understanding method in an embodiment of the present application.

[0038] S101: Input a set of environmental images taken by cameras with multiple perspectives at the same time into a backbone network to generate bird's-eye view features corresponding to the set of environmental images.

[0039] Among them, the environmental image set refers to a collection of scene images obtained by multiple cameras simultaneously shooting from different angles, which is used to capture all-round information about the vehicle's surrounding environment; the backbone network refers to the deep neural network architecture used for feature extraction, usually adopting classic network structures such as ResNet or VGG; the bird's-eye view feature represents the feature representation extracted after converting the environmental image to a bird's-eye view, which contains the planar layout and structural information of the scene.

[0040] After acquiring multi-perspective image data of the vehicle's surroundings, the computing device needs to convert these images into easy-to-understand feature representations. Specifically, the computing device first preprocesses the images captured by multiple cameras, including image denoising, illumination normalization, and other operations, and then inputs the preprocessed images into a pre-trained backbone network. The backbone network maps the input image to a high-dimensional feature space through multi-layer convolution operations and feature transformations, and simultaneously performs perspective transformations to generate a bird's-eye view representation of the scene. Finally, the computing device extracts the key features of these bird's-eye views, including structured information such as road boundaries, lane lines, and intersections.

[0041] In this step, the computing device converts the multi-camera image features into BEV features (i.e., bird's-eye view features) to provide a unified representation of the surrounding environment for various autonomous driving perception tasks. As an example, we use the traditional backbone to generate multi-view feature maps Then the PV image features Transformed into BEV characteristics

[0042] In some embodiments, the conversion from multi-perspective images to bird's-eye view features can be achieved in a variety of ways: Optionally, the computing device can first transform images of different perspectives into a unified bird's-eye view through a homography matrix, and then use a multi-branch feature extraction network to process the transformed images separately, and finally integrate the features of each branch into a unified representation through a feature fusion module; Optionally, the computing device can also first independently extract features from the image of each perspective, and then transform the feature map to a bird's-eye view through a projection transformation, and finally perform weighted fusion of features from multiple perspectives through an attention mechanism. It is understandable that other feature extraction and perspective transformation methods can also be used to achieve the conversion of multi-perspective images to bird's-eye view features, which are not limited here.

[0043] It should be noted that the Backbone Network model is a deep learning model for multi-view environmental perception. During the training phase, the model receives environmental images from different cameras and their corresponding annotated data. Through forward propagation and backward optimization, it uses a composite loss function containing multiple components such as L1 loss, Focal Loss, and cross-entropy loss to optimize network parameters. The model adopts a deep convolutional architecture similar to ResNet or VGG, extracting hierarchical features of the image through multi-layer convolution operations and residual connections. When in use, the model takes environmental images taken simultaneously by multiple cameras as input. After continuous convolutional layers and feature transformations, the images are mapped from different perspectives to a unified bird's-eye view feature space, and finally outputs a bird's-eye view feature representation of the scene. For example, when a car is at an intersection, the model can uniformly convert images from multiple cameras such as the front view and side view into bird's-eye view features viewed from top to bottom, providing a basis for subsequent scene understanding.

[0044] S102. Calculate the spatial transformation matrix of the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames based on the vehicle motion parameters, perform spatiotemporal alignment on the obtained K+1 frame bird's-eye view features, and obtain multi-frame fusion features.

[0045] Among them, the vehicle motion parameters represent the displacement and posture change information of the vehicle between adjacent moments, including translation vectors and rotation angles; the spatial transformation matrix refers to the affine transformation matrix that describes the spatial correspondence between feature maps; spatiotemporal alignment means transforming feature maps at different moments into the same coordinate system; and the multi-frame fusion feature representation integrates the feature representation of information at multiple moments.

[0046] After acquiring features from multiple consecutive bird's-eye view frames, the computing device needs to factor in the changes in perspective brought about by vehicle motion. Specifically, the computing device first obtains relative motion information between adjacent frames based on data from the vehicle's odometer or IMU sensor. Then, based on this motion information, it constructs a spatial transformation matrix that describes how to transform the feature maps from the previous frames to the coordinate system of the current frame. The computing device uses these transformation matrices to align the feature maps from the K+1 frames, ensuring they are spatially consistent. Finally, the computing device performs temporal fusion on the aligned features to obtain fused features containing information from multiple frames.

[0047] In this step, the computing device can enhance the robustness of the spatiotemporal consistency alignment by introducing multi-frame historical BEV features. It can reduce the computational complexity of the alignment process, maintain high accuracy, and at the same time, utilize more historical information to improve the computing device's perception of the dynamic environment. The current environment is represented by the temporal self-attention method TSA and combined with multi-frame historical BEV features. Given the BEV query Q at the current timestamp t, select the historical BEV features of multiple timestamps (for example, K = 3), which are represented as B t-1 , B t-2 , B t-3 Next, according to the car's self-motion (such as translation and rotation), B t-k Align with the current BEV query Q so that features at the same grid correspond to the same real-world location. Here, the aligned historical BEV features are denoted as B′ t-k . The model is as follows: where Q p represents the BEV query at p = (x, y). For the first sample of each sequence, the BEV feature {Q, B′ t-1 ,B′ t-2 ,B′ t-3} is replaced by repeated BEV query {Q,Q,Q,Q}. q,p,x represent query, reference point and input features respectively. i represents attention head, N head Indicates the total number of attention heads. j is the key index of the sample, N key The total number of sampled keys for each header. and is the learnable weight, where C is the feature dimension. is the predicted attention weight, through Perform normalization. is the prediction of the reference point p. The offset x(p+Δp ij ) represents the position p+Δp ij The features are extracted by bilinear interpolation.

[0048] For BEV queries, the query Q at p = (x, y) in Q p The corresponding real-world position (x ′ ,y ′ )for: Where H, W are the spatial shapes of BEV queries, s is the resolution of the BEV grid, (x ′ ,y ′ ) is the coordinate with the position of the car as the origin.

[0049] In some embodiments, spatiotemporal alignment and fusion of features can be achieved in a variety of ways: Optionally, the computing device can use a Kalman filter to make a smooth estimate of vehicle motion, calculate a transformation matrix based on the smoothed motion parameters, then perform feature resampling through bilinear interpolation, and finally use a gated recurrent unit (GRU) to perform temporal feature fusion; Optionally, the computing device can also directly calculate the pixel-level correspondences between feature maps through optical flow estimation, construct a transformation matrix based on these correspondences, and then fuse multi-frame features using an adaptive weighted method. It is understandable that other motion estimation and feature fusion methods can also be used to achieve spatiotemporal alignment of multi-frame features, which is not limited here.

[0050] S103: Input the multi-frame fusion features into the map prior model to obtain the bird's-eye view correction features.

[0051] Among them, the multi-frame fusion feature represents the scene feature representation after spatiotemporal alignment and fusion; the map prior model refers to a pre-trained deep learning model containing standard map information, which is used to provide structured prior knowledge of the scene; the bird's-eye view correction feature represents the scene feature after correction by map prior knowledge, which has more accurate road structure information.

[0052] After acquiring the multi-frame fusion features, the computing device needs to use map prior knowledge to optimize and correct the features. Specifically, the computing device first inputs the multi-frame fusion features into a pre-trained map prior model. This model uses an attention mechanism to match and compare the input features with the road structure information in the standard map, identifying possible deviations and noise in the features. The computing device then corrects the features based on the matching results, including supplementing missing road information, correcting inaccurate lane line positions, and adjusting road topology relationships. Finally, the computing device outputs the corrected bird's-eye view features, which better reflect the structure of the actual road scene.

[0053] In this step, after the spatiotemporal consistency alignment is performed, the computing device can fuse a customized map prior model (SAMAP) to further refine the BEV features. SAMAP uses readily available navigation maps (SD maps) and satellite maps for lane map prediction. The learned robust map prior is seamlessly integrated into the online driving topology reasoning model, significantly enhancing lane perception and topology reasoning, and enhancing scalability, reducing dependence on training data captured by consistent sensor configurations. That is, a map prior model is trained using standard definition (SD) maps and satellite maps, called SAMAP. The features extracted from the SD and satellite maps are sequentially cross-involved in the BEV feature map encoded with position embedding, and the BEV features fused with the SAMAP prior features can be obtained, namely B prior for: For the establishment of the prior map, firstly, N points are uniformly sampled along each polyline of the SD map, and then the number of polylines m is filled ′ to maintain consistent counts of maps within each training batch. This reformulates each SD map as R ′ =[r ′ 1,…,r ′ m ,0,…] m′ ,and The parameters Then use sinusoidal coding E sin R ′ Convert from 2D coordinates to the corresponding sinusoidal embedding with d dimensions R ″ =E sin (R ′ ) Through the refined road polyline R ″ and related attributes α ′ , we generated SD map features with linear layers and Transformer encoder F SD =Enc(Linear(Concat(R ″ ,α ′ )),M) where M is a binary mask representing valid polylines.

[0054] For satellite maps, ResNet-50 is used as the image backbone To extract features and flatten it to dimension.

[0055] Get SD maps from Open-StreetMap (OSM). SD maps provide comprehensive geographic information for all parts of the world. Given a specific GPS location and direction, extract m road polylines R = [r1,…,r m ]. Each polyline is represented as an ordered set of 2D points and contains k related attributes Such as road type and number of lanes. The number of polylines and their respective lengths may vary. The extracted map is then rotated and cropped to generate an egocentric view covering the desired spatial extent.

[0056] Obtain satellite maps from the Mapbox Raster Map API, which segments the Earth's surface into raster images at different zoom levels. Given GPS coordinates, the map is projected onto a 2D map plane, the appropriate image index is identified, and the API is queried to retrieve the relevant imagery. The zoom level is set to 20, corresponding to approximately 0.11 meters per pixel. A process of stitching, rotating, and cropping the map fragments produces a satellite map aligned to a specific location, orientation, and spatial extent.

[0057] In some embodiments, feature correction based on map prior knowledge can be achieved in a variety of ways: Optionally, the computing device can first construct a scene graph representation, represent the road elements in the fused features as nodes, and represent the relationships between them as edges, and then interact with the scene graph of the standard map through a graph attention network, and update the feature representations of the nodes and edges based on the interaction results; Optionally, the computing device can also use a multi-scale feature pyramid structure to match the map prior information at different spatial resolutions, and optimize global and local information through top-down and bottom-up feature fusion. It is understandable that other feature correction and knowledge fusion methods can also be used to achieve feature optimization based on map priors, which are not limited here.

[0058] It should be noted that the map prior model (SAMAP) is a two-stream neural network that integrates standard map knowledge and satellite imagery. During training, the model first samples road polylines from the standard definition map at equal intervals, using zero padding to ensure that all polylines have the same number of sample points. It then uses a sine function to map the two-dimensional coordinates into a high-dimensional embedding vector, which is then concatenated with attribute information such as road type and number of lanes before being fed into a Transformer encoder. Simultaneously, a ResNet-50 network is used to extract visual features from satellite imagery. The model architecture consists of two parallel feature extraction branches: a Transformer branch that processes standard map information and a ResNet-50 branch that processes satellite imagery. The features from these two branches are interactively fused through an attention mechanism. In practice, the model inputs road data from a SD map and satellite imagery of the corresponding area. The model adaptively fuses the features from these two information sources through cross-attention computation, outputting enhanced BEV features that incorporate rich prior knowledge. For example, when handling complex intersections, the model can fuse road topology information from the standard map with the actual road shape from satellite imagery, providing more accurate scene understanding.

[0059] S104: Decode the bird's-eye view correction features based on the topology decoder to generate a lane topology map.

[0060] Among them, the topology decoder represents the neural network module used to convert the feature map into a structured road topology representation; the lane topology map refers to a graph structure representation that describes the connection relationship between lanes in the road network, including information about lane nodes and connecting edges.

[0061] After acquiring the corrected bird's-eye view features, the computing device needs to decode them into a specific road topology. Specifically, the computing device first uses the multi-layer decoding module of the topology decoder to gradually convert high-dimensional features into geometric and semantic representations of road elements. The decoding process includes multiple subtasks, such as lane line detection, intersection recognition, and connection relationship reasoning. The computing device then constructs an initial lane topology map based on the decoding results, where nodes represent lane segments or key points, and edges represent the connections between lanes. Finally, the computing device optimizes and refines the topology map to ensure it complies with traffic regulations and the actual road structure.

[0062] In this step, the topology decoder includes a lane segment conversion module and a topology sequence decoder. The lane segment conversion module predicts a lane map, and keypoints from this predicted lane map are fed into the topology sequence decoder to construct keypoint cues, which are then connected to an edge sequence. The topology sequence decoder infers the relationships between discrete keypoints and reconstructs them into a coherent lane map. In other words, the lane segment conversion module and the topology sequence decoder form a topology decoder, which simultaneously learns a lane topology sequence extracted from a directed acyclic graph (DAG) and a lane map containing geometric information. Random sequence cues to sequence learning extract unordered keypoints from the lane map predicted by the lane segment conversion module, which are then fed into the topology sequence decoder to ultimately reconstruct an ordered and complete lane map. In this way, the lane segment conversion module learns powerful long-range perception and accurate topology reasoning from the topology sequence decoder.

[0063] Lane segment conversion module through BEV characteristics The self-attention and cross-attention updates can learn the query Q l , the updated query is passed to the prediction head for lane segmentation. The start and end points of the predicted lane map are extracted as keypoint hints y K The ground-truth (GT) lane map is converted into an edge sequence y E Edge sequence and key point hint y K concatenated and fed into the topological sequence decoder where they are combined with the BEV features The prediction sequence is supervised by the target edge sequence and can be decoded into a channel graph. During testing, the topology sequence decoder does not perform inference. Instead, the lane segment conversion module predicts the centerline and adjacency matrix. In the lane segment conversion module, the set of instance-level queries is represented as where N L is a preset number of queries, which is usually larger than the number of centerlines in the lane map. The queries are fed into the lane segment conversion module to obtain updated queries: Where LaneD represents the lane segmentation procedure. In each lane segmentation procedure layer, the lane query is updated sequentially through the self-attention module, the lane attention module, and the feedforward network. We use the MLP to generate the 3D coordinates and topology A of the lane L: The MLPs are independent of each other. To represent the lane map in more detail, not only the topology but also the offsets of the left and right lane boundaries, the types of these boundaries, and crosswalks can be predicted.

[0064] Based on the prediction of the lane segment conversion module, we sort the predicted lanes from high to low confidence and convert the ground truth map into an edge sequence yE and use the prediction adjacency matrix to filter out any duplicate keypoints in each predicted lane: The coordinates of the key points are discretized. The object of random order prompt to sequence learning can be expressed as: where w i represents the class weight, express All previous tags, Is the predicted target sequence. The key point prompt y K Relative to the marginal sequence y E The two are disordered, and then connected and fed back to the topological sequence decoder to interact with the BEV features, so that the sequence topological decoder can guide the network to infer the relationship between discrete key points. In this way, the network can infer the correct association between disordered key point prompts, which is more conducive to capturing long-range relationships.

[0065] The edge sequence is concatenated with the keypoint cues and then fed into the topological sequence decoder for training. Each decoder layer of the topological sequence decoder consists of a self-attention module, a cross-attention module, and a feedforward network. The causal mask in the self-attention module maintains the autoregressive property. The output of the trained topological sequence decoder can be expressed as: Among them, TopoSeqD represents the topological sequence decoder, Represents the predicted edge sequence. To leverage the long-range understanding and sequence relationship capabilities of sequence-to-sequence learning, we promote the interaction between the lane segment conversion module and the sequence topology decoder at keypoint cues. The overall structure brings the following advantages to the extraction and refinement of BEV features: (1) Enhanced feature refinement: The model can selectively focus on relevant regions of the BEV features based on the sequence. This targeted attention emphasizes the accurate reconstruction of key lane maps or key scene areas, improving the BEV features. (2) Improved long-range dependencies: The topological sequence decoder enhances the capture of long-range dependencies between long-range keypoints. It injects contextual information between keypoints into the lane segment conversion module, enabling it to predict more aligned lane segment endpoints. (3) Context aggregation: By focusing on specific keypoint cues, the model can reduce the influence of irrelevant or redundant information in the BEV features. Improve feature extraction efficiency and reduce noise in the final prediction.

[0066] In some embodiments, decoding features into a topology map can be achieved in a variety of ways: optionally, the computing device can first use a fully convolutional network to generate a heat map of lane lines and key points, then extract discrete lane nodes through a clustering algorithm, and finally predict the connection relationship between nodes based on a graph reasoning network; optionally, the computing device can also use an end-to-end graph generation network to directly predict the adjacency matrix and node features of the graph structure from the feature map, and optimize the structure of the topology map through iterative refinement. It is understood that other feature decoding and graph construction methods can also be used to achieve the conversion from features to a topology map, which is not limited here.

[0067] It should be noted that the topology decoder is a two-stage neural network that decodes feature maps into structured road topology. During training, the model uses data containing lane annotations and topology sequence annotations, optimizing parameters through a multi-task learning framework. Loss functions include Focal Loss for lane classification, Cross Entropy Loss for segmentation, and Focal Loss for topological relationships. The model architecture consists of two core modules: the Lane Segment Conversion Module first processes input features using self-attention and cross-attention to extract keypoint coordinates and generate an adjacency matrix. The Topology Sequence Decoder then uses a multi-layer decoder to infer the connectivity between keypoints based on their positional information and local features. In practice, the model receives bird's-eye view features corrected by map priors and, through the serial processing of these two modules, ultimately outputs a complete lane topology map. For example, for a Y-shaped intersection, the model first identifies key bifurcations and endpoints, then accurately infers the connections between these points, forming a realistic lane network structure.

[0068] S105: Compare the lane topology map with the labeled data, calculate the error loss function, and optimize the network parameters of the backbone network and the topology decoder based on the error loss function.

[0069] Among them, the labeled data represents the manually labeled real scene topology information; the error loss function refers to the mathematical metric used to measure the difference between the generated topology map and the real topology map; the network parameters represent the learnable weight parameters in the backbone network and the topology decoder.

[0070] After generating the lane topology map, the computing device needs to optimize model performance by comparing it with the annotated data. Specifically, the computing device first aligns the generated lane topology map with the annotated data of the corresponding scene and calculates the structural similarity and semantic consistency between them. The computing device then constructs an error loss function consisting of multiple components, each focusing on accuracy in different aspects such as topology, geometry, and semantic attributes. Finally, the computing device uses a backpropagation algorithm to calculate the gradient of the loss function with respect to the network parameters and updates the parameters using gradient descent, making the topology map generated by the model closer to the real scene.

[0071] In this step, the computing device introduces an overall loss L evaluation model in order to further utilize semantic and geometric information. Among them, the overall loss formula is as follows: in represents L1 loss. Represents the Focal Loss loss for lane classification. Including cross entropy loss and Dice Loss loss. represents the cross entropy loss, which is used to classify the left and right lane types (e.g., invisible, solid, dashed). Is the Focal Loss loss, used to supervise the predicted adjacency matrix A and the GT adjacency matrix The relationship information between them. represents the maximum likelihood loss, which is used to supervise the topological sequence decoder to predict the label. The weights of each loss are represented by α1, α2, α3, α4, α5 and α6 respectively.

[0072] In some embodiments, network parameter optimization can be achieved in a variety of ways: optionally, the computing device can adopt a multi-task learning framework to simultaneously optimize the loss functions of multiple subtasks, such as topology prediction, lane attribute classification, and connection relationship reasoning, balancing the importance of each task by setting different weight coefficients; optionally, the computing device can also use an adversarial learning strategy, introduce a discriminator network to evaluate the authenticity of the generated topology map, and improve model performance by alternating optimization of the generator and discriminator. It is understandable that other parameter optimization and learning strategies can also be used to improve model accuracy, which is not limited here.

[0073] S106 : Generate an optimized topology map based on the optimized backbone network and the topology decoder, and determine a scene topology result based on the optimized topology map.

[0074] Among them, the optimized topology map represents the new lane topology map generated using the optimized network parameters; the scene topology result refers to the final output structured representation of the road scene, which contains complete lane network information.

[0075] After completing network parameter optimization, the computing device needs to generate the final scene topology. Specifically, the computing device first reprocesses the input image using the optimized backbone network to extract more accurate scene features. Then, using the optimized topology decoder, the features are converted into an optimized topology graph with higher structural accuracy and semantic consistency. Finally, the computing device performs post-processing on the optimized topology graph, including topology simplification, redundancy elimination, and constraint verification, to obtain the final scene topology result.

[0076] In some embodiments, the final topology can be generated in a variety of ways: Alternatively, the computing device can use a probabilistic graphical model to reason about the optimized topology graph, optimizing the connectivity between nodes through a message passing algorithm while incorporating local and global structural constraints. Alternatively, the computing device can use a rule engine to verify and modify the topology graph, ensuring that the generated scenario topology complies with predefined traffic rules and road design specifications. It is understood that other graph optimization and constraint processing methods can also be used to generate the final scenario topology, and these are not limited here.

[0077] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , which is another flow chart of the scene topology understanding method in an embodiment of the present application.

[0078] S201: Input a set of environmental images captured by cameras with multiple perspectives at the same time into a backbone network to generate bird's-eye view features corresponding to the set of environmental images.

[0079] Referring to step S101 , the computing device generates a bird's-eye view feature.

[0080] It should be noted that before the multi-view images are input into the backbone network, a detailed image preprocessing process needs to be supplemented. The system first unifies the spatial resolution of the original images captured by each camera and scales the images to a preset uniform size (such as 1920×1080 pixels). Then, illumination normalization is performed, and the brightness distribution of the image is adjusted through histogram equalization or gamma correction to reduce the impact of illumination differences at different times and under different weather conditions. Image enhancement is then performed, including adaptive contrast adjustment and noise suppression, to improve the signal-to-noise ratio and detail performance of the image. Finally, perspective calibration is performed, and images from different perspectives are projected into a unified world coordinate system based on known camera parameters (intrinsic and extrinsic matrix). These preprocessing steps ensure that the images input to the backbone network have consistent quality and spatial reference.

[0081] S202. Calculate the spatial transformation matrix of the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames based on the vehicle motion parameters, perform spatiotemporal alignment on the obtained K+1 frame bird's-eye view features, and obtain multi-frame fusion features.

[0082] Referring to step S102 , the computing device generates multi-frame fusion features.

[0083] In some embodiments, the computing device will perform historical image association, that is, the computing device will obtain the bird's-eye view features at the current moment as query features, and determine the historical bird's-eye view features of the previous K frames corresponding to the query features; determine the spatial transformation matrix based on the translational motion parameters and rotational motion parameters of the vehicle; spatially align the historical bird's-eye view features with the query features based on the spatial transformation matrix to obtain an associated feature sequence; and fuse the associated feature sequence and the query features to obtain a multi-frame fusion feature.

[0084] Among them, the query feature represents the bird's-eye view feature at the current moment that needs to be processed; the historical bird's-eye view feature refers to the scene feature obtained at the previous K frames; the translation motion parameter represents the change in the vehicle's position in space; the rotation motion parameter refers to the change in the vehicle's posture angle; the spatial transformation matrix is ​​used to describe the geometric transformation relationship between features at different moments; the associated feature sequence represents the set of historical features that have been spatially aligned; and the multi-frame fusion feature refers to a comprehensive feature representation that integrates temporal information.

[0085] When performing multi-frame feature fusion, the computing device needs to take into account the changes in perspective brought about by vehicle motion. Specifically, the computing device first extracts the bird's-eye view features of the current frame and the previous K frames from the feature cache. Then, the computing device obtains motion parameters based on the vehicle's odometer data or inertial measurement unit and constructs a matrix describing the coordinate transformation. Next, the computing device uses this matrix to perform an affine transformation on the historical features, aligning them to the coordinate system of the current frame. Finally, the computing device integrates the aligned sequence features with the current features through the feature fusion module to generate fused features containing rich temporal information.

[0086] In some embodiments, spatiotemporal alignment and fusion of features can be achieved in a variety of ways: Optionally, the computing device can first estimate the vehicle's motion state using an extended Kalman filter, then construct an accurate transformation matrix based on the state estimate, then use bilinear interpolation for feature resampling, and finally fuse multiple frame features by weighted averaging; Optionally, the computing device can also use a deep feature matching method to first extract key point descriptors from the feature map, then estimate the spatial transformation relationship through descriptor matching, then use a spatial transformation network for feature alignment, and finally adaptively fuse features through an attention mechanism. It is understandable that other feature alignment and fusion methods can also be used to achieve effective integration of spatiotemporal information, which is not limited here.

[0087] It should be noted that the system has designed a complete temporal feature processing framework, which achieves precise spatial alignment of features through bilinear interpolation. The alignment process takes into account the vehicle's six-degree-of-freedom motion parameters. During the feature fusion stage, the system adopts a dynamic weighting scheme based on the attention mechanism, adaptively determining the importance of features at different times by calculating the similarity matrix of query-key-value pairs. To improve the robustness of feature fusion, the system introduces residual connections and layer normalization operations, and adjusts the focus of the attention distribution through a learnable temperature parameter. The entire fusion process is trained in an end-to-end manner, and the loss function incorporates multiple components such as reconstruction error, temporal consistency, and feature smoothness. For example, when the vehicle is driving in a dynamic traffic environment, this mechanism can effectively handle feature misalignment caused by vehicle motion and scene changes, ensuring the accuracy of feature fusion.

[0088] S203 , performing equally spaced sampling on each polyline in the standard definition map to obtain a plurality of sampling points, and making the number of sampling points of all polylines equal by zero filling to generate a road polyline sequence.

[0089] Among them, the standard definition map represents electronic map data containing standardized road information; the polyline refers to a sequence of line segments that describes the road geometry; the equally spaced sampling refers to selecting sampling points at fixed distance intervals on the polyline; zero padding refers to adding zero values ​​to make sequences of different lengths reach the same dimension; the road polyline sequence represents the standardized road geometry after sampling and padding.

[0090] When processing standard map data, the computing device needs to convert irregular road geometry into a uniformly formatted sequence representation. Specifically, the computing device first traverses each road polyline in the standard map definition and calculates the locations of sampling points based on a preset sampling interval. The computing device then performs a sampling operation on each polyline to obtain the coordinate information of the sampling points. Because the lengths of different road segments vary, the number of sampled points may vary. Therefore, the computing device adds zeros to the end of shorter sequences to ensure that all polyline sequences have the same length, facilitating subsequent batch processing.

[0091] In some embodiments, sampling and normalization of road polylines can be achieved through various methods: Optionally, the computing device can first calculate the total length of the polyline, determine the sampling interval based on the target number of sampling points, then use linear interpolation to uniformly sample the polyline, and finally normalize the sequence of sampling points. Alternatively, the computing device can also use an adaptive sampling strategy, increasing the sampling density in areas with greater curvature and reducing the number of sampling points in straight segments, using a dynamic programming algorithm to ensure that the sampling results retain the road's geometric characteristics to the greatest extent possible. It is understood that other sampling and normalization methods can also be used to achieve serialized representation of road geometric information, and these are not limited here.

[0092] S204: Perform sinusoidal function mapping on the two-dimensional coordinates of the road polyline sequence to generate a multi-dimensional embedding vector.

[0093] Among them, the two-dimensional coordinates represent the planar position information of the road polyline sampling points; the sine function mapping refers to the mathematical transformation of spatial coordinates into high-dimensional feature space using trigonometric functions; the multidimensional embedding vector represents the road feature representation after nonlinear transformation.

[0094] After obtaining the road polyline sequence, the computing device needs to convert the spatial coordinate information into a feature representation that is easier for the network to process. Specifically, the computing device first inputs the two-dimensional coordinates of each sampling point into sine and cosine functions of different frequencies, generating a series of periodic activation values. This transformation encodes the spatial position information into a high-dimensional vector, allowing the distance in the feature space to better reflect the relative relationship of the coordinates. The computing device then combines the transformation results of all sampling points into a sequence, forming a multidimensional embedding vector that contains rich position encoding information.

[0095] In some embodiments, the mapping of coordinates to feature space can be achieved in a variety of ways: optionally, the computing device can use a multi-scale position encoding scheme to capture local and global spatial relationships by combining sinusoidal functions of different frequencies, and use a hierarchical feature aggregation strategy to enhance representation capabilities; optionally, the computing device can also use a learning-based nonlinear mapping network to optimize the coordinate transformation function through end-to-end training, so that the generated features are more suitable for subsequent task processing. It is understandable that other feature mapping methods can also be used to achieve high-dimensional representation of spatial coordinates, which is not limited here.

[0096] S205: Concatenate the multidimensional embedding vector with the attribute information including the road type and the number of lanes, and input the concatenated result into the Transformer encoder to generate a first feature vector.

[0097] Among them, attribute information represents the semantic labels describing road features; the splicing operation refers to connecting multiple feature vectors into a larger vector in dimension; the Transformer encoder refers to a neural network module based on the self-attention mechanism; the first feature vector represents the road feature representation that integrates geometric and semantic information.

[0098] After obtaining the multidimensional embedding vector for the road, the computing device needs to fuse the geometric features with the semantic attribute information. Specifically, the computing device first converts attribute information such as road type and number of lanes into a one-hot encoded vector. The computing device then concatenates these attribute vectors with the multidimensional embedding vector along the feature dimension to form a combined feature containing complete road information. Finally, the computing device inputs the combined feature into the Transformer encoder. Through a multi-head self-attention mechanism and feedforward network layers, the first feature vector is generated that captures the long-range dependencies between features.

[0099] In some embodiments, feature fusion and encoding can be achieved through a variety of methods: Alternatively, the computing device can employ a hierarchical attention mechanism, first encoding geometric and attribute features separately, then learning the interaction between the two features through a cross-modal attention module, and finally generating a unified representation through a feature aggregation network. Alternatively, the computing device can employ conditional encoding, using attribute information as a conditional vector to guide the encoding of geometric features, dynamically adjusting the importance of different features through a gating mechanism. It is understood that other feature fusion and encoding methods can also be employed to achieve a unified representation of road information, and these are not limited here.

[0100] S206. Extract image features of the satellite map based on the ResNet-50 network, and reconstruct the image features into a two-dimensional matrix to obtain a second eigenvector.

[0101] The ResNet-50 network refers to a deep convolutional neural network with a residual connection structure; the satellite map represents a high-resolution map image taken from an aerial bird's-eye view; the image feature represents the visual features extracted from the satellite image; and the two-dimensional matrix represents a feature representation with a spatial structure.

[0102] After acquiring satellite map data, the computing device needs to extract the visual information contained therein. Specifically, the computing device first feeds the satellite map image into a pre-trained ResNet-50 network, extracting multi-scale visual features through multi-layer convolution operations. The computing device then performs spatial pyramid pooling on the feature maps at different levels to preserve the spatial structure of the features. Finally, the computing device reorganizes the pooled features into a two-dimensional matrix format, aligning them with the representation of other features.

[0103] In some embodiments, satellite image feature extraction can be achieved through a variety of methods: Alternatively, the computing device can use a multi-branch feature extraction network to process images in parallel at different spatial resolutions, integrate multi-scale information through a feature pyramid network, and finally generate a unified representation through an adaptive feature reconstruction module. Alternatively, the computing device can also employ an attention-enhanced feature extraction scheme, introducing spatial attention and channel attention mechanisms to adaptively highlight important regions and feature channels based on task requirements. It is understood that other feature extraction methods can also be used to achieve effective representation of satellite image information, and this is not limited here.

[0104] S207: Construct a map priori model based on the first eigenvector and the second eigenvector.

[0105] Among them, the first eigenvector represents the feature representation containing road geometry and semantic information; the second eigenvector represents the visual feature representation extracted from satellite images; and the map prior model represents the pre-trained model used to provide scene structured knowledge.

[0106] After obtaining the two feature vectors, the computing device needs to construct a map model that can provide prior knowledge. Specifically, the computing device first designs a feature fusion module that calculates the correlation between the two feature vectors using an attention mechanism. It then adaptively fuses the features based on the correlation weights to generate a unified scene representation. Finally, the computing device transforms the fused features using a multilayer perceptron network to construct a complete map prior model.

[0107] In some embodiments, the construction of a map prior model can be achieved through various methods: Alternatively, the computing device can use a hierarchical feature fusion architecture to establish associations between features at different semantic levels, and through progressive information integration, build a prior model with multi-scale understanding capabilities. Alternatively, the computing device can also use a graph neural network framework to convert the two feature vectors into node representations and establish structured relationships between features through a message passing mechanism. It is understood that other model construction methods can also be used to achieve the representation of map prior knowledge, and this is not limited here.

[0108] S208: Calculate the correlation coefficient between the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames.

[0109] Among them, the correlation coefficient represents a numerical indicator that describes the similarity between features at different times; the feature component represents the decomposition of high-dimensional features into multiple basic components; the distance matrix represents the pairwise distance measurement between feature components; and the normalization operation represents the standardized processing of mapping numerical values ​​to a fixed range.

[0110] After obtaining the features of multiple bird's-eye view frames, the computing device needs to evaluate the temporal correlation between them. Specifically, the computing device first performs singular value decomposition on each frame's features to extract the main feature components. It then calculates the Euclidean or cosine distance between the feature components of the current frame and those of previous frames to construct a complete distance matrix. Finally, the computing device normalizes the distance matrix and maps the correlation coefficients to the [0, 1] interval to facilitate subsequent weight calculations.

[0111] For example, for each frame bird's-eye view feature The main feature components are obtained through singular value decomposition (SVD): F = UΣV^T, where U and V are the left and right singular vector matrices, respectively, and Σ is the singular value matrix. The components corresponding to the first k largest singular values ​​are retained to obtain the reduced-dimensional feature representation F_r. For the current frame feature F_t and the historical frame feature F_(ti), their correlation coefficient is calculated: r_(t, i) = exp(-||F_(t, r) - F_(ti, r)||_2^2 / τ); where τ is a temperature parameter used to adjust the sensitivity of the correlation. All correlation coefficients are then normalized using the softmax function: w_i=exp(r_(t, i)) / Σ_jexp(r_(t, j)); the weight w_i obtained in this way not only reflects the similarity of features at different times, but also ensures that the sum of the weights is 1.

[0112] In some embodiments, the calculation of feature correlation can be achieved in a variety of ways: optionally, the computing device can use an attention mechanism to calculate the similarity between features, capture the correlation at different semantic levels through a multi-head attention module, and finally obtain the final correlation coefficient through weighted averaging; optionally, the computing device can also use a kernel correlation analysis method to map features to a high-dimensional kernel space and measure the nonlinear correlation between features by calculating the eigenvalue decomposition of the kernel matrix. It is understandable that other correlation measurement methods can also be used to evaluate the degree of correlation between features, which is not limited here.

[0113] In some embodiments, the computing device performs feature decomposition to calculate the correlation, that is, the computing device performs feature decomposition operations on the bird's-eye view features of each frame to obtain feature components; calculates the distance matrix between the feature components corresponding to the bird's-eye view features of the current frame and the feature components corresponding to the bird's-eye view features of the previous K frames; performs a normalization operation on the distance matrix to obtain a correlation coefficient.

[0114] Among them, the eigendecomposition operation represents the mathematical transformation of decomposing high-dimensional features into their main components; the eigencomponents refer to the basic components of the eigenvectors; the distance matrix represents the similarity measure between different eigencomponents; the normalization operation is used to standardize the values ​​to a specific range; and the correlation coefficient represents a standardized indicator of the degree of quantified feature similarity.

[0115] When processing features from multiple bird's-eye view frames, the computing device needs to evaluate the correlation between them. Specifically, the computing device first performs principal component analysis or singular value decomposition on each frame's features to extract the most representative feature components. The computing device then calculates the Euclidean distance or cosine similarity between the feature components of the current frame and those of previous frames to construct a complete distance relationship matrix. Finally, the computing device normalizes the distance matrix using min-max normalization or a softmax function to obtain a normalization coefficient that reflects the degree of feature correlation.

[0116] In some embodiments, the measurement of feature correlation can be achieved in a variety of ways: optionally, the computing device can first use an autoencoder to reduce the dimension and reconstruct the features, then calculate the statistical dependence between the features through a mutual information estimator, then build a probability distribution model based on the kernel density estimation method, and finally quantify the degree of correlation of the features through information theory indicators; optionally, the computing device can also use a contrastive learning framework to first build positive and negative sample pairs, then optimize the representation space of the features through metric learning, then use a temperature parameter to adjust the smoothness of the similarity distribution, and finally determine the correlation level through an adaptive threshold. It is understandable that other correlation analysis methods can also be used to achieve accurate assessment of the degree of feature association, which is not limited here.

[0117] S209: Generate feature fusion weights according to the correlation coefficient.

[0118] Among them, the feature fusion weight represents the coefficient that controls the importance of features at different times; the weight generation network represents the neural network module that maps the correlation coefficient to the fusion weight.

[0119] After obtaining the correlation coefficients, the computing device needs to convert them into appropriate feature fusion weights. Specifically, the computing device first uses a multi-layer neural network to perform a nonlinear transformation on the correlation coefficients to generate initial weight values. The weights are then normalized using a softmax function to ensure that the sum of all weights is 1. Finally, the computing device performs temperature scaling on the weights to adjust the smoothness of the weight distribution and avoid overly sharp weight distributions.

[0120] In some embodiments, weight generation can be achieved in a variety of ways: optionally, the computing device can use a gating mechanism to dynamically control the flow of information based on the correlation coefficient, and adaptively adjust the importance of features through a learnable gating unit; optionally, the computing device can also use a gradient-based weight optimization method to directly learn the optimal fusion weights through backpropagation, while introducing regularization constraints to ensure the sparsity of the weights. It is understood that other weight generation methods can also be used to achieve dynamic adjustment of feature importance, which is not limited here.

[0121] S210 , performing weighted calculation on multi-frame fusion features based on feature fusion weights to obtain temporal features.

[0122] Among them, weighted calculation refers to the operation of linear combination of features based on weights; time series feature refers to the feature representation that integrates multi-frame time series information.

[0123] After obtaining the fusion weights, the computing device performs weighted fusion on the multi-frame features. Specifically, the computing device first multiplies each frame feature by its corresponding weight to obtain a weighted feature. Then, the computing device sums all weighted features element-wise to obtain a preliminary fusion result. Finally, the computing device normalizes the fusion result to generate a time series feature with good numerical stability.

[0124] In some embodiments, weighted feature fusion can be achieved in a variety of ways: Optionally, the computing device can use a recursive neural network structure, taking weighted features as the input sequence, and gradually integrating temporal information through gated recurrent units to generate fused features containing long-term dependencies; Optionally, the computing device can also use a self-attention mechanism to allow features at different times to interact directly, achieving more flexible information fusion through multi-level feature aggregation. It is understood that other feature fusion methods can also be used to achieve effective integration of temporal information, which is not limited here.

[0125] It should be noted that during the feature fusion stage, the system processes multi-frame features through spatiotemporal alignment and dynamic weighting. First, historical features are spatially aligned. An affine transformation matrix T_(t, k) is constructed based on the vehicle's motion state to transform the features. This matrix incorporates the vehicle's translation and rotation information, ensuring that historical features are accurately aligned to the current coordinate system. The aligned features are then adaptively weighted and fused using an attention mechanism. The weighting coefficients take into account both the temporal decay factor exp(-k / τ_t) and the feature similarity exp(-D(F_t, F_(tk)^aligned) / τ_s), where τ_t and τ_s control the sensitivity to temporal decay and spatial similarity, respectively. The system incorporates a multi-head attention mechanism to calculate inter-feature correlations. An attention score matrix is ​​constructed using a learnable query matrix W_Q, a key matrix W_K, and a value matrix W_V, enabling dynamic and selective feature fusion. The fused features undergo residual connections and layer normalization to ensure stable information transfer. For example, when an autonomous vehicle passes through a dynamically changing intersection scene, the fusion mechanism can effectively integrate observation information at different times, retaining the stable structural information in historical features while responding to the dynamic changes of the scene in a timely manner.

[0126] S211. Modify the map prior model based on the time series features.

[0127] Among them, the temporal feature represents the feature representation containing multi-frame temporal information; the map prior model correction represents the process of dynamically updating the model according to the temporal information.

[0128] After obtaining the time series features, the computing device uses this information to update the map prior model. Specifically, the computing device first compares the time series features with the output of the current map prior model and calculates the difference between them. Then, based on this difference information, the computing device makes local adjustments to the model parameters to enable the model to adapt to dynamic changes in the scene. Finally, the computing device updates the model's internal state, completing the dynamic correction of the map prior knowledge.

[0129] In some embodiments, dynamic model correction can be achieved through a variety of methods: Alternatively, the computing device can use an online learning algorithm to continuously update model parameters based on temporal features, gradually adapting the model to new scene information through gradient descent optimization. Alternatively, the computing device can employ a meta-learning framework to rapidly adjust the model through a rapid adaptation mechanism while maintaining the model's ability to retain historical knowledge. It is understood that other model update methods can also be used to achieve dynamic adaptation of map prior knowledge, and these are not limited here.

[0130] It should be noted that the model correction process employs an adaptive learning strategy, dynamically adjusting model parameters by comparing the fused features with the prior model's predictions. The system first calculates feature differences ΔF_s at multiple scales and uses a max-pooling operation to obtain feature representations at different receptive fields, ensuring that it can capture both local details and changes in global structure. Based on these multi-scale feature differences, the system constructs an adaptive update gating unit g = sigmoid(W_g · [ΔF_1||ΔF_2||ΔF_4]) to control the extent of parameter updates. Model parameter updates utilize momentum optimization, smoothly adjusting the parameters via an exponential moving average mechanism: θ_ema = β · θ_ema + (1-β) · θ_new. The momentum coefficient β dynamically adjusts with the number of updates, ensuring that the model can maintain stability while responsive to scene changes. The update process also incorporates regularization constraints, using the L2 norm to limit the magnitude of parameter changes and prevent the model from over-adapting to temporary anomalies. For example, when the road environment undergoes continuous changes (such as construction diversion), the correction mechanism can gradually adjust the model parameters to adapt to the new scene characteristics while maintaining the ability to generalize to other scenes.

[0131] S212: Input the multi-frame fusion features into the map prior model to obtain the bird's-eye view correction features.

[0132] Referring to step S103 , the computing device generates a bird's-eye view correction feature.

[0133] S213: Decode the bird's-eye view correction features based on the topology decoder to generate a lane topology map.

[0134] Referring to step S104 , the computing device generates a lane topology map.

[0135] In some embodiments, the computing device will perform feature decoding based on the lane segment conversion module and the topological sequence decoder, that is, the computing device will process the bird's-eye view correction features through the lane segment conversion module, extract the lane key point coordinates and connection relationships, and generate a lane adjacency matrix; filter redundant feature points based on the lane adjacency matrix to obtain a key point set containing starting point information and end point information; input the key point set into the topological sequence decoder to obtain the connection relationship between all key points in the key point set; connect all point data in the key point set according to the connection relationship to obtain a lane topology map.

[0136] Among them, the lane segment conversion module represents the neural network component that maps features into structured lane representations; the lane adjacency matrix refers to a two-dimensional matrix that describes the connection relationship between lanes; the key point set represents the important nodes that characterize the road topology structure; the topology sequence decoder refers to the neural network module used to infer the connection relationship between nodes; and the lane topology map is used to represent the complete road network structure.

[0137] After obtaining the bird's-eye view corrected features, the computing device needs to convert them into a specific road structure representation. Specifically, the computing device first processes the features through the lane segment conversion module, extracting the locations of key points representing the lanes and preliminary connectivity information. The computing device then constructs a lane adjacency matrix and uses this matrix to filter out redundant or noisy points, retaining key points with important topological significance. The computing device then inputs the filtered key points into a topology sequence decoder, which uses deep learning to infer the detailed connectivity between the points. Finally, based on the inference results, the computing device connects the key points into a complete lane network, forming the final topology map.

[0138] In some embodiments, the conversion from features to topology can be achieved in a variety of ways: optionally, the computing device can first use a fully convolutional network to generate a key point heat map, then extract discrete key points through non-maximum suppression, then use a graph attention network to infer the connection relationship between points, and finally generate smooth lane lines through curve fitting; optionally, the computing device can also use an end-to-end graph generation method, first directly generate a set of key points through a point prediction network, then use an edge prediction module to estimate the connection probability between point pairs, then construct an initial topology through a maximum spanning tree algorithm, and finally refine the final structure through graph optimization. It is understandable that other feature decoding methods can also be used to achieve road topology reconstruction, which is not limited here.

[0139] It should be noted that the system uses a hierarchical feature extraction and graph structure inference approach to generate and optimize the topology graph. The feature extraction network constructs multi-scale feature representations using residual blocks and a feature pyramid network. Each layer of features is enhanced with spatial and channel-wise attention mechanisms to ensure that key structural information is captured. During the graph structure inference phase, the system first constructs an initial graph structure based on node prediction probabilities and spatial distance thresholds. Node features are then iteratively optimized within a graph neural network using a designed message passing function that comprehensively considers node and edge features and spatial positional relationships. The system incorporates a comprehensive topology-constrained optimization framework, formalizing physical rules such as connectivity, smoothness, and cross-constraints through the energy function E(G) to ensure that the generated topology conforms to actual road design specifications. During the attribute inference phase, the system simultaneously predicts node and edge attributes using a multi-task learning framework, and a rule verification module is designed to ensure the rationality of attribute assignments. The entire optimization process employs an alternating iterative strategy, repeatedly updating the graph attention network, edge prediction network, and attribute prediction network until the topology converges or a preset number of iterations is reached. For example, when dealing with a complex overpass, the mechanism can accurately model the spatial relationships of multi-layer roads while ensuring the rationality of lane connections and the consistency of traffic rules.

[0140] In some embodiments, the computing device will adjust and verify the results, that is, the computing device will extract the spatial position features of the lane nodes in the optimized topology map to generate initial node features; aggregate the initial node features based on the graph neural network to obtain enhanced node features; predict the attribute labels of each lane node based on the enhanced node features; the attribute labels include lane type, driving direction and priority level; based on the attribute labels, the priority of the lane connection relationship in the optimized topology map is adjusted to generate a scene topology result with traffic rule constraints.

[0141] Among them, spatial position features represent the coordinate information of nodes on a two-dimensional plane; initial node features refer to the basic feature representation containing location information; graph neural networks represent deep learning models that process graph structured data; enhanced node features refer to the enriched feature representation after information transmission; attribute labels represent semantic information that describes the functional characteristics of lanes; and priority levels are used to indicate the importance of right-of-way between lanes.

[0142] After obtaining the optimized topology map, the computing device needs to apply traffic rule constraints to it. Specifically, the computing device first extracts the coordinate information of each node from the topology map and encodes it into an initial feature vector. The computing device then uses a graph neural network to perform multiple rounds of message passing, enabling each node to perceive its local and global topological environment. Next, the computing device predicts lane attribute information such as type, direction, and priority based on the enhanced node features. Finally, the computing device adjusts the connectivity between lanes based on the predicted attribute labels to ensure that the topology conforms to actual traffic rules.

[0143] In some embodiments, attribute prediction and rule constraints can be achieved in a variety of ways: optionally, the computing device can first build a multi-layer graph convolutional network to process node features, then capture the dependencies between nodes through the edge attention mechanism, then use a multi-task learning framework to simultaneously predict multiple attributes, and finally use a rule verification module to ensure the consistency of attribute assignment; optionally, the computing device can also use a conditional random field model to first generate an initial probability distribution of attributes based on node features, then introduce a pairwise potential energy function to model the mutual constraints between attributes, then optimize the overall label assignment through a confidence propagation algorithm, and finally use heuristic rules for post-processing optimization. It is understandable that other attribute reasoning methods can also be used to achieve reasonable modeling of traffic rules, which is not limited here.

[0144] It should be noted that the graph neural network is a deep graph learning model for node feature enhancement and attribute prediction. During the training process, the model uses lane node data with complete attribute annotations (including lane type, driving direction and priority level), optimizes network parameters through supervised learning, and uses classification loss functions for different attributes. The model is based on a message passing mechanism, and realizes information exchange and feature enhancement between nodes through multi-layer graph convolution or attention layers. Each layer aggregates the information of neighboring nodes to update the feature representation of the central node. When applied, the model takes the position features of the nodes in the topological graph as input, and gradually enhances the feature representation of the nodes through multiple rounds of message passing mechanisms, and finally predicts the complete attribute label of each node. For example, when processing a complex intersection, the model can accurately infer the type of each lane (straight lane, turning lane, etc.), driving direction and traffic priority based on the spatial position relationship of the nodes, providing the necessary traffic rules information for autonomous driving decisions.

[0145] S214. Compare the lane topology map with the labeled data, calculate the error loss function, and optimize the network parameters of the backbone network and the topology decoder based on the error loss function.

[0146] Referring to step S105 , the computing device optimizes the network parameters.

[0147] S215 . Generate an optimized topology map based on the optimized backbone network and the topology decoder, and determine a scene topology result based on the optimized topology map.

[0148] Referring to step S106 , the computing device determines the scene topology result.

[0149] In the embodiments of this application, due to the use of multi-perspective image input, multi-frame feature fusion, and a dynamic optimization scheme based on map priors, it is possible to fully perceive environmental information and achieve spatiotemporal alignment of features, effectively solving the problem in existing technologies where single feature extraction is difficult to take into account road structure information at different scales, thereby achieving accurate understanding of environmental topology in complex scenarios. Through the attribute reasoning and rule constraints of graph neural networks, the system can accurately identify semantic information such as lane type, driving direction, and priority level, providing comprehensive scene cognition support for autonomous driving decisions. At the same time, parameter optimization based on the error loss function ensures that the system has good environmental adaptability and robustness.

[0150] The following describes the computing device in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a computing device in an embodiment of the present application.

[0151] It should be noted that Figure 3The structure of the computing device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0152] like Figure 3 As shown, the computing device includes a CPU 301, which can perform various appropriate actions and processes according to the programs stored in the ROM 302 or the programs loaded from the storage unit 308 into the RAM 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0153] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, push button switches, etc.; an output section 307 including a liquid crystal display (LCD), an audio output device, indicator lights, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 310 as needed, so that computer programs read therefrom can be installed into the storage section 308 as needed.

[0154] In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309 and / or installed from the removable medium 311. When the computer program is executed by the CPU 301, the various functions defined in the present invention are performed.

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings.

[0156] Specifically, the computing device of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the scene topology understanding method provided in the above embodiment is implemented.

[0157] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the computing device described in the above embodiments, or may exist independently and not incorporated into the computing device. The storage medium carries one or more computer programs, which, when executed by a processor of the computing device, enable the computing device to implement the scene topology understanding method provided in the above embodiments.

[0158] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0159] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

Claims

1. A scene topology understanding method, characterized in that: Applied to a computing device, the method includes: Inputting a set of environmental images taken by cameras with multiple perspectives at the same time into the backbone network, and generating bird's-eye view features corresponding to the set of environmental images; According to the vehicle motion parameters, the spatial transformation matrix of the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames is calculated, and the obtained K+1 frame bird's-eye view features are spatially and temporally aligned to obtain multi-frame fusion features; Inputting the multi-frame fusion features into a map prior model to obtain a bird's-eye view correction feature; Decoding the bird's-eye view correction feature based on a topology decoder to generate a lane topology map; Comparing the lane topology map with the annotated data, calculating an error loss function, and optimizing network parameters of the backbone network and the topology decoder based on the error loss function; An optimized topology map is generated based on the optimized backbone network and the topology decoder, and a scene topology result is determined based on the optimized topology map.

2. The method according to claim 1, characterized in that The step of calculating the spatial transformation matrix of the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames based on the vehicle motion parameters, performing spatiotemporal alignment on the obtained K+1 frame bird's-eye view features, and obtaining multi-frame fusion features specifically includes: Obtaining a bird's-eye view feature at the current moment as a query feature, and determining a historical bird's-eye view feature of the previous K frames corresponding to the query feature; Determine a spatial transformation matrix according to the translational motion parameters and the rotational motion parameters of the vehicle; spatially aligning the historical bird's-eye view features with the query features based on the spatial transformation matrix to obtain a correlation feature sequence; The associated feature sequence and the query feature are fused to obtain a multi-frame fusion feature.

3. The method according to claim 1, characterized in that Before the step of inputting the multi-frame fusion features into a map prior model to obtain a bird's-eye view correction feature, the method further includes: Perform equal-interval sampling on each polyline in the standard definition map to obtain multiple sampling points, and use zero padding to make the number of sampling points of all polylines equal to generate a road polyline sequence; Performing sinusoidal function mapping on the two-dimensional coordinates of the road polyline sequence to generate a multi-dimensional embedding vector; Concatenating the multidimensional embedding vector with attribute information including road type and number of lanes, and inputting the concatenated result into a Transformer encoder to generate a first feature vector; Extracting image features of the satellite map based on a ResNet-50 network, and reconstructing the image features into a two-dimensional matrix to obtain a second eigenvector; A map priori model is constructed based on the first eigenvector and the second eigenvector.

4. The method according to claim 1, wherein The topology decoder includes a lane segment conversion module and a topology sequence decoder; the step of decoding the bird's-eye view correction feature based on the topology decoder to generate a lane topology map specifically includes: Processing the bird's-eye view correction features through the lane segment conversion module, extracting the coordinates and connection relationships of lane key points, and generating a lane adjacency matrix; Filtering redundant feature points based on the lane adjacency matrix to obtain a key point set including start point information and end point information; Inputting the key point set into the topological sequence decoder to obtain the connection relationship between all key points in the key point set; All point data in the key point set are connected according to the connection relationship to obtain a lane topology map.

5. The method according to claim 1, wherein After the step of decoding the bird's-eye view correction feature based on a topology decoder to generate a lane topology map, the method further includes: Extracting spatial position features of lane nodes in the optimized topology graph to generate initial node features; Aggregating the initial node features based on a graph neural network to obtain enhanced node features; Predicting attribute labels for each lane node based on the enhanced node features; the attribute labels include lane type, driving direction, and priority level; The lane connection relationship in the optimized topology graph is prioritized based on the attribute labels to generate a scene topology result with traffic rule constraints.

6. The method according to claim 1, characterized in that Before the step of inputting the multi-frame fusion features into a map prior model to obtain a bird's-eye view correction feature, the method further includes: Calculate the correlation coefficient between the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames; generating a feature fusion weight according to the correlation coefficient; Performing weighted calculation on the multi-frame fusion features based on the feature fusion weights to obtain temporal features; The map prior model is modified based on the temporal features.

7. The method according to claim 6, characterized in that The step of calculating the correlation coefficient between the bird's-eye view features at the current moment and the bird's-eye view features of the previous K frames specifically includes: Perform feature decomposition on the features of each frame of the bird's-eye view to obtain feature components; Calculate the distance matrix between the feature components corresponding to the bird's-eye view features of the current frame and the feature components corresponding to the bird's-eye view features of the previous K frames; A normalization operation is performed on the distance matrix to obtain a correlation coefficient.

8. A computing device, characterized in that The computing device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the computing device to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a computing device, the computing device is caused to perform the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a computing device, the computing device is caused to perform the method according to any one of claims 1 to 7.

Images