A topological semantic memory enhanced autonomous driving planning method

Abstract

The application belongs to the technical field of automatic driving, and is especially a topological semantic memory enhanced automatic driving planning method, which comprises the following steps: S1: constructing a topological semantic scene graph of a current driving scene; S2: constructing a scene alignment embedding model and a topological semantic memory library; S3: recalling historical priori based on the topological semantic memory library and fusing planning input; and S4: generating a future path and a speed curve and completing model training. The application can convert visual perception information into a topological semantic scene graph, recall scene priori with similar driving logic from the topological semantic memory library based on topological consistency, and jointly use current visual information, navigation state information, self-motion state and historical driving priori for future trajectory planning, so as to improve the safety, robustness, real-time performance and planning reliability of an automatic driving system in a complex open environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to an autonomous driving planning method enhanced with topological semantic memory. Background Technology

[0002] Autonomous driving systems need to generate safe, smooth, and traffic-compliant future driving trajectories in real time based on the vehicle's current perception of the traffic environment, navigation targets, and its own motion state. Traditional autonomous driving systems typically employ an architecture where perception, prediction, decision-making, planning, and control modules execute sequentially. While this architecture offers strong engineering interpretability, the reliance on manually designed intermediate representations and interfaces between modules makes them prone to error propagation. When the vehicle is in complex scenarios such as occlusion, dense traffic flow, complex intersections, unprotected turns, or sudden obstacles, perception and prediction errors can further affect the planning results, leading to unstable trajectories or unreliable decisions.

[0003] End-to-end autonomous driving methods directly establish a mapping relationship from sensor inputs to planned trajectories or control commands through neural networks, enabling joint optimization of environmental understanding and motion planning processes within a unified model. Common inputs include onboard camera images, navigation commands, vehicle speed, acceleration, heading angle, and other state information, while outputs typically include future trajectory points, driving actions, or vehicle control variables. Compared to traditional modular methods, end-to-end methods reduce manual rules and intermediate interface design, offering greater flexibility in modeling complex scenarios. However, most existing end-to-end methods rely on implicit parameterized knowledge acquired during model training for planning. When the distribution of test scenarios differs significantly from that of training data, the model is susceptible to the influence of insufficient long-tail data, potentially leading to erroneous decisions, trajectory deviations, or behavioral instability in rare and dangerous scenarios, complex interaction scenarios, or open road scenarios.

[0004] Visual semantic understanding models and visual language action models offer new technical pathways for end-to-end autonomous driving. These models can connect image information, navigation intent, textual semantics, and scene understanding, enabling autonomous driving systems to possess stronger environmental understanding and semantic reasoning capabilities. Furthermore, they combine semantic understanding with trajectory generation, allowing the model to directly generate future driving trajectories or action planning results based on visual input, navigation information, and vehicle status. While these methods improve the semantic understanding capabilities of autonomous driving systems, they still tend to over-rely on the model's internal parameter memory in long-tailed and complex scenarios, making it difficult to fully utilize historical driving experience and expert prior knowledge.

[0005] Historical experience enhancement techniques can be used to alleviate the model's over-reliance on parametric knowledge. The basic idea is to extract information relevant to the current input from external knowledge bases, historical scene libraries, or expert experience bases, and provide it as additional prior knowledge to the planning model. For autonomous driving tasks, if scenario examples with similar driving logic to the current scenario can be found in the historical driving scene library, it can provide a reference for the current vehicle's trajectory planning, thereby improving the model's generalization ability in long-tail scenarios. However, directly using raw visual features for historical scene matching still has significant shortcomings. Raw images or multi-view visual features have high dimensionality, and large-scale matching will lead to significant storage and computational overhead, making it difficult to meet the real-time requirements of closed-loop autonomous driving. Furthermore, visual similarity does not necessarily mean identical driving logic. Two scenarios may have similar road backgrounds and vehicle distributions, but due to differences in traffic signal states, lane topology, preceding vehicle behavior, or navigation goals, the actual planning strategies may be completely different. Simply relying on image similarity can easily introduce erroneous historical priors.

[0006] A more suitable approach to acquiring historical priors for autonomous driving tasks should focus on traffic elements, interaction relationships, and road topological constraints within the scene, rather than just the appearance of the image. Topological semantic scene graphs can represent key elements such as the vehicle itself, surrounding vehicles, traffic signs, traffic lights, and lane structure as nodes, and represent relationships between vehicles, signal constraints, lane affiliation, and interaction relationships as edges, thus forming a structured scene representation oriented towards driving decision-making. Compared to raw visual features, topological semantic scene graphs can filter out irrelevant visual noise, retain core information affecting driving decisions, and reduce the redundancy of historical scene matching representations. By encoding the topological semantic scene graph into compact low-dimensional vectors and ensuring that the vector distances are consistent with the scene topology and interaction logic, scene priors that are more closely matched to the current driving task can be obtained from the topological semantic memory.

[0007] Based on the above background, it is necessary to construct an autonomous driving planning method with topological semantic memory enhancement. Summary of the Invention

[0008] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0009] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0010] A topological semantic memory-enhanced autonomous driving planning method includes the following steps:

[0011] S1: Obtain the vehicle's current input information, identify traffic elements through a visual encoder and perception module; use traffic elements as nodes, guide relationships, active relationships, inactive relationships and dependency relationships as edges, and record the vehicle's physical state to form a topological semantic scene graph;

[0012] S2: Input the topological semantic scene graph into the scene alignment embedding model. The topological relation-aware encoder models traffic elements and their semantic relationships, and the graph structure decoder reconstructs the topological semantic scene graph. During the training phase, graph structure reconstruction loss and topological alignment loss are used to ensure that the scene latent vectors maintain consistency in topological structure, temporal evolution, and driving logic. In the offline phase, historical scene latent vectors, the original topological semantic scene graph, scene physical state, historical path, and speed information are stored as driving priors in the scene primitive vector database of the topological semantic memory.

[0013] S3: During online runtime, the current topological semantic scene graph is encoded into a query vector, and the top-K historical scene priors that are similar to the current scene in terms of traffic topology, interaction relationships and driving logic are retrieved from the topological semantic memory. The current vision, navigation target, vehicle speed and historical scene priors are concatenated into a unified input for the driving planning model.

[0014] S4: The driving planning model extracts spatial path information and longitudinal speed information based on the fused input, and outputs the future path and speed curve through the path decoder and speed decoder to obtain the planned trajectory of the vehicle in the future period of time, thus completing the model training.

[0015] As a preferred embodiment of the autonomous driving planning method with topological semantic memory enhancement described in this invention, in step S1, the input information includes visual input, navigation command information, and vehicle status information. The visual input is a forward-looking camera image, a multi-frame continuous image, or a multi-view camera image. The navigation command information includes a target point, route direction, or high-level navigation command. The vehicle status information includes the current position, speed, heading angle, acceleration, braking status, and lane number.

[0016] As a preferred embodiment of the topological semantic memory-enhanced autonomous driving planning method described in this invention, the specific method of step S1 is as follows: Let the current visual input be... Navigation instruction information is The vehicle status is Historical driving priors recalled from the topological semantic memory are The driving planning process is then represented as:

[0017] in, This represents a topological semantic memory-enhanced driving planning model. This represents the future planning result output by the model; the planning result is decomposed into two parts: future path and future velocity.

[0018]

[0019] in, Indicates a spatial path point at several future time steps. This represents the velocity curve or longitudinal velocity planning result for the corresponding time step.

[0020] Identify key elements relevant to driving decisions from visual input, including the vehicle itself, nearby vehicles, traffic signs, traffic lights, and lane elements, and establish scene primitives centered on the vehicle at the current moment. Scene primitives Defined as:

[0021]

[0022] in, This represents a set of interactive elements, which includes at least the vehicle itself, nearby vehicles, traffic signs, traffic lights, and lane elements. It represents a set of semantic relations, including at least guiding relations, active relations, inactive relations, and dependency relations; It represents a set of physical states, used to describe the position, size, speed, heading angle, state category, and distance of various interactive elements;

[0023] Based on the above scenario primitives, a topological semantic scenario graph is constructed. :

[0024]

[0025] in, This represents a set of nodes, where each node corresponds to a traffic element. This represents a set of edges, where each edge corresponds to a semantic relationship or topological constraint between two traffic elements.

[0026] As a preferred embodiment of the autonomous driving planning method with enhanced topological semantic memory described in this invention, in step S2, the network structure of the scene alignment embedding model includes a topological relation-aware encoder, a temporal attention pooling layer, and a graph structure decoder. The topological relation-aware encoder includes a relation graph convolutional layer, a Transformer layer, an attention pooling layer, and a linear mapping layer, used to generate scene latent vectors based on node attributes, edge relationships, and temporal changes in consecutive frames. The graph structure decoder includes a linear mapping layer, a Sigmoid activation layer, and an edge relationship prediction layer, used to reconstruct a topological semantic scene graph based on the scene latent vectors.

[0027] As a preferred embodiment of the autonomous driving planning method with enhanced topological semantic memory described in this invention, the specific method of S2 is as follows: for continuous driving scenarios, the input is represented as a graph sequence composed of multiple time-step topological semantic scene graphs:

[0028]

[0029] in, These represent the time intervals from the first moment to the second moment in a continuous driving scenario. The topological semantic scene graph constructed at each moment;

[0030] The topology relation-aware encoder first models different types of traffic relationships through a relation graph convolutional layer. For the i-th node in the topological semantic scene graph, the topology relation-aware encoder weights and aggregates the state information of associated traffic elements based on the semantic relationship type, interaction strength, and adjacency relationship between nodes, and then fuses the node's own state to obtain the updated node representation; the node feature update of the i-th layer is represented as:

[0031]

[0032] in, Represents a node In the Features of the layer Represents a set of relation types. Indicates a relationship Next and Node The set of connected neighbor nodes, Representing relations The corresponding learnable weights, Indicates the self-join weight. Indicates the strength of interaction between nodes. Represents the normalization coefficient. Indicates the activation function;

[0033] After obtaining the graph structure features, the topology-aware encoder models the temporal changes between continuous scenes through the Transformer layer, and aggregates the structural information of different time steps and different traffic elements through the attention pooling layer; for the th One point of attention is:

[0034]

[0035]

[0036] in, These represent the query input features, key input features, and value input features, respectively, that are input to the attention layer. , , They represent the first The learnable linear projection matrix corresponding to each attention head; , and They represent the first The query matrix, key matrix, and value matrix in each attention head; This represents the output of the m-th attention head. This is used to scale the attention score to avoid excessively large dot product results when the feature dimension is large;

[0037] The outputs of multiple attention heads are concatenated and linearly mapped to obtain the latent scene vector used for retrieval and alignment:

[0038]

[0039] in, They represent The output of each attention head; This indicates the number of attention heads in a multi-head attention mechanism; This represents the learnable linear mapping matrix after multi-head attention output;

[0040] A graph structure decoder reconstructs a topological semantic scene graph from the scene latent vectors, and a graph structure reconstruction loss is set. The graph structure decoder takes the scene latent vectors as input and predicts the edge relationships and semantic constraints between nodes through a linear mapping layer and a sigmoid activation layer, thereby obtaining the reconstructed topological semantic scene graph.

[0041]

[0042] in, The graph structure reconstruction loss is used to ensure that the topological semantic scene graph reconstructed by the graph structure decoder based on the scene latent vectors remains consistent with the real topological semantic scene graph. Indicates batch size. Indicates the first The number of frames in a scene segment Represents the set of edges in the prediction graph. Represents the set of edges in the real graph. This indicates the first [section / part] in the scene segment. One time step;

[0043] Set the topology alignment loss during the training phase:

[0044]

[0045] in, This represents the topology alignment loss, used to constrain the distance in the scene's latent vector space to remain consistent with the difference in the real scene's topology. This indicates the number of scene segments in the training batch; and These represent the indices of two different scene segments within the batch; and They represent the first The scene fragment and the first The latent vector of a scene obtained by encoding a scene fragment using a scene alignment embedding model; Indicates the first The scene graph sequence and the first The graph structure dynamic temporal warping distance between a sequence of scene graphs is used to measure the real differences between two consecutive driving scenes in terms of topology, temporal evolution, and interaction relationships.

[0046] The total training loss of the scene alignment embedding model is:

[0047]

[0048] in, and These represent the weights of the graph structure reconstruction loss and the topology alignment loss, respectively.

[0049] After the historical driving scenario is constructed through topological semantic scene graph and embedded into a scene, the corresponding scene potential vector is obtained. This vector, along with the original topological semantic scene graph, scene physical state, historical path, speed information, and other driving priors, is stored in the scene primitive vector database in the topological semantic memory, thus forming a searchable, sortable, and reusable topological semantic memory.

[0050] As a preferred embodiment of the autonomous driving planning method with enhanced topological semantic memory described in this invention, the specific method of S3 is as follows: during the online operation phase, a current topological semantic scene map is generated based on the current perception results. The current query vector is obtained by embedding the model through scene alignment. Suppose that the first... Each scene vector is The distance between the current scene and the historical scene is then represented as:

[0051]

[0052] according to Sort by size from smallest to largest and select the first few. One scenario is used as a candidate prior:

[0053]

[0054] in, Indicates the number of scenes in the memory bank. Indicates the historical prior quantity of the recall;

[0055] After recalling historical priors, the current visual input, perception results, navigation commands, vehicle speed, and historical scene priors are uniformly input into the driving planning model. The visual input is processed by a visual encoder to obtain multi-frame or multi-view visual features, which are then combined, downsampled, flattened, and linearly projected to form a visual word sequence. :

[0056]

[0057] in, Indicates the number of visual lexical units. Indicates lexical dimension;

[0058] The navigation target point and the vehicle speed are mapped into command words by the command encoder. :

[0059]

[0060] in, This indicates the features corresponding to the target point or route instructions. Indicates the vehicle's speed and other motion characteristics;

[0061] The historical scene priors obtained from the recall are mapped to search terms through a linear projection layer or a multilayer perceptron. :

[0062]

[0063] Finally, visual terms, instruction terms, and retrieval terms are concatenated into a unified input sequence. :

[0064]

[0065] The input sequence simultaneously includes the current visual environment, navigation intent, vehicle motion state, and historical driving priors, providing a complete context for the subsequent driving planning model to generate path and speed curves.

[0066] As a preferred embodiment of the autonomous driving planning method with topological semantic memory enhancement described in this invention, the model training in step S4 includes three stages: scene alignment embedding model training, visual perception module training, and driving planning model training.

[0067] As a preferred embodiment of the topological semantic memory-enhanced autonomous driving planning method described in this invention, the specific method of S4 is as follows: after receiving the fused input sequence, the driving planning model introduces two types of learnable query vectors: path query vector and speed query vector; the path query is used to extract information related to future spatial paths, and the speed query is used to extract information related to future longitudinal speed changes; the model inference process is represented as follows:

[0068]

[0069] in, This indicates the hidden state corresponding to the path query. This indicates the hidden state corresponding to the speed query; subsequently, the future path is output through the path decoder and speed decoder respectively. and velocity curve :

[0070]

[0071]

[0072] Finally, the planned trajectory of the vehicle over a future period is generated. :

[0073] .

[0074] As a preferred embodiment of the topological semantic memory-enhanced autonomous driving planning method described in this invention, the training loss of the visual perception module is expressed as follows:

[0075]

[0076] in, This indicates the predicted loss based on the traffic element category. This represents the state regression loss of traffic elements. and Indicates the weighting coefficient;

[0077] The training loss of the driving planning model is expressed as:

[0078]

[0079] in, Indicates the predicted path point. Represents the actual path points. Indicates the predicted speed. Indicates actual speed. and These represent the weights of path loss and velocity loss, respectively.

[0080] Compared with the prior art, the beneficial effects of the present invention are:

[0081] A topological semantic memory representation for driving planning has been constructed. This invention transforms raw visual perception information into a topological semantic scene graph containing the vehicle, surrounding traffic participants, lane structure, traffic signals, and their interactions, and stores historical driving scenarios in a scene memory database in the form of a structured graph representation. Compared to directly storing raw images or implicit visual features, this method can explicitly preserve key traffic elements, spatial relationships, and road topological constraints that influence driving decisions, giving historical driving experience an organized, measurable, and reusable structured representation. This invention transforms raw visual perception information into a compact topological semantic scene graph and encodes it into a searchable low-dimensional scene representation, avoiding large-scale retrieval of raw images or high-dimensional visual features, which helps reduce storage costs and online retrieval latency.

[0082] This invention achieves more efficient and accurate historical scene prior recall. Instead of relying solely on original image features or visual appearance similarity for historical scene matching, it uses a low-dimensional representation and topological consistency retrieval based on a topological semantic scene graph. On one hand, this approach avoids large-scale online retrieval of high-dimensional visual features, reducing storage overhead and inference latency. On the other hand, it effectively distinguishes scenes with similar visual appearances but different traffic signals, lane connections, interaction relationships, or driving logic, thereby improving the accuracy of matching the recalled scene prior with the current planning task.

[0083] This invention proposes a scene alignment embedding model for topological consistency. The model encodes the topological semantic scene graph into a compact, low-dimensional vector representation, ensuring that the vector distances in the embedding space remain consistent with the topological structure, temporal evolution, and interaction logic between scenes. This embedding model not only preserves the structural semantic information in the scene graph but also enables the historical scene database to possess searchable, sortable, and measurable vectorized indexing capabilities, thereby providing more reliable historical driving priors for subsequent driving planning models.

[0084] This invention implements a driving planning reasoning mechanism that integrates current perception with historical prior knowledge. During the reasoning phase, the invention uniformly inputs current visual information, navigation status, vehicle motion status, and recalled historical driving prior knowledge into the driving planning model, and generates future paths and speed curves through path queries and speed queries, respectively. This approach transforms the driving planning process from being driven solely by current perception to being jointly driven by "current scenario constraints + historical experience priors," enabling the generation of safer, smoother, and traffic-logically consistent future trajectories in complex interactive scenarios and long-tailed hazardous scenarios, thereby improving the stability and interpretability of autonomous driving planning results. Attached Figure Description

[0085] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0086] Figure 1 This is an overall framework diagram of an autonomous driving planning method with enhanced topological semantic memory according to the present invention.

[0087] Figure 2 This is a schematic diagram illustrating the construction of a visual-to-topological-semantic scene graph in an autonomous driving planning method with enhanced topological semantic memory according to the present invention.

[0088] Figure 3 This is a scene alignment embedding model structure diagram of an autonomous driving planning method with topological semantic memory enhancement according to the present invention;

[0089] Figure 4 This is a structural diagram of the driving planning model for an autonomous driving planning method with enhanced topological semantic memory according to the present invention.

[0090] Figure 5 This is an example of a method for autonomous driving planning enhanced with topological semantic memory according to the present invention, showing a hazard map of the side lanes, wherein (a) is a failed planning map of the method without using topological semantic memory, and (b) is a successful planning map of the method of the present invention.

[0091] Figure 6 This is an example of a left turn without traffic lights in an embodiment of an autonomous driving planning method enhanced with topological semantic memory according to the present invention, wherein (a) is a planning failure diagram of the method without using topological semantic memory, and (b) is a planning success diagram of the method of the present invention.

[0092] Figure 7 This is a comparison chart of retrieval time and driving performance under different retrieval methods in an embodiment of the autonomous driving planning method with topological semantic memory enhancement of the present invention. Detailed Implementation

[0093] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0094] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0095] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0096] like Figure 1 As shown, the method of this invention is executed in four steps: "construction of topological semantic scene graph, construction of scene alignment embedding and memory bank, retrieval of topological semantic memory, and output of driving plan". The system first converts the visual information currently perceived by the vehicle into a topological semantic scene graph, and then encodes the scene graph into a scene latent vector through a scene alignment embedding model, and stores it in or queries the scene primitive vector database in the topological semantic memory bank; in the online driving plan stage, the system recalls historical scene priors that are consistent with the current driving logic based on the current scene latent vector, and inputs the priors, current visual information, navigation instructions and vehicle status into the driving plan model, and finally outputs the future planned path and speed curve.

[0097] The topological semantic memory of this invention refers to a structured memory bank composed of a topological semantic scene graph of a historical driving scenario, scene latent vectors, scene physical states, historical planned paths, and speed information. The scene primitive vector database is used to store and retrieve scene latent vectors and their associated historical driving priors. This memory bank does not simply store raw images or high-dimensional visual features, but rather preserves key traffic elements, topological relationships, and historical driving priors that influence driving decisions, thereby providing searchable, measurable, and reusable external experience for driving planning in complex scenarios.

[0098] Specifically, an autonomous driving planning method enhanced with topological semantic memory includes the following steps:

[0099] Step 1: Construct a topological semantic scene graph for the current driving scenario: First, obtain the vehicle's current input information, including visual input, navigation command information, and vehicle status information. Visual input can be images from a forward-facing camera, multiple consecutive frames of images, or images from multiple viewpoint cameras; navigation command information can include target points, route directions, or high-level navigation commands; vehicle status information can include current position, speed, heading angle, acceleration, braking status, and lane number, etc. Let the current visual input be... Navigation instruction information is The vehicle status is Historical driving priors recalled from the topological semantic memory are The driving planning process can then be represented as:

[0100] in, This represents a topological semantic memory-enhanced driving planning model. This represents the future planning result output by the model. This planning result can be further decomposed into two parts: the future path and the future velocity.

[0101]

[0102] in, Indicates a spatial path point at several future time steps. This represents the velocity curve or longitudinal velocity planning result for the corresponding time step.

[0103] like Figure 2 As shown, the system identifies key elements related to driving decisions from visual input, including the vehicle itself, nearby vehicles, traffic signs, traffic lights, and lane elements, and establishes scene primitives centered on the vehicle. Current moment Scene primitives Defined as:

[0104]

[0105] in, This represents a set of interactive elements, which includes at least the vehicle itself, nearby vehicles, traffic signs, traffic lights, and lane elements. It represents a set of semantic relations, including at least guiding relations, active relations, inactive relations, and dependency relations; It represents a set of physical states, used to describe the position, size, speed, heading angle, state category, and distance of various interactive elements.

[0106] Based on the above scenario primitives, a topological semantic scenario graph is constructed. :

[0107]

[0108] in, This represents a set of nodes, where each node corresponds to a traffic element. This represents a set of edges, where each edge corresponds to a semantic relationship or topological constraint between two traffic elements. For example, a guidance relationship can be established between the vehicle and the vehicle in front, a dependency relationship can be established between the vehicle and the current lane, and an active constraint relationship can be established between a traffic light and a controlled lane. This step transforms the raw visual information into a structured, interpretable, and computable topological semantic representation, providing a unified data format for subsequent scene memory construction and prior retrieval.

[0109] Step 2: Construct a scene alignment embedding model and a topological semantic memory: such as Figure 3As shown, this invention constructs a scene alignment embedding model to encode a topological semantic scene graph into a low-dimensional scene latent vector. The network structure of this model includes a topology relation-aware encoder, a temporal attention pooling layer, and a graph structure decoder. The topology relation-aware encoder includes a relation graph convolutional layer, a Transformer layer, an attention pooling layer, and a linear mapping layer, used to generate scene latent vectors based on node attributes, edge relationships, and temporal changes in consecutive frames. The graph structure decoder includes a linear mapping layer, a Sigmoid activation layer, and an edge relationship prediction layer, used to reconstruct the topological semantic scene graph based on the scene latent vectors. It should be noted that the generation of the scene latent vector is part of the encoder output process, and the topology alignment constraint is implemented through training loss; these two are not set separately as independent structural units. For continuous driving scenarios, the input can be represented as a graph sequence composed of topological semantic scene graphs at multiple time steps.

[0110]

[0111] in, These represent the time intervals from the first moment to the second moment in a continuous driving scenario. The topological semantic scene graph constructed at each moment.

[0112] The topology-aware encoder first models different types of traffic relationships through a relationship graph convolutional layer. For the topological semantic scene graph... The encoder, based on the semantic relationship type, interaction strength, and adjacency relationship between nodes, weights and aggregates the state information of associated traffic elements, and integrates the node's own state to obtain an updated node representation. Layer node feature update can be represented as:

[0113]

[0114] in, Represents a node In the Features of the layer Represents a set of relation types. Indicates a relationship Next and Node The set of connected neighbor nodes, Representing relations The corresponding learnable weights, Indicates the self-join weight. Indicates the strength of interaction between nodes. Represents the normalization coefficient. This represents the activation function.

[0115] After obtaining graph structure features, the topology-aware encoder models the temporal changes between continuous scenes through a Transformer layer and aggregates structural information from different time steps and different traffic elements through an attention pooling layer. For the , One point of attention is:

[0116]

[0117]

[0118] in, These represent the query input features, key input features, and value input features, respectively, that are input to the attention layer. , , They represent the first The learnable linear projection matrix corresponding to each attention head; , and They represent the first The query matrix, key matrix, and value matrix in each attention head. This represents the output of the m-th attention head. This is used to scale the attention score, preventing the dot product result from becoming too large when the feature dimension is large.

[0119] The outputs of multiple attention heads are concatenated and linearly mapped to obtain the latent scene vector used for retrieval and alignment:

[0120]

[0121] in, They represent The output of each attention head; This indicates the number of attention heads in a multi-head attention mechanism; This represents the learnable linear mapping matrix after the multi-head attention output.

[0122] To avoid losing crucial topological information during low-dimensional encoding, this invention reconstructs a topological semantic scene graph from scene latent vectors using a graph structure decoder, and sets a graph structure reconstruction loss. This graph structure decoder takes scene latent vectors as input and predicts edge relationships and semantic constraints between nodes through a linear mapping layer and a sigmoid activation layer, thereby obtaining the reconstructed topological semantic scene graph.

[0123]

[0124] in, The graph structure reconstruction loss is used to ensure that the topological semantic scene graph reconstructed by the graph structure decoder based on the scene latent vectors remains consistent with the real topological semantic scene graph. Indicates batch size. Indicates the first The number of frames in a scene segment Represents the set of edges in the prediction graph. Represents the set of edges in the real graph. This indicates the first [section / part] in the scene segment. Each time step.

[0125] To ensure that the distances in the latent vector space of a scene reflect the topological differences between different driving scenarios, this invention further sets up a topological alignment loss during the training phase:

[0126]

[0127] in, This represents the topology alignment loss, used to constrain the distance in the scene's latent vector space to remain consistent with the difference in the real scene's topology. This indicates the number of scene segments in the training batch; and These represent the indices of two different scene segments within the batch; and They represent the first The scene fragment and the first The latent vector of a scene obtained by encoding a scene fragment using a scene alignment embedding model; Indicates the first The scene graph sequence and the first The graph structure dynamic temporal warping distance between a sequence of scene graphs is used to measure the real differences between two consecutive driving scenes in terms of topology, temporal evolution, and interaction.

[0128] The total training loss of the scene alignment embedding model is:

[0129]

[0130] in, and These represent the weights of the graph structure reconstruction loss and the topology alignment loss, respectively.

[0131] After the historical driving scenario is constructed through topological semantic scene graph and embedded into a scene, the corresponding scene potential vector is obtained. This vector, along with the original topological semantic scene graph, scene physical state, historical path, speed information, and other driving priors, is stored in the scene primitive vector database in the topological semantic memory, thus forming a searchable, sortable, and reusable topological semantic memory.

[0132] Step 3: Recall historical priors based on the topological semantic memory and fuse them with planning input: During the online operation phase, the system generates the current topological semantic scene graph based on the current perception results. The current query vector is obtained by embedding the model through scene alignment. Let the first element in the topological semantic memory be... Each scene vector is Then the distance between the current scene and the historical scene can be expressed as:

[0133]

[0134] according to Sort by size from smallest to largest and select the first few. One scenario is used as a candidate prior:

[0135]

[0136] in, Indicates the number of scenes in the memory bank. This indicates the number of prior historical scenes recalled. The retrieval process is based on low-dimensional scene latent vectors, which avoids the storage and computational overhead of directly retrieving original images or high-dimensional visual features. At the same time, because the scene latent vectors are subject to topological consistency constraints, the system can prioritize recalling historical scenes with similar driving logic, rather than scenes that are simply visually similar.

[0137] like Figure 4 As shown, after recalling historical priors, the system inputs the current visual input, perception results, navigation commands, vehicle speed, and historical scene priors into the driving planning model. The visual input is processed by a visual encoder to obtain multi-frame or multi-view visual features, which are then combined, downsampled, flattened, and linearly projected to form a visual word sequence. :

[0138]

[0139] in, Indicates the number of visual lexical units. Indicates the lexical dimension.

[0140] The navigation target point and the vehicle speed are mapped into command words by the command encoder. :

[0141]

[0142] in, This indicates the features corresponding to the target point or route instructions. It indicates the vehicle's speed and other motion characteristics.

[0143] The historical scene priors obtained from the recall are mapped to search terms through a linear projection layer or a multilayer perceptron. :

[0144]

[0145] Finally, visual terms, instruction terms, and retrieval terms are concatenated into a unified input sequence. :

[0146]

[0147] The input sequence simultaneously includes the current visual environment, navigation intent, vehicle motion state, and historical driving priors, providing a complete context for the subsequent driving planning model to generate path and speed curves.

[0148] Step 4: Generate future path and velocity curves and complete model training: After receiving the fused input sequence, the driving planning model introduces two types of learnable query vectors: path query vector and velocity query vector. The path query is used to extract information related to the future spatial path, and the velocity query is used to extract information related to future longitudinal velocity changes. The model inference process can be represented as follows:

[0149]

[0150] in, This indicates the hidden state corresponding to the path query. This indicates the hidden state corresponding to the speed query. Subsequently, the future path is output by the path decoder and speed decoder respectively. and velocity curve :

[0151]

[0152]

[0153] Finally, the planned trajectory of the vehicle over a future period is generated. :

[0154]

[0155] This decoupled planning method can model the lateral path changes and longitudinal speed changes of the vehicle separately, making the planning results easier to control and helping to generate safe, smooth future driving trajectories that meet the constraints of the current scenario.

[0156] Model training can be conducted in stages. The first stage involves training the scene alignment embedding model to generate latent vectors that maintain structural information and topological consistency. The second stage involves training the visual perception module to predict the traffic element categories and physical states required for scene graph construction from the current visual input. The third stage involves training the driving planning model to output the future path and speed curve after fusing visual terms, instruction terms, and retrieval terms.

[0157] The training loss of the visual perception module can be expressed as:

[0158]

[0159] in, Indicates the predicted loss based on the category of traffic elements. This represents the state regression loss of traffic elements. and This represents the weighting coefficient.

[0160] The training loss of the driving planning model can be expressed as:

[0161]

[0162] in, Indicates the predicted path point. Represents the actual path points. Indicates the predicted speed. Indicates actual speed. and These represent the weights of path loss and speed loss, respectively. Through the above training method, this invention can simultaneously ensure the effectiveness of topological semantic memory representation, the accuracy of historical prior recall, and the reliability of the final driving planning result.

[0163] Example:

[0164] This embodiment uses a closed-loop autonomous driving simulation platform to implement and verify the invention. The experimental method is as follows: First, visual input, navigation target, vehicle speed, and future trajectory labels during vehicle driving are collected, and historical driving segments are constructed into a sequence of topological semantic scene graphs. Each scene graph uses the vehicle, nearby vehicles, traffic lights, traffic signs, and lane elements as nodes, and guiding relationships, active relationships, non-active relationships, and dependent relationships as edges, recording the position, size, speed, heading angle, state category, lane number, and distance from the vehicle for each type of node. Then, the historical scene graph sequence is input into a scene alignment embedding model to obtain a low-dimensional scene potential vector, which is stored together with the corresponding historical path, speed information, and physical state in the scene primitive vector database of the topological semantic memory. During online operation, the system constructs the current perception result into a topological semantic scene graph, obtains the current query vector through the scene alignment embedding model, and recalls historical driving priors consistent with the current scene topology from the database. The recall result, along with the current visual lexical and navigation / speed command lexical, is input into the driving planning model, and the future planned path and speed curve are output by the path query and speed query, respectively.

[0165] In one specific implementation, the visual encoder employs CLIP-ViT, and the driving planning model uses a lightweight Transformer decoding structure. The scene primitive vector database in the topological semantic memory contains 800,000 historical scene primitives, and each online planning iteration recalls the top-10 relevant historical scene priors from the database. The number of path queries is set to 20, the number of speed queries is set to 10, and the number of multi-head attention heads is set to 8. The training process adopts a phased approach: first, the scene alignment embedding model is trained to obtain scene latent vectors that maintain topological consistency; then, the perception module is trained to predict traffic element categories and physical states from visual input; finally, the driving planning model is trained to generate future path and speed curves based on visual primitives, instruction primitives, and retrieval primitives. Figure 5 and Figure 6 By selecting driving segments from typical scenarios in closed-loop testing, including dangerous obstacles, complex traffic flow, and unsignalized intersections, and under the same initial state, navigation target, and simulation environment, the method without topological semantic memory was compared. Figure 5 (a) and Figure 6 (a) and the method of the present invention Figure 5 (b) and Figure 6 (b) The planning results were compared.

[0166] Table 1 shows the closed-loop driving performance results obtained by uniformly evaluating this invention with methods such as AD-MLP, TCP, UniAD-Base, VAD, ORION, and Simlingo under the same closed-loop test route, the same set of traffic scenarios, and the same evaluation indicators. The evaluation indicators include driving score and success rate.

[0167] Table 1: Comparison Results of Closed-Loop Driving Performance

[0168]

[0169] As shown in Table 1, the present invention achieved a driving score of 89.12 and a success rate of 70.42% in closed-loop driving evaluation, which is higher than other comparative methods in the table. This result indicates that by introducing topological semantic memory into the driving planning model, the system can more effectively utilize historical driving priors and generate more reliable planning results in complex interaction scenarios and long-tailed hazardous scenarios.

[0170] Depend on Figure 5 and Figure 6 It is known that in scenarios with dangerous obstacles, complex traffic flow, or turns at unsignalized intersections, methods without topological semantic memory are prone to trajectory deviation or unstable decision-making. This invention, however, combines current perception information with historical scenario priors to generate safer trajectories that better fit the constraints of the traffic scenario. To further verify the effectiveness of the historical prior recall method and the embedded model training objective, this embodiment, while maintaining the driving planning model structure, training data, and closed-loop test route unchanged, sets four ablation configurations: not using historical priors, using original visual feature similarity retrieval, using topological semantic scene graph retrieval but only employing graph reconstruction training, and using the complete method of this invention. Table 2 is derived from these configurations.

[0171] Table 2: Comparison Results of Different Retrieval Methods and Embedding Model Training Methods

[0172]

[0173] As shown in Table 2, while visual similarity retrieval alone can bring some improvement, the improvement is limited. Using topological semantic scene graph retrieval significantly improves driving scores and success rates. Further introducing topological alignment constraints yields the best results. This demonstrates that the present invention can not only achieve low-dimensional historical prior recall but also more accurately recall historical scenes consistent with the current driving logic.

[0174] To evaluate the relationship between real-time performance and planning performance under different retrieval database sizes, this embodiment varies the size of the topological semantic memory database and statistically analyzes the low-dimensional vector retrieval time based on the topological semantic scene graph, the corresponding driving score, and the driving score retrieved based on the original visual feature similarity. This yields... Figure 6 .

[0175] Depend on Figure 7 It is known that a larger historical scene database typically provides richer driving priors, which helps improve planning performance, but also incurs additional retrieval time overhead. This invention transforms the original high-dimensional visual retrieval into low-dimensional topological semantic vector retrieval through topological semantic scene graph compression and scene-aligned embedding, enabling the system to achieve a better balance between real-time performance and planning performance. Therefore, this result further demonstrates that this invention not only improves the accuracy of historical prior recall but also enhances the engineering feasibility of the retrieval-enhanced driving planning method.

[0176] To verify the contribution of the current perceptual graph prior and the three-stage training strategy to the final planning performance, this embodiment removes the current perceptual graph prior or changes the training process from three stages to two stages, while keeping other network structures, training data and evaluation protocols consistent, and obtains Table 3 under the same closed-loop testing protocol.

[0177] Table 3: Comparison Results of Graph Prior and Training Strategies

[0178]

[0179] As shown in Table 3, using the current perception map as an explicit semantic anchor input to the system can further improve the matching degree between historical priors and the current scene; the three-stage training strategy can also significantly improve the stability of the final planning result. In summary, the above experimental results demonstrate that this invention, through the collaborative design of topological semantic scene graph construction, scene alignment embedding, historical prior recall, and driving planning, achieves efficient utilization of historical driving experience and improves the planning performance and safety reliability of the autonomous driving system in complex traffic scenarios.

[0180] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for autonomous driving planning enhanced with topological semantic memory, characterized in that, Includes the following steps: S1: Obtain the vehicle's current input information and identify traffic elements through a visual encoder and perception module; Using traffic elements as nodes, and guiding, active, inactive, and dependent relationships as edges, and recording the physical state of vehicles, a topological semantic scene graph is formed. The input information includes visual input, navigation command information, and vehicle status information. The visual input is a forward-view camera image, a multi-frame continuous image, or a multi-view camera image. The navigation command information includes the target point, route direction, or high-level navigation command. The vehicle status information includes the current position, speed, heading angle, acceleration, braking status, and lane number. The specific method is as follows: Let the current visual input be... Navigation instruction information is The vehicle status is Historical driving priors recalled from the topological semantic memory are The driving planning process is then represented as: in, This represents a topological semantic memory-enhanced driving planning model. This represents the future planning result output by the model; the planning result is decomposed into two parts: future path and future velocity. in, Indicates a spatial path point at several future time steps. This represents the velocity curve or longitudinal velocity planning result for the corresponding time step. Identify key elements relevant to driving decisions from visual input, including the vehicle itself, nearby vehicles, traffic signs, traffic lights, and lane elements, and establish scene primitives centered on the vehicle at the current moment. Scene primitives Defined as: in, This represents a set of interactive elements, which includes at least the vehicle itself, nearby vehicles, traffic signs, traffic lights, and lane elements. It represents a set of semantic relations, including at least leading relations, active relations, inactive relations, and dependency relations; It represents a set of physical states, used to describe the position, size, speed, heading angle, state category, and distance of various interactive elements; Based on the above scenario primitives, a topological semantic scenario graph is constructed. : in, This represents a set of nodes, where each node corresponds to a traffic element. This represents a set of edges, where each edge corresponds to a semantic relationship or topological constraint between two traffic elements; S2: Input the topological semantic scene graph into the scene alignment embedding model. The topological relation-aware encoder models traffic elements and their semantic relationships, and the graph structure decoder reconstructs the topological semantic scene graph. During the training phase, graph structure reconstruction loss and topological alignment loss are used to ensure that the scene latent vectors maintain consistency in topological structure, temporal evolution, and driving logic. In the offline phase, historical scene latent vectors, the original topological semantic scene graph, scene physical state, historical path, and speed information are stored as driving priors in the scene primitive vector database of the topological semantic memory. The network structure of the scene alignment embedding model includes a topology-aware encoder, a temporal attention pooling layer, and a graph structure decoder. The topology-aware encoder includes a relation graph convolutional layer, a Transformer layer, an attention pooling layer, and a linear mapping layer, which are used to generate scene latent vectors based on node attributes, edge relationships, and temporal changes in consecutive frames. The graph structure decoder includes a linear mapping layer, a Sigmoid activation layer, and an edge relationship prediction layer, which are used to reconstruct a topological semantic scene graph based on the scene latent vectors. Specifically, for continuous driving scenarios, the input is represented as a graph sequence consisting of multiple time-step topological semantic scene graphs: in, These represent the time intervals from the first moment to the second moment in a continuous driving scenario. The topological semantic scene graph constructed at each moment; The topology-aware encoder first models different types of traffic relationships through a relation graph convolutional layer. For the topological semantic scene graph, the first... The topology-aware encoder, based on the semantic relationship type, interaction strength, and adjacency relationship between nodes, weights and aggregates the state information of associated traffic elements, and integrates the node's own state to obtain an updated node representation; Layer node feature update is represented as: in, Represents a node In the Features of the layer Represents a set of relation types. Indicates a relationship Next and Node The set of connected neighbor nodes, Representing relations The corresponding learnable weights, Indicates the self-join weight. Indicates the strength of interaction between nodes. Represents the normalization coefficient. Indicates the activation function; After obtaining the graph structure features, the topology-aware encoder models the temporal changes between continuous scenes through the Transformer layer, and aggregates the structural information of different time steps and different traffic elements through the attention pooling layer; for the th One point of attention is: in, These represent the query input features, key input features, and value input features, respectively, that are input to the attention layer. , , They represent the first The learnable linear projection matrix corresponding to each attention head; , and They represent the first The query matrix, key matrix, and value matrix in each attention head; This represents the output of the m-th attention head. This is used to scale the attention score to avoid excessively large dot product results when the feature dimension is large; The outputs of multiple attention heads are concatenated and linearly mapped to obtain the scene latent vector used for retrieval and alignment. : in, They represent The output of each attention head; This indicates the number of attention heads in a multi-head attention mechanism; This represents the learnable linear mapping matrix after multi-head attention output; A graph structure decoder reconstructs a topological semantic scene graph from the scene latent vectors, and a graph structure reconstruction loss is set. The graph structure decoder takes the scene latent vectors as input and predicts the edge relationships and semantic constraints between nodes through a linear mapping layer and a sigmoid activation layer, thereby obtaining the reconstructed topological semantic scene graph. in, The graph structure reconstruction loss is used to ensure that the topological semantic scene graph reconstructed by the graph structure decoder based on the scene latent vectors remains consistent with the real topological semantic scene graph. Indicates batch size. Indicates the first The number of frames in a scene segment Represents the set of edges in the prediction graph. Represents the set of edges in the real graph. This indicates the first [section / part] in the scene segment. One time step; Set the topology alignment loss during the training phase: in, This represents the topology alignment loss, used to constrain the distance in the scene's latent vector space to remain consistent with the difference in the real scene's topology. This indicates the number of scene segments in the training batch; and These represent the indices of two different scene segments within the batch; and They represent the first The scene fragment and the first The latent vector of a scene obtained by encoding a scene fragment using a scene alignment embedding model; Indicates the first The scene graph sequence and the first The graph structure dynamic temporal warping distance between a sequence of scene graphs is used to measure the real differences between two consecutive driving scenes in terms of topology, temporal evolution, and interaction relationships. Total training loss of scene alignment embedding model for: in, and These represent the weights of the graph structure reconstruction loss and the topology alignment loss, respectively. After the historical driving scenario is constructed through topological semantic scenario graph and embedded into the scenario, the corresponding scenario potential vector is obtained. This vector, along with the original topological semantic scenario graph, scenario physical state, historical path, speed information, and other driving priors, is stored in the scenario primitive vector database in the topological semantic memory, thus forming a searchable, sortable, and reusable topological semantic memory. S3: During online runtime, the current topological semantic scene graph is encoded into a query vector, and the top-K historical scene priors that are similar to the current scene in terms of traffic topology, interaction relationships and driving logic are retrieved from the topological semantic memory. The current vision, navigation target, vehicle speed and historical scene priors are concatenated into a unified input for the driving planning model. S4: The driving planning model extracts spatial path information and longitudinal speed information based on the fused input, and outputs the future path and speed curve through the path decoder and speed decoder to obtain the planned trajectory of the vehicle in the future period of time, thus completing the model training.

2. The autonomous driving planning method with topological semantic memory enhancement according to claim 1, characterized in that, The specific method of S3 is as follows: during the online operation phase, a current topological semantic scene map is generated based on the current perception results. The current query vector is obtained by embedding the model through scene alignment. ; Suppose that the first... Each scene vector is The distance between the current scene and the historical scene is then represented as: according to Sort by size from smallest to largest and select the first few. One scenario is used as a candidate prior: in, Indicates the number of scenes in the memory bank. Indicates the historical prior quantity of the recall; After recalling historical priors, the current visual input, perception results, navigation commands, vehicle speed, and historical scene priors are uniformly input into the driving planning model. The visual input is processed by a visual encoder to obtain multi-frame or multi-view visual features, which are then combined, downsampled, flattened, and linearly projected to form a visual word sequence. : in, Indicates the number of visual lexical units. Indicates lexical dimension; The navigation target point and the vehicle speed are mapped into command words by the command encoder. : in, This indicates the features corresponding to the target point or route instructions. Indicates the vehicle's speed and other motion characteristics; The historical scene priors obtained from the recall are mapped to search terms through a linear projection layer or a multilayer perceptron. : Finally, visual terms, instruction terms, and retrieval terms are concatenated into a unified input sequence. : The input sequence simultaneously includes the current visual environment, navigation intent, vehicle motion state, and historical driving priors, providing a complete context for the subsequent driving planning model to generate path and speed curves.

3. The autonomous driving planning method with topological semantic memory enhancement according to claim 1, characterized in that, In S4, model training includes three stages: scene alignment embedding model training, visual perception module training, and driving planning model training.

4. The autonomous driving planning method with topological semantic memory enhancement according to claim 3, characterized in that, The specific method of S4 is as follows: after receiving the fused input sequence, the driving planning model introduces two types of learnable query vectors: path query vector and speed query vector; the path query is used to extract information related to future spatial paths, and the speed query is used to extract information related to future longitudinal speed changes; the model inference process is represented as follows: in, This indicates the hidden state corresponding to the path query. This indicates the hidden state corresponding to the speed query; subsequently, the future path is output through the path decoder and speed decoder respectively. and velocity curve : Finally, the planned trajectory of the vehicle over a future period is generated. : 。 5. The autonomous driving planning method with topological semantic memory enhancement according to claim 3, characterized in that, The training loss of the visual perception module is expressed as: in, Indicates the predicted loss based on the category of traffic elements. This represents the state regression loss of traffic elements. and Indicates the weighting coefficient; The training loss of the driving planning model is expressed as: in, Indicates the predicted path point. Represents the actual path points. Indicates the predicted speed. Indicates actual speed. and These represent the weights of path loss and velocity loss, respectively.

Images