A topological reasoning method, system, device and storage medium for driving scenarios

Abstract

The embodiments of the present application relate to the field of autonomous driving technology, and in particular to a topological reasoning method, system, device and storage medium for driving scenes, the method comprising: extracting multi-scale image features; various types of instances obtain information from the multi-scale image features, and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; obtain embeddings of traffic element instance feature vectors, and interact and exchange messages between instance feature vectors and embeddings of traffic element instance feature vectors through scene topological relationships to obtain and aggregate important knowledge; based on the scene knowledge graph, knowledge extraction is performed on traffic element instances of different categories, and topological maps of traffic elements, lane centerlines and driving scenes are predicted respectively. The present application proposes a topological reasoning network, generates high-quality driving scene topology by predicting instances (lanes, traffic elements) and relationships between instances in the driving scene, and predicts the driving scene topology in an end-to-end manner.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of autonomous driving technology, and in particular to a topological reasoning method, system, device, and storage medium for driving scenarios. Background Art

[0002] The driving system is a complex, multi-module collaborative system. Understanding the topology of the driving scene is crucial for achieving autonomous driving. Driving scene topology encompasses two aspects: the connectivity between lanes and the matching between lanes and traffic elements (such as traffic lights and road signs). Existing autonomous driving systems rely heavily on pre-generated high-precision maps to capture the driving scene topology. In the absence of high-precision maps, end-to-end learning methods that can fully understand the driving scene topology are underdeveloped.

[0003] While existing HD map learning technologies can predict the spatial coordinate representations of some HD map elements (including lane dividers, road edges, and crosswalks), deriving lane connectivity and road topology from HD map elements requires complex post-processing. Furthermore, the assignment of traffic elements to lanes is often performed in the monocular image domain. Establishing the relationship between traffic elements in a monocular image and lane centerlines in a surround-view 3D coordinate system remains a challenging problem. Summary of the Invention

[0004] The embodiments of the present application provide a topological reasoning method, system, device and storage medium for driving scenarios. By proposing a topological reasoning network, high-quality driving scenario topology is generated by predicting instances (lanes, traffic elements) in the driving scenario and the relationships between instances, and the driving scenario topology is predicted in an end-to-end manner.

[0005] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a topological reasoning method for driving scenes, comprising the following steps: extracting multi-scale image features based on multi-view surround camera images; based on a deformable attention mechanism, various types of instances obtain information from the multi-scale image features, and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; based on a traffic element embedding network, obtain embeddings of traffic element instance feature vectors; and based on a scene graph neural network, the feature vectors of instances and the embeddings of traffic element instance feature vectors interact and exchange messages with each other through scene topological relationships to obtain and aggregate important knowledge and obtain traffic element instances of different categories; based on a scene knowledge graph, knowledge extraction is performed on traffic element instances of different categories to obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors; based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors, the topological maps of traffic elements, lane centerlines and driving scenes are predicted respectively.

[0006] In some exemplary embodiments, based on the processed traffic element instance feature vector and the updated lane centerline instance feature vector, the traffic elements, lane centerlines and the topology map of the driving scene are predicted respectively, including: using a traffic element prediction head to predict the 2D bounding box of the traffic element and the confidence probability of each category in the traffic element; using a lane centerline prediction head to predict the ordered point set of the lane centerline and the confidence probability of the lane centerline; using a topology prediction head to predict the topological relationship between lanes and between lanes and traffic elements.

[0007] In some exemplary embodiments, a shared feature extractor is used to extract multi-scale image features from multi-view surround camera images; the multi-scale image features include front view image features and bird's-eye view features; wherein the bird's-eye view features are constructed by a view converter.

[0008] In some exemplary embodiments, the embedding of the traffic element instance feature vector is shown in formula (1):

[0009]

[0010] Where, embedding represents the weight; i represents the i-th decoder layer;

[0011] Based on the scene graph neural network, the calculation process of obtaining different categories of traffic element instances is shown in formula (2):

[0012]

[0013]

[0014]

[0015]

[0016] Among them, G ll Represents the topological graph between lanes; G lt Represents the topology between lanes and traffic elements.

[0017] In some exemplary embodiments, for a topological graph between lanes and traffic elements, the computation process for obtaining a processed traffic element instance feature vector and an updated lane centerline instance feature vector is as shown in formula (3):

[0018]

[0019]

[0020] Among them, β lt β is a hyperparameter that determines the strength of feature propagation between traffic elements and lane centerlines. lt =0.5.

[0021] In some exemplary embodiments, for a topological map between lanes, the calculation process of obtaining the processed traffic element instance feature vector and the updated lane centerline instance feature vector is as shown in formula (4):

[0022]

[0023]

[0024] Among them, β lt β is a hyperparameter that determines the strength of feature propagation between lane centerlines. lt =0.5.

[0025] In some exemplary embodiments, the traffic element embedding network includes a multi-layer perceptron; the traffic element embedding network is used to filter out redundant 2D image information in the instance feature vector of the traffic element, abstract and extract the traffic rule semantic information contained in the instance feature vector of the traffic element, and align the feature space with the lane centerline.

[0026] In a second aspect, an embodiment of the present application also provides a topological reasoning system for a driving scene, comprising a feature acquisition module, a deformable decoding module, a scene graph neural network module, a knowledge extraction module, and a prediction module connected in sequence; the feature acquisition module is used to extract multi-scale image features based on multi-view surround camera images; the deformable decoding module is used to obtain information from multi-scale image features for various types of instances based on a deformable attention mechanism, and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; the scene graph neural network module is used to obtain embeddings of traffic element instance feature vectors based on traffic element embedding networks; and based on the scene graph neural network, the feature vectors of instances and the embeddings of traffic element instance feature vectors interact and exchange messages with each other through scene topological relationships to obtain and aggregate important knowledge and obtain traffic element instances of different categories; the knowledge extraction module is used to perform knowledge extraction on traffic element instances of different categories based on the scene knowledge graph to obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors; the prediction module is used to predict the topological maps of traffic elements, lane centerlines, and driving scenes respectively based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors.

[0027] In addition, the present application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned topological reasoning method for the driving scenario.

[0028] In addition, the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the topological reasoning method for the above-mentioned driving scenario.

[0029] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0030] An embodiment of the present application provides a topological reasoning method, system, device and storage medium for driving scenes, the method comprising the following steps: extracting multi-scale image features based on multi-view surround camera images; based on a deformable attention mechanism, various types of instances obtain information from the multi-scale image features to obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; based on a traffic element embedding network, obtaining embeddings of traffic element instance feature vectors; and based on a scene graph neural network, interacting and exchanging messages between the feature vectors of instances and the embeddings of traffic element instance feature vectors through scene topological relationships to obtain and aggregate important knowledge to obtain traffic element instances of different categories; based on a scene knowledge graph, performing knowledge extraction on traffic element instances of different categories to obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors; based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors, predicting the topological maps of traffic elements, lane centerlines and driving scenes respectively.

[0031] This application proposes a Topology Reasoning Network (TopoNet) to predict driving scene topology in an end-to-end manner. To reason about scene topology in a single network, TopoNet consists of two branches, one for predicting traffic elements and the other for predicting lane centerlines, which share a backbone network for extracting image features. This application uses a Transformer Decoder-based architecture to predict traffic elements and lane centerline instances, and uses a multi-layer perceptron to predict driving scene topology.

[0032] Because lane centerlines contain road topology, their topology information can be implicitly encoded in their predecessor and successor centerlines and related traffic elements. Therefore, this application designs a Scene Graph Neural Network (SGNN) to pass messages between the feature vectors of each instance. To better extract abstract traffic rule knowledge from traffic elements, this application proposes a Traffic Element Embedding Network.

[0033] In addition, this application proposes a scene knowledge graph (SKG) to capture abstract traffic rules and topological knowledge from different types of traffic element instances and lane centerline instances with different connection relationships. Specifically, based on the different categories of traffic elements (such as no left turn, etc.) and lane centerline connection relationships (such as successor, predecessor, etc.), this application designs a scene graph neural network that transmits information on the scene knowledge graph. By updating the feature vectors of each instance, the perceptual prediction quality of each instance and the reasoning quality of the driving scene topology are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] One or more embodiments are exemplarily described by the pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0035] Figure 1 A flowchart of a topological reasoning method for a driving scenario provided in one embodiment of the present application;

[0036] Figure 2 A schematic diagram of the structure of a topology reasoning network provided in one embodiment of the present application;

[0037] Figure 3 A schematic diagram of a module of a topological reasoning system for a driving scenario provided in one embodiment of the present application;

[0038] Figure 4 A schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0039] As we can see from the background, while existing HD map learning technologies can predict the spatial coordinate representations of some HD map elements (including lane dividers, road edges, and crosswalks), deriving lane connectivity and road topology from HD map elements requires complex post-processing. Furthermore, the assignment of traffic elements to lanes is often performed in the monocular image domain, and establishing the relationship between traffic elements in a monocular image and lane centerlines in a 3D coordinate system remains a challenging problem.

[0040] Currently, autonomous driving is a hot topic in both industry and academia. With the popularity of bird's-eye view perception, current work in high-definition map perception focuses on learning HD map elements through segmentation and vectorization methods. HD map segmentation aims to predict the semantics of each bird's-eye view grid, such as lane lines, crosswalks, and drivable areas. Vectorization methods abstract map information into a set of sparse polylines. This process explicitly models the spatial relationships between HD map elements and produces a vectorized map that is friendly to downstream autonomous driving tasks. One related technique proposes directly representing each map element as a series of vectorized spatial points and using a decoder similar to DETR to predict the key points of the map element. An autoregressive model is then used to recover the detailed outline of the map element from the key points. Another related technique directly uses the DETR decoder to predict the vectorized spatial points of the map element and removes ambiguity in the modeling of the map element through permutation matching during the training phase. Recently, a related technique proposes constructing map elements as a graph structure by first predicting vertices and then detecting edges using a graph neural network module. However, this graph neural network generates all vertex features simultaneously, resulting in a lack of instance-level interaction. There are also technologies that propose a DETR-like network to detect lane centerlines and their preceding and succeeding relationships.

[0041] Driving scene understanding primarily involves summarizing the positional relationships between elements in an outdoor environment. In industry, some have proposed an optimization method for automatically constructing lane topology and traffic light-lane relationships based on internal data. In academia, some are using convolutional networks to incorporate heterogeneous metadata as additional input to solve the traffic light-to-lane assignment (TL2LA) problem.

[0042] To solve the above technical problems, the embodiments of the present application provide a topological reasoning method, system, device and storage medium for driving scenes, including the following steps: extracting multi-scale image features based on multi-view surround camera images; based on the deformable attention mechanism, various types of instances obtain information from the multi-scale image features, and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; based on the traffic element embedding network, obtain the embedding of traffic element instance feature vectors; and based on the scene graph neural network, the feature vectors of the instances and the embeddings of the traffic element instance feature vectors interact and exchange messages with each other through the scene topological relationship, obtain and aggregate important knowledge, and obtain traffic element instances of different categories; based on the scene knowledge graph, knowledge extraction is performed on traffic element instances of different categories to obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors; based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors, the topological maps of traffic elements, lane centerlines and driving scenes are predicted respectively. An embodiment of the present application provides a topological reasoning method for driving scenarios. By proposing a topological reasoning network, high-quality driving scenario topology is generated by predicting instances (lanes, traffic elements) in the driving scenario and the relationships between instances, and the driving scenario topology is predicted in an end-to-end manner.

[0043] The following detailed description of the various embodiments of the present application is provided in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that many technical details are provided in the various embodiments of the present application to facilitate a better understanding of the present application. However, even without these technical details and the various variations and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0044] See Figure 1 , an embodiment of the present application provides a topological reasoning method for a driving scenario, comprising the following steps:

[0045] Step S1: extracting multi-scale image features based on multi-view surround camera images.

[0046] Step S2: Based on the deformable attention mechanism, various types of instances obtain information from multi-scale image features and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively.

[0047] Step S3: Based on the traffic element embedding network, the embedding of the traffic element instance feature vector is obtained; and based on the scene graph neural network, the feature vector of the instance and the embedding of the traffic element instance feature vector interact and exchange messages with each other through the scene topology relationship, acquire and aggregate important knowledge, and obtain traffic element instances of different categories.

[0048] Step S4: Based on the scene knowledge graph, knowledge extraction is performed on traffic element instances of different categories to obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors.

[0049] Step S5: Based on the processed traffic element instance feature vector and the updated lane centerline instance feature vector, the topology of the traffic element, the lane centerline, and the driving scene are predicted respectively.

[0050] This application proposes a topology reasoning network (TopoNet) to predict driving scene topology in an end-to-end manner. This is the first end-to-end framework that uses surround-view vehicle sensors to understand complex driving scene topologies. The topology reasoning network TopoNet generates high-quality driving scene topology by predicting instances (lanes, traffic elements) in the driving scene and the relationships between instances. This application evaluated the TopoNet provided by this application on the challenging driving scene understanding dataset OpenLane-V2. In all perception and topology metrics, the performance of the driving scene topology reasoning method provided by this application is ahead of previous methods.

[0051] In some embodiments, a shared feature extractor is used in step S1 to extract multi-scale image features from the multi-view surround camera image; the multi-scale image features include front view image features and bird's-eye view features; wherein the bird's-eye view features are constructed by a view converter.

[0052] This application provides a topological reasoning network (TopoNet) that solves the topological reasoning problem of driving scenarios in an end-to-end manner. Figure 2 Figure 2 shows a schematic diagram of the structure of a topology reasoning network. Figure 2 As shown, given a multi-view surround camera image as input, the shared feature extractor generates multi-scale image features, including the front view image feature F PV And the bird's-eye view feature F BEV , where the bird's-eye view feature is constructed by the view transformer (View Transform).

[0053] After obtaining the front view image feature F PV And the bird's-eye view feature F BEVNext, the feature encoding phase begins. Specifically, in the deformable decoder phase, TopoNet processes traffic elements (TE) and lane centerlines (LC) as two parallel branches. Various types of instance queries first acquire information from the feature graph through a deformable attention mechanism. Then, in the scene graph neural network (SGNN) phase, instance queries interact and exchange messages through scene topology relationships, acquiring and aggregating important knowledge.

[0054] like Figure 2 As shown in the figure, Traffic Element Query represents the feature vector of the instance corresponding to the traffic element; Centerline Query represents the feature vector of the instance corresponding to the lane centerline; Adjacency Matrix is ​​the adjacency matrix.

[0055] In some embodiments, the traffic element embedding network includes a multi-layer perceptron; the traffic element embedding network is used to filter redundant 2D image information in the feature vector of the instance of the traffic element, abstract and extract the traffic rule semantic information contained in the feature vector of the instance of the traffic element, and align the feature space with the center line of the lane.

[0056] Since traffic elements are often marked on 2D front-view images, the query of traffic elements contains a large number of features from the front-view image F. PV The 2D image information of the traffic element instance Query has a feature space that is different from the bird's-eye view features of the lane centerline, and this information is unnecessary for lane centerline prediction. Therefore, this application introduces an embedding network (EmbeddingNetwork), which consists of a two-layer multi-layer perceptron (MLP). It aims to filter out the redundant 2D image information in the traffic element instance Query, abstract and extract the traffic rule semantic information it contains, and align the feature space with the lane centerline.

[0057] In some embodiments, the embedding of the traffic element instance feature vector (Traffic Element Query) is shown in formula (1):

[0058]

[0059] Where, embedding represents the weight; i represents the i-th decoder layer;

[0060] After the embedding of the feature vector of the traffic element instance is calculated, the SGNN operation process is performed. In step S3, the calculation process of obtaining traffic element instances of different categories based on the scene graph neural network is shown in formula (2):

[0061]

[0062]

[0063]

[0064]

[0065] Among them, G ll Represents the topological graph between lanes; G lt Represents the topology between lanes and traffic elements.

[0066] For different categories of traffic element instances, this application proposes a scene knowledge graph (SKG), which implements different learnable weights for knowledge extraction. lt The adjacency matrix A lt , the learnable weight matrix W for different classifications lt , and the classification confidence S of the traffic element instance t , SGNN lt The operation process is shown in the following formula (3). ll , which is calculated as shown in the following formula (4).

[0067] In some embodiments, for a topological graph between lanes and traffic elements, the computation process for obtaining a processed traffic element instance feature vector and an updated lane centerline instance feature vector is as shown in formula (3):

[0068]

[0069]

[0070] Among them, β lt β is a hyperparameter that determines the strength of feature propagation between traffic elements and lane centerlines. lt =0.5.

[0071] In some embodiments, for the topological map between lanes, the calculation process of obtaining the processed traffic element instance feature vector and the updated lane centerline instance feature vector is as shown in formula (4):

[0072]

[0073]

[0074] Among them, β lt β is a hyperparameter that determines the strength of feature propagation between lane centerlines. lt =0.5.

[0075] In some embodiments, in step S5, based on the processed traffic element instance feature vector and the updated lane centerline instance feature vector, the traffic elements, lane centerlines and the topology map of the driving scene are predicted respectively, including: using a traffic element prediction head to predict the 2D bounding box of the traffic element and the confidence probability of each category in the traffic element; using a lane centerline prediction head to predict the ordered point set of the lane centerline and the confidence probability of the lane centerline; using a topology prediction head to predict the topological relationship between lanes and between lanes and traffic elements.

[0076] After obtaining the processed traffic element Query and the lane centerline Query updated by SGNN, this application first uses the traffic element prediction head (TE Head) to predict the 2D bounding box of the traffic element and the confidence probability of each classification, and then uses the lane centerline prediction head (LC Head) to predict the ordered point set of the lane centerline and its confidence probability. Finally, this application uses the topology prediction head (Topology Head) to predict the topological relationship between lanes and between lanes and traffic elements. This application predicts traffic elements, lane centerlines, and driving scene topology maps after each layer of Decoder, and supervises them with labels. During reasoning, this application only uses the output of the last layer of Decoder as the output result of TopoNet.

[0077] This application proposes a topological reasoning network (TopoNet), which is the first method to achieve comprehensive driving scene understanding using on-board sensors. The core innovations of TopoNet are: (1) It uses an end-to-end approach to simultaneously predict multi-classified traffic elements, lane centerlines, and driving scene topology. (2) It uses a scene graph neural network and scene knowledge graph to transfer knowledge between traffic elements and lane centerlines.

[0078] Existing high-precision map learning technologies (VectorMapNet, MapTR, etc.) can only predict some high-precision map elements and cannot fully represent road topology information. The existing TL2LA method also does not solve the problem of matching lane lines in surround view images with traffic elements in front view images. Compared with the existing technology, the topological reasoning method for driving scenes provided by the embodiment of the present application solves the lane centerline perception problem, traffic element recognition problem, and driving scene topology reasoning problem between lanes and traffic elements and between lanes in an end-to-end manner in a single network for the first time through TopoNet. In addition, TopoNet uses scene graph neural networks and scene knowledge graphs to extract abstract traffic rule knowledge of traffic elements and improve the reasoning performance of driving scene topology.

[0079] See Figure 3 The embodiment of the present application also provides a topological reasoning system for a driving scene, including a feature acquisition module 101, a deformable decoding module 102, a scene graph neural network module 103, a knowledge extraction module 104, and a prediction module 105 connected in sequence; the feature acquisition module 101 is used to extract multi-scale image features based on multi-view surround camera images; the deformable decoding module 102 is used to obtain information from various types of instances of the multi-scale image features based on the deformable attention mechanism, and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; the scene graph neural network module 103 is used to obtain instances of traffic elements based on the traffic element embedding network Embedding of feature vectors; and based on the scene graph neural network, the feature vectors of the instance and the embedding of the feature vectors of the traffic element instance interact and exchange messages with each other through the scene topological relationship, acquire and aggregate important knowledge, and obtain traffic element instances of different categories; the knowledge extraction module 104 is used to extract knowledge from traffic element instances of different categories according to the scene knowledge graph, and obtain the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors; the prediction module 105 is used to predict the topology of traffic elements, lane centerlines and driving scenes respectively according to the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors.

[0080] To verify the feasibility and effectiveness of the proposed driving scene topology reasoning method and system, experiments were conducted on the challenging OpenLane-V2 dataset, which covers complex urban scenes and includes 960K instance-level annotations and 891K topological relationship labels. This dataset contains 1,000 scenes lasting approximately 15 seconds, including multi-view images from 7 cameras and annotations at 2Hz.

[0081] This application migrated the previous state-of-the-art method to the OpenLane-V2 dataset under a fair setting, and added traffic element prediction and driving scene topology prediction branches. The experimental results are shown in the following table:

[0082]

[0083] Among them, DET l represents the performance of lane centerline detection, DET t Represents the performance of traffic element prediction, TOP ll Represents the prediction performance of the topological relationship between lanes, TOP lt The OLS metric represents the prediction performance of the topological relationship between lanes and traffic elements. The OLS metric is a weighted sum of the above four evaluation metrics, reflecting the overall performance of the model for this task. As can be seen, TopoNet demonstrates comprehensive performance improvements compared to previous methods.

[0084] See Figure 4 Another embodiment of the present application provides an electronic device, comprising: at least one processor 110; and a memory 111 communicatively connected to the at least one processor; wherein the memory 111 stores instructions that can be executed by the at least one processor 110, and the instructions are executed by the at least one processor 110 so that the at least one processor 110 can execute any of the above method embodiments.

[0085] The memory 111 and the processor 110 are connected using a bus. The bus may include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors 110 and the memory 111. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor 110 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits the data to the processor 110.

[0086] The processor 110 is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 111 may be used to store data used by the processor 110 when performing operations.

[0087] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0088] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the above-mentioned methods of each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0089] Based on the above technical solutions, the embodiments of the present application provide a topological reasoning method, system, device and storage medium for driving scenes, which include the following steps: extracting multi-scale image features based on multi-view surround camera images; based on the deformable attention mechanism, various types of instances obtain information from the multi-scale image features, and obtain feature vectors of instances corresponding to traffic elements and lane centerlines respectively; based on the traffic element embedding network, obtain the embedding of the traffic element instance feature vector; and based on the scene graph neural network, the feature vector of the instance and the embedding of the traffic element instance feature vector interact and exchange messages with each other through the scene topological relationship, obtain and aggregate important knowledge, and obtain traffic element instances of different categories; based on the scene knowledge graph, perform knowledge extraction on traffic element instances of different categories, and obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors; based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors, predict the topological maps of traffic elements, lane centerlines and driving scenes respectively.

[0090] This application proposes a Topology Reasoning Network (TopoNet) to predict driving scene topology in an end-to-end manner. To reason about scene topology in a single network, TopoNet consists of two branches, one for predicting traffic elements and the other for predicting lane centerlines, which share a backbone network for extracting image features. This application uses a Transformer Decoder-based architecture to predict traffic elements and lane centerline instances, and uses a multi-layer perceptron to predict driving scene topology.

[0091] Because lane centerlines contain road topology, their topology information can be implicitly encoded in their predecessor and successor centerlines and related traffic elements. Therefore, this application designs a Scene Graph Neural Network (SGNN) to pass messages between the feature vectors of each instance. To better extract abstract traffic rule knowledge from traffic elements, this application proposes a Traffic Element Embedding Network.

[0092] In addition, this application proposes a scene knowledge graph (SKG) to capture abstract traffic rules and topological knowledge from different types of traffic element instances and lane centerline instances with different connection relationships. Specifically, based on the different categories of traffic elements (such as no left turn, etc.) and lane centerline connection relationships (such as successor, predecessor, etc.), this application designs a scene graph neural network that transmits information on the scene knowledge graph. By updating the feature vectors of each instance, the perceptual prediction quality of each instance and the reasoning quality of the driving scene topology are improved.

[0093] Those skilled in the art will appreciate that the above-described embodiments are specific examples for implementing the present application, and that in actual applications, various changes in form and detail may be made thereto without departing from the spirit and scope of the present application. Any person skilled in the art may make changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined in the claims.

Claims

1. A topological reasoning method for driving scenarios, characterized in that: include: Extract multi-scale image features based on multi-view surround camera images; Based on the deformable attention mechanism, various types of instances obtain information from the multi-scale image features to obtain feature vectors of instances corresponding to the traffic elements and the lane centerlines, respectively. Based on the traffic element embedding network, the embedding of the traffic element instance feature vector is obtained; and based on the scene graph neural network, the feature vector of the instance and the embedding of the traffic element instance feature vector interact with each other through the scene topology relationship, exchange messages, obtain and aggregate important knowledge, and obtain traffic element instances of different categories; Based on the scene knowledge graph, knowledge is extracted from different categories of traffic element instances to obtain the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors; Based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors, the topology of traffic elements, lane centerlines, and driving scenarios are predicted respectively. The embedding of the traffic element instance feature vector is shown in formula (1): （1） in, represents the embedding of the feature vector of the traffic element instance; embedding represents the weight of the traffic element embedding network; i represents the i-th decoder layer; t represents the traffic element instance; Represents the feature vector of traffic element instance before embedding; Based on the scene graph neural network, the calculation process of obtaining different categories of traffic element instances is shown in formula (2): （2） in, Represents the original lane instance feature vector; ( ) represents the lane-lane scenario knowledge graph network; represents the lane instance feature vector after lane-lane scene graph enhancement; Represents the topological graph between lanes; represents the topology graph between lanes at the i-1th decoder layer; ( ) represents the lane-traffic element scene graph network; Represents the lane instance feature vector after being enhanced by the lane-traffic element scene graph network; Represents the topological graph between lanes and traffic elements; Represents the topological graph between lanes and traffic elements at the i-1th decoder layer; represents the residual term; ( ) represents the channel downsampling operation; ReLU ( ) represents the linear rectification function; concat ( , ) represents the dimension superposition operation; represents the lane instance feature vector after the final scene graph enhancement; For the topological graph between lanes and traffic elements, the calculation process of obtaining the processed traffic element instance feature vector and the updated lane centerline instance feature vector is shown in formula (3): （3） in, Knowledge graph between lanes and traffic elements; Represents the relationship adjacency matrix between lanes and traffic elements at the i-1th decoder layer; represents the lane instance feature after being enhanced by the lane-traffic element knowledge graph; y is the traffic element node; N ( x ) is a set of traffic element nodes; c t It is the category of traffic elements; C t It is a collection of traffic element categories; A hyperparameter that determines how strongly features are propagated between traffic elements and lane centerlines. =0.5; represents the classification confidence of traffic element node y; Represents the edge weight between x and y in the knowledge graph; Represents the trainable weights of the knowledge graph; Represents the embedding of the feature vector of the traffic element y instance; For the topological map between lanes, the calculation process of obtaining the processed traffic element instance feature vector and the updated lane centerline instance feature vector is shown in formula (4): （4） in, Represents the knowledge graph between lanes; stack() represents the feature stacking operation; represents the lane-lane relationship adjacency matrix of the i-1th decoder layer; represents the lane-lane adjacency matrix of the transposed i-1th decoder layer; I Connect the array to the unit; Represents the lane instance features after being enhanced by the lane-lane knowledge graph; c l is the connection relationship category; C l is the lane connection relationship category set; A hyperparameter that determines how strongly features propagate between lane centerlines. =0.5; represents the knowledge graph edge weight between lane x and lane y in the cth connection relationship; Represents learnable parameters in the lane-to-lane knowledge graph; Represents the instance feature vector of lane y.

2. The topological reasoning method for driving scenarios according to claim 1, characterized in that: Based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors, the topology of traffic elements, lane centerlines, and driving scenarios are predicted, including: Using a traffic element prediction head, predict the 2D bounding box of the traffic element and the confidence probability of each category in the traffic element; Using a lane centerline prediction head, an ordered set of points of the lane centerline and a confidence probability of the lane centerline are predicted; A topology prediction head is used to predict the topological relationships between lanes and between lanes and traffic elements.

3. The topological reasoning method for driving scenarios according to claim 1, characterized in that: A shared feature extractor is used to extract multi-scale image features from multi-view surround camera images; The multi-scale image features include front view image features and bird's-eye view features; wherein the bird's-eye view features are constructed by a perspective converter.

4. The topological reasoning method for driving scenarios according to claim 1, characterized in that: The traffic element embedding network includes a multi-layer perceptron; the traffic element embedding network is used to filter redundant 2D image information in the instance feature vector of the traffic element, abstract and extract the traffic rule semantic information contained in the instance feature vector of the traffic element, and align the feature space with the lane centerline.

5. A topological reasoning system for driving scenarios, characterized in that: It includes a feature acquisition module, a deformable decoding module, a scene graph neural network module, a knowledge extraction module and a prediction module connected in sequence; The feature acquisition module is used to extract multi-scale image features based on the multi-view surround camera image; The deformable decoding module is used to obtain information from the multi-scale image features based on the deformable attention mechanism for various types of instances, and obtain feature vectors of instances corresponding to the traffic elements and the lane centerlines respectively; The scene graph neural network module is used to obtain the embedding of the traffic element instance feature vector based on the traffic element embedding network; and based on the scene graph neural network, the feature vector of the instance and the embedding of the traffic element instance feature vector interact with each other through the scene topological relationship, exchange messages, obtain and aggregate important knowledge, and obtain traffic element instances of different categories; The knowledge extraction module is used to extract knowledge from different categories of traffic element instances based on the scene knowledge graph to obtain processed traffic element instance feature vectors and updated lane centerline instance feature vectors; The prediction module is used to predict the topology of traffic elements, lane centerlines and driving scenes respectively based on the processed traffic element instance feature vectors and the updated lane centerline instance feature vectors; The embedding of the traffic element instance feature vector is shown in formula (1): （1） Where embedding represents the weight; i represents the i-th decoder layer; t represents the traffic element instance; Represents the feature vector of traffic element instance before embedding; Based on the scene graph neural network, the calculation process of obtaining different categories of traffic element instances is shown in formula (2): （2） in, Represents the original lane instance feature vector; ( ) represents the lane-lane scenario knowledge graph network; represents the lane instance feature vector after lane-lane scene graph enhancement; Represents the topological graph between lanes; represents the topology graph between lanes at the i-1th decoder layer; ( ) represents the lane-traffic element scene graph network; Represents the lane instance feature vector after being enhanced by the lane-traffic element scene graph network; Represents the topological graph between lanes and traffic elements; Represents the topological graph between lanes and traffic elements at the i-1th decoder layer; represents the residual term; ( ) represents the channel downsampling operation; ReLU ( ) represents the linear rectification function; concat ( , ) represents the dimension superposition operation; represents the lane instance feature vector after the final scene graph enhancement; For the topological graph between lanes and traffic elements, the calculation process of obtaining the processed traffic element instance feature vector and the updated lane centerline instance feature vector is shown in formula (3): （3） in, Knowledge graph between lanes and traffic elements; Represents the relationship adjacency matrix between lanes and traffic elements at the i-1th decoder layer; represents the lane instance feature after being enhanced by the lane-traffic element knowledge graph; y is the traffic element node; N ( x ) is a set of traffic element nodes; c t It is the category of traffic elements; C t It is a collection of traffic element categories; A hyperparameter that determines how strongly features are propagated between traffic elements and lane centerlines. =0.5; represents the classification confidence of traffic element node y; Represents the edge weight between x and y in the knowledge graph; Represents the trainable weights of the knowledge graph; Represents the embedding of the feature vector of the traffic element y instance; For the topological map between lanes, the calculation process of obtaining the processed traffic element instance feature vector and the updated lane centerline instance feature vector is shown in formula (4): （4） in, Represents the knowledge graph between lanes; stack() represents the feature stacking operation; represents the lane-lane relationship adjacency matrix of the i-1th decoder layer; represents the lane-lane adjacency matrix of the transposed i-1th decoder layer; I Connect the array to the unit; Represents the lane instance features after being enhanced by the lane-lane knowledge graph; c l is the connection relationship category; C l is the lane connection relationship category set; A hyperparameter that determines how strongly features propagate between lane centerlines. =0.5; represents the knowledge graph edge weight between lane x and lane y in the cth connection relationship; Represents learnable parameters in the lane-to-lane knowledge graph; Represents the instance feature vector of lane y.

6. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to execute the topological reasoning method for a driving scenario as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the topology reasoning method for a driving scenario according to any one of claims 1 to 4 is implemented.

Images