A traffic light and lane association method and device based on scene coding
By employing scene coding technology and utilizing semantic vector data and graph convolutional networks, the high cost problem of traffic light and lane association was solved, achieving efficient and low-cost association prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN ZHONGHAITING DATA TECH CO LTD
- Filing Date
- 2023-08-10
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies for linking traffic lights and lanes in high-precision map production suffer from high production costs and difficulties in data acquisition. In particular, rule-based and traffic flow status-based methods require extensive manual annotation or costly data acquisition.
A scene-based encoding approach is adopted. By acquiring semantic vector data of lanes and traffic lights, the association relationship is established using domain knowledge and clustering methods. The association relationship between target traffic lights and lanes is predicted by combining graph convolutional networks and attention mechanisms.
It reduces the cost of high-precision map production and model training, decreases data requirements and model complexity, and improves data utilization and prediction accuracy.
Smart Images
Figure CN117034034B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of high-precision map production and deep learning technology, specifically relating to a method and apparatus for associating traffic lights and lanes based on scene coding. Background Technology
[0002] In the field of autonomous driving, high-precision maps are a crucial infrastructure. High-precision maps not only provide road information but also detailed map elements such as traffic lights, lane markings, and traffic signs. To ensure that autonomous vehicles can smoothly and safely comply with traffic rules, we need to understand the relationship between traffic lights and lanes. The following are some technical background and methodologies:
[0003] 1. Rule-based and manual map creation: The accuracy of traffic light and lane association depends on the quality of high-precision maps. Map creation requires capturing detailed information about the location, orientation, and lanes controlled by traffic lights. Data collected by sensors such as LiDAR and cameras can generate high-precision maps, including road geometry, lane lines, traffic lights, and ground arrows. Based on ground arrows and lane relationships at intersections, and supplemented by rules and manual analysis, this method has the main drawback of varying traffic light styles across different regions, making it difficult to apply uniform rules (traffic light settings are sometimes based on direction, sometimes on lanes, sometimes on non-motorized vehicles, sometimes on repeated settings for ease of observation, and sometimes on the interaction between main and auxiliary lanes, etc.). This increases the workload of manual processing and raises the cost of high-precision map creation.
[0004] 2. Associating traffic lights and lanes based on traffic flow status: Based on the traffic flow status at the intersection, find the correspondence between traffic light status and lane traffic flow, and find the lanes controlled by the traffic lights in the actual situation. The disadvantage of this method is that it requires obtaining traffic flow data at the intersection, and the cost of obtaining traffic flow data at a large number of intersections in major cities is high.
[0005] 3. When an autonomous vehicle passes through an intersection, it records the detection results of lanes and traffic lights in the image from the forward-facing camera. The relationships between lanes and traffic lights are then manually labeled to create training data. This data is used to train a deep learning model to establish the association between lanes and traffic lights. The drawback of this approach is that it requires a large amount of image annotation data from various cities. While the data acquisition cost is less than the second method, it is still relatively high. Summary of the Invention
[0006] To reduce the production cost and training cost of high-precision map models, a first aspect of this invention provides a method for associating traffic lights and lanes based on scene encoding, comprising: acquiring lane semantic vector data and traffic light semantic vector data; establishing an association between roads and traffic lights based on the lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods, and constructing a vector scene map based on the association; encoding the vector scene map based on graph convolutional networks and attention mechanisms; and predicting the association between target traffic lights and lanes based on the encoded vector scene map.
[0007] In some embodiments of the present invention, the step of establishing the association between roads and traffic lights based on the lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods, and constructing a vector scene map based on the association includes: grouping intersections and traffic lights respectively based on the lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods; associating the grouped intersection data with the traffic light data based on domain knowledge and Bayesian classification methods; and standardizing the associated intersection data and traffic light data to construct a vector scene map.
[0008] Furthermore, the process of standardizing the associated intersection data and traffic light data to construct a vector scene map includes: determining the origin and direction of each intersection; normalizing the data range of each intersection; and extracting the connection relationships of all lanes within each intersection.
[0009] In some embodiments of the present invention, encoding the vector scene map based on graph convolutional networks and attention mechanisms includes: performing local encoding of the vector scene map based on graph convolutional networks; and performing global encoding of the locally encoded vector scene map through an attention mechanism.
[0010] Furthermore, the local or global encoding is pre-trained using BERT.
[0011] In the above embodiments, predicting the correlation between the target traffic light and the lane based on the encoded vector scene map includes:
[0012] Based on the encoded vector scene map, the relationship matrix between the target traffic lights and lanes is predicted using preset correlation and activation functions.
[0013] A second aspect of the present invention provides a traffic light and lane association device based on scene encoding, comprising: an acquisition module for acquiring lane semantic vector data and traffic light semantic vector data; a construction module for establishing an association relationship between roads and traffic lights based on the lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods, and constructing a vector scene map according to the association relationship; an encoding module for encoding the vector scene map based on a graph convolutional network and an attention mechanism; and a prediction module for predicting the association relationship between target traffic lights and lanes based on the encoded vector scene map.
[0014] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the scene-coding-based traffic light and lane association method provided in the first aspect of the present invention.
[0015] In a fourth aspect, the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the scene-coding-based traffic light and lane association method provided in the first aspect of the present invention.
[0016] The beneficial effects of this invention are:
[0017] This invention relates to a method and apparatus for associating traffic lights and lanes based on scene encoding. The method includes: acquiring lane semantic vector data and traffic light semantic vector data; establishing an association between roads and traffic lights based on the lane semantic vector data and traffic light semantic vector data using domain knowledge and clustering methods, and constructing a vector scene map based on the association; encoding the vector scene map based on a graph convolutional network and attention mechanism; and predicting the association between target traffic lights and lanes based on the encoded vector scene map. As can be seen, this invention combines deep learning with domain knowledge of road conditions, using vector semantic data instead of image data, which reduces the data and samples required for training the scene encoding model, lowers the complexity of the model structure, and thus reduces the cost of model construction; consequently, it reduces the cost of producing high-precision maps. Attached Figure Description
[0018] Figure 1 This is a basic flowchart illustrating the traffic light and lane association method based on scene coding in some embodiments of the present invention.
[0019] Figure 2 This is a schematic diagram illustrating the specific process of the traffic light and lane association method based on scene coding in some embodiments of the present invention.
[0020] Figure 3 This is a schematic diagram illustrating the scene map encoding principle in some embodiments of the present invention;
[0021] Figure 4 Schematic diagram of a scene-coded traffic light and lane association device in some embodiments of the present invention;
[0022] Figure 5 Schematic diagrams of the structure of electronic devices in some embodiments of the present invention. Detailed Implementation
[0023] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0024] refer to Figure 1 and Figure 2 In a first aspect of the present invention, a method for associating traffic lights and lanes based on scene coding is provided, comprising: S100. Semantic vector data and traffic light semantic vector data; S200. Using traffic light semantic vector data and lane semantic vector data, establishing an association relationship between roads and traffic lights through domain knowledge and clustering methods, and constructing a vector scene map based on the association relationship; S300. Encoding the vector scene map using a network and attention mechanism; S400. Using the vector scene map, predicting the association relationship between target traffic lights and lanes.
[0025] In step S100 of some embodiments of the present invention, lane semantic vector data and traffic light semantic vector data are acquired; specifically, there are two types of input data:
[0026] 1) Lane-level topology data: This data expresses the connection relationships at the lane level, generally expressed by the geometry and continuity of the lane centerline.
[0027] 2) Traffic light data: This data mainly includes the location, orientation, and type (direction, circularity, and other features that can be extracted using target detection) of traffic lights;
[0028] Both of these types of data are semantic vector data. With proper design, the relationship between traffic lights and lanes can be inferred using relatively small local scene data, eliminating the need for large-scale images to label the relationship.
[0029] In step S200 of some embodiments of the present invention, the step of establishing the association between roads and traffic lights based on lane semantic vector data and traffic light semantic vector data, through domain knowledge and clustering methods, and constructing a vector scene map according to the association includes:
[0030] S201. Based on lane semantic vector data and traffic light semantic vector data, intersections and traffic lights are grouped separately using domain knowledge and clustering methods;
[0031] Specifically, intersection grouping: Based on domain knowledge of real-world road conditions, traffic lights often appear at right-of-way selection points, mostly located at intersections. Traffic light and lane data for all directions around an intersection can be processed uniformly with high efficiency. Furthermore, data from different intersections are independent of each other, allowing for parallel processing through a distributed approach to improve efficiency. For traffic lights not located at intersections, their virtual intersection locations can be processed together, following the same logic. A virtual intersection refers to an intersection with only one lane in and one lane out.
[0032] Traffic light grouping: Based on domain knowledge of real-world road conditions, traffic lights often appear side by side. To group traffic lights into groups based on their position and orientation, the clustering method can be DBSCAN (density-based clustering algorithm). The orientation of each group of traffic lights is consistent.
[0033] S202. Based on domain knowledge and Bayesian classification methods, the grouped intersection data is associated with traffic light data;
[0034] Specifically, based on domain knowledge of real-world road conditions, the road about to enter an intersection is often associated with a set of traffic lights on the opposite side. Therefore, based on this knowledge, the scope of the scene data can be further narrowed down by splitting the data from multiple directions at the intersection. Based on the direction of the road entering the intersection and the orientation of the traffic lights, the relationship between the road entering the intersection and the traffic light group can be inferred.
[0035] The specific implementation method is to use Bayesian classification. First, select some intersections to label the relationship between roads and traffic lights. The relationship is expressed by four attributes: road direction, traffic light orientation, angle, and distance. After the data is labeled, no training is required. On a new dataset, the Bayesian classification method is used to estimate the maximum likelihood probability of the relationship.
[0036] S203. Standardize the associated intersection data and traffic light data to construct a vector scene map.
[0037] It is understandable that once the relationship between roads and traffic lights is determined, scene data can be constructed based on the lane groups of the road, the relationship between lanes within the lane groups, and the traffic light groups. The scene data of a single road entering the intersection is much smaller than the original image data and the scene data of the entire intersection, eliminating unnecessary redundant information and greatly improving data utilization.
[0038] Furthermore, in step S203, the standardization processing of the associated intersection data and traffic light data to construct a vector scene map includes:
[0039] S2031. Determine the origin and direction of each intersection. Set the origin:
[0040] The origin is defined as the vertex of the leftmost lane entering the intersection (as shown in the attached diagram). Figure 2 (Point O); take the direction of the leftmost lane entering the intersection as the upward direction, and rotate all coordinates to that direction.
[0041] S2032. Normalize the data range for each intersection;
[0042] Specifically, the range of a regular intersection is 100m * 100m, and the coordinates are normalized by dividing by 100.
[0043] S2033. Extract the connection relationships of all lanes within each intersection.
[0044] Specifically, based on the lane-level topology, all the connections of the lane entering the intersection are extracted, including the lanes inside the intersection and the lanes outside the intersection. When using it, the lanes inside the intersection are removed, and the topology can be expressed directly using the lane entering the intersection itself and the connecting lanes outside the intersection. The lanes inside the intersection are not needed.
[0045] At this point, all vector scene data is ready, including lane group data entering the intersection, lanes outside the intersection associated with that lane group, and traffic light groups.
[0046] In step S300 of some embodiments of the present invention, encoding the vector scene map based on graph convolutional networks and attention mechanisms includes:
[0047] S301. Local encoding of vector scene maps based on graph convolutional networks;
[0048] S302. Global encoding is performed on the locally encoded vector scene map through an attention mechanism.
[0049] Specifically, the scene encoding of vector data consists of a two-layer graph structure:
[0050] 1) The first-layer subgraph structure consists of the relationships between lane lines entering the intersection, where lane line objects are vertices of the graph. The lane line entering the intersection and its associated lane lines outside the intersection form an edge. If a lane entering the intersection has two associated lanes outside the intersection, then that lane vertex has two edges. The encoding of the first-layer subgraph is performed by a GCN (Graph Convolutional Network).
[0051] 2) The second-layer global graph structure uses the result of the first-layer subgraph of the lane lines entering the intersection as vertices, and each traffic light as a vertex to construct a fully connected graph based on attention (attention mechanism), where all vertices are interconnected.
[0052] scene_embedding=global_graph(concat(sub_graph(L in ,L out ),TL))),
[0053] L in Lane markings indicating entry into the intersection;
[0054] L out Lane markings indicating the exit from the intersection;
[0055] TL stands for traffic light;
[0056] sub_graph represents the subgraph encoding;
[0057] concat means to merge vertices;
[0058] global_graph represents global encoding;
[0059] cene_embedding represents the scene encoding.
[0060] Furthermore, the local or global encoding is pre-trained using BERT. Scene encoding can also be pre-trained to improve performance. The pre-training method can refer to the mask method of BERT (Bidirectional Encoder Representation from Transformers), and the pre-training method will not be elaborated in this invention.
[0061] In step S400 of the above embodiment, predicting the association between the target traffic light and the lane based on the encoded vector scene map includes: predicting the relationship matrix between the target traffic light and the lane based on the encoded vector scene map and through a preset association function and activation function.
[0062] Specifically, based on scenario coding, the correlation between entry lanes and traffic lights at intersections is predicted.
[0063] A=sigmoid(link_prediction(scene_embedding)),
[0064] Here, `link_prediction` represents the function for predicting association relationships, which can be easily implemented using an MLP; `scene_embedding` represents the scene encoding; `sigmoid` represents the activation function used for binary classification; and `A` represents the association matrix (see attached). Figure 3 The matrix in the bottom right corner; the training loss function uses Focal Loss, which is used to handle classification problems and can also solve model performance issues caused by imbalanced data. If you want to further improve prediction performance, you can use a dynamic graph iteration method, but the essence is still relation prediction. The relation matrix is now obtained.
[0065] It should be noted that, Figure 3 In the diagram, L1–L7 represent lane centerlines, T1–T4 represent traffic lights, T1 is a directional arrow type, and T2, T3, and T4 are circular types. J1 represents an intersection, O represents the origin, and the dashed lines within the intersection represent lanes within the intersection, which are not included in the calculation. The Subgraph on the right represents the subgraph construction process, and the Globalgraph represents the global graph construction process. The matrix in the lower right corner represents the relationships between L1, L2, L3 and T1, T2, T3, T4.
[0066] Example 2
[0067] refer to Figure 4 In a second aspect, the present invention provides a traffic light and lane association device 1 based on scene encoding, comprising: an acquisition module 11 for acquiring lane semantic vector data and traffic light semantic vector data; a construction module 12 for establishing a road and traffic light association relationship based on the lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods, and constructing a vector scene map according to the association relationship; an encoding module 13 for encoding the vector scene map based on a graph convolutional network and an attention mechanism; and a prediction module 14 for predicting the association relationship between target traffic lights and lanes according to the encoded vector scene map.
[0068] Furthermore, the construction module 12 includes: a grouping unit, used to group intersections and traffic lights respectively based on lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods; an association unit, used to associate the grouped intersection data with the traffic light data based on domain knowledge and Bayesian classification methods; and a construction unit, used to standardize the associated intersection data and traffic light data to construct a vector scene map.
[0069] Example 3
[0070] refer to Figure 5A third aspect of the present invention provides an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the scene-coded traffic light and lane association method of the first aspect of the present invention.
[0071] Electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.
[0072] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, hard disks; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic device 500 with various devices is provided, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.
[0073] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by a processing device 501, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium described in embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0074] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more computer programs, which, when executed by the electronic device, cause the electronic device to:
[0075] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and Python—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0076] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for associating traffic lights and lanes based on scene coding, characterized in that, include: Acquire lane semantic vector data and traffic light semantic vector data; Based on the lane semantic vector data and traffic light semantic vector data, the association between roads and traffic lights is established through domain knowledge and clustering methods, and a vector scene map is constructed according to the association. Based on the lane semantic vector data and traffic light semantic vector data, intersections and traffic lights are grouped separately through domain knowledge and clustering methods. Based on domain knowledge and Bayesian classification methods, the grouped intersection data and traffic light data are associated. The correlated intersection and traffic light data are standardized to construct a vector scene map. The vector scene map is encoded using graph convolutional networks and an attention mechanism: local encoding of the vector scene map is performed using a graph convolutional network; then, global encoding of the locally encoded vector scene map is performed using an attention mechanism. Based on the encoded vector scene map, predict the relationship between the target traffic lights and lanes: Based on the encoded vector scene map, predict the relationship matrix between the target traffic lights and lanes through preset relationship functions and activation functions.
2. The method for associating traffic lights and lanes based on scene coding according to claim 1, characterized in that, The step of standardizing the associated intersection data and traffic light data to construct a vector scene map includes: Determine the origin and direction of each intersection; Normalize the data range for each intersection; Extract the connection relationships of all lanes within each intersection.
3. The method for associating traffic lights and lanes based on scene coding according to claim 1, characterized in that, The local or global encoding is pre-trained using BERT.
4. A traffic light and lane association device based on scene coding, characterized in that, include: The acquisition module is used to acquire lane semantic vector data and traffic light semantic vector data; The construction module is used to establish the association between roads and traffic lights based on the lane semantic vector data and traffic light semantic vector data, using domain knowledge and clustering methods, and to construct a vector scene map based on the association: based on the lane semantic vector data and traffic light semantic vector data, intersections and traffic lights are grouped separately using domain knowledge and clustering methods; based on domain knowledge and Bayesian classification methods, the grouped intersection data and traffic light data are associated. The correlated intersection and traffic light data are standardized to construct a vector scene map. The encoding module is used to encode the vector scene map based on graph convolutional networks and attention mechanisms: local encoding of the vector scene map based on graph convolutional networks; A global encoding of the locally encoded vector scene map is performed using an attention mechanism. The prediction module is used to predict the relationship between target traffic lights and lanes based on the encoded vector scene map: based on the encoded vector scene map, it predicts the relationship matrix between target traffic lights and lanes through preset relationship functions and activation functions.
5. An electronic device, comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the scene-coded association method for traffic lights and lanes as described in any one of claims 1 to 3.
6. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the scene-coding-based traffic light and lane association method as described in any one of claims 1 to 3.