Method, device and electronic equipment for predicting association of a marker with a lane

Abstract

A method, device and electronic equipment for predicting the association relationship between a marker and a lane are disclosed, and relate to the technical field of intelligent driving. The method comprises determining driving environment data collected by a vehicle during driving; determining input graph structure data for representing an initial relationship between a road marker and a lane based on the driving environment data; and processing the input graph structure data based on a graph neural network model to obtain an association relationship between a preset type of marker in the road marker and the lane. In the process of processing the input graph structure data by the graph neural network model, the technical solution of the disclosure does not need to manually design threshold parameters and algorithm processes, thereby improving the efficiency and accuracy of determining the association relationship between the traffic light and the lane, and ensuring safe driving.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, and in particular to a method, apparatus, and electronic device for predicting the relationship between signs and lanes. Background Technology

[0002] Currently, vehicles in assisted or autonomous driving mode can select the correct traffic light by determining the relationship between the traffic light and the lane, and then perform the corresponding response operation based on the signal emitted by that traffic light (used to indicate the status of the traffic light, such as whether the red light is on or the green light is on).

[0003] In existing technologies, the relationship between traffic lights and lanes is typically determined by manually designing threshold parameters and algorithm processes. This requires a significant amount of manual editing and verification, resulting in both a large workload and low efficiency. Summary of the Invention

[0004] Currently, determining the relationship between traffic lights and lanes by manually designing threshold parameters and algorithm processes requires a lot of manual editing and verification work, which is not only troublesome but also inefficient.

[0005] To address the aforementioned technical problems, this disclosure provides a method, apparatus, and electronic device for predicting the relationship between traffic lights and lanes, which can improve the efficiency and accuracy of determining the relationship between traffic lights and lanes, thereby ensuring safe driving.

[0006] The first aspect of this disclosure provides a method for predicting the association between road signs and lanes, comprising: determining driving environment data collected by a vehicle during driving; determining input graph structure data representing the initial relationship between road signs and lanes based on the driving environment data; and processing the input graph structure data based on a graph neural network model to obtain the association between road signs of a preset type and lanes.

[0007] A second aspect of this disclosure provides a method for training a graph neural network model, comprising: determining multiple sets of sample input graph structure data and sample road signs corresponding to the sample input graph structure data, and the sample association relationship between sample preset type signs and sample lanes; wherein the sample input graph structure data is used to represent the initial relationship between sample road signs and sample lanes; processing the sample input graph structure data based on the initial graph neural network model to obtain the predicted association relationship between sample preset type signs and sample lanes; using the predicted association relationship as the initial training output of the initial graph neural network model, and the sample association relationship as supervision information, iteratively training the initial graph neural network model to obtain the trained graph neural network model.

[0008] A third aspect of this disclosure provides an apparatus for predicting the association between road signs and lanes, comprising: a first determining module for determining driving environment data collected by a vehicle during driving; a second determining module for determining input graph structure data representing the initial relationship between road signs and lanes based on the driving environment data; and a prediction module for processing the input graph structure data based on a graph neural network model to obtain the association between preset type road signs and lanes.

[0009] A fourth aspect of this disclosure provides a graph neural network model training apparatus, comprising: a determining module, configured to determine sample association relationships between sample preset type signs and sample lanes in multiple sets of sample input graph structure data and sample road signs corresponding to the sample input graph structure data; wherein the sample input graph structure data is used to represent the initial relationship between sample road signs and sample lanes; a processing module, configured to process the sample input graph structure data based on the initial graph neural network model to obtain the predicted association relationship between sample preset type signs and sample lanes; and a training module, configured to use the predicted association relationship as the initial training output of the initial graph neural network model and the sample association relationship as supervision information to iteratively train the initial graph neural network model to obtain the trained graph neural network model.

[0010] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the method of predicting the association between markers and lanes described in the first aspect, or for performing the graph neural network model training method described in the second aspect.

[0011] A sixth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the method for predicting the association between markers and lanes as described in the first aspect, or a computer program for executing the graph neural network model training method as described in the second aspect.

[0012] In this embodiment of the disclosure, when determining the association between a preset type of road sign and a lane, it is only necessary to determine the input graph structure data representing the initial relationship between the road sign and the lane based on the driving environment data collected during vehicle operation. Then, the input graph structure data is processed by a graph neural network model to obtain the association between the preset type of road sign and the lane. Since the graph neural network model is a pre-trained model, there is no need to manually design threshold parameters and algorithm flows during the processing of the input graph structure data by the graph neural network model. Therefore, it can improve the efficiency and accuracy of determining the association between traffic lights and lanes, ensuring safe driving. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of a scenario where a vehicle is driving at a traffic intersection, provided by an exemplary embodiment of this disclosure.

[0014] Figure 2 This is a flowchart illustrating a method for predicting the relationship between signs and lanes, provided by an exemplary embodiment of this disclosure.

[0015] Figure 3 This is a schematic diagram of an input graph structure data provided in an exemplary embodiment of this disclosure.

[0016] Figure 4 This is a schematic diagram of an output graph structure data provided by an exemplary embodiment of this disclosure.

[0017] Figure 5 This is a flowchart illustrating another method for predicting the association between signs and lanes, provided by an exemplary embodiment of this disclosure.

[0018] Figure 6 This is a flowchart illustrating another method for predicting the relationship between a sign and a lane, provided by an exemplary embodiment of this disclosure.

[0019] Figure 7 This is a flowchart illustrating another method for predicting the relationship between a sign and a lane, provided by an exemplary embodiment of this disclosure.

[0020] Figure 8 This is a schematic diagram of a first graph structure data provided by an exemplary embodiment of this disclosure.

[0021] Figure 9 This is a schematic diagram of the composition structure of a second graph structure data provided by an exemplary embodiment of the present disclosure.

[0022] Figure 10 This is a flowchart illustrating another method for predicting the relationship between signs and lanes, provided by an exemplary embodiment of this disclosure.

[0023] Figure 11 This is a flowchart illustrating another method for predicting the association between a sign and a lane, provided by an exemplary embodiment of this disclosure.

[0024] Figure 12 This is a flowchart illustrating an exemplary embodiment of a graph neural network model training method provided in this disclosure.

[0025] Figure 13 This is a flowchart illustrating a method for determining multiple sets of sample input graph structure data provided by an exemplary embodiment of this disclosure.

[0026] Figure 14 This is a flowchart illustrating another method for determining multiple sets of sample input graph structure data provided by an exemplary embodiment of this disclosure.

[0027] Figure 15 This is a flowchart illustrating another method for determining multiple sets of sample input graph structure data provided by an exemplary embodiment of this disclosure.

[0028] Figure 16 This is a flowchart illustrating another method for determining multiple sets of sample input graph structure data provided by an exemplary embodiment of this disclosure.

[0029] Figure 17 This is a system block diagram of a graph neural network model provided in an exemplary embodiment of the present disclosure.

[0030] Figure 18 This is a flowchart illustrating another graph neural network model training method provided by an exemplary embodiment of this disclosure.

[0031] Figure 19 This is a flowchart illustrating another graph neural network model training method provided by an exemplary embodiment of this disclosure.

[0032] Figure 20 This is a schematic diagram of the composition of an apparatus for predicting the relationship between signs and lanes, provided in an exemplary embodiment of this disclosure.

[0033] Figure 21 This is a schematic diagram of the composition of another device for predicting the relationship between signs and lanes, provided by an exemplary embodiment of this disclosure.

[0034] Figure 22 This is a schematic diagram of the composition of another device for predicting the relationship between signs and lanes provided in an exemplary embodiment of this disclosure.

[0035] Figure 23 This is a schematic diagram of the composition of another device for predicting the relationship between signs and lanes provided in an exemplary embodiment of this disclosure.

[0036] Figure 24 This is a schematic diagram of the composition of an apparatus for predicting the relationship between a sign and a lane, provided in an exemplary embodiment of this disclosure.

[0037] Figure 25 This is a schematic diagram of the composition of another device for predicting the relationship between signs and lanes, provided in an exemplary embodiment of this disclosure.

[0038] Figure 26 This is a schematic diagram of the composition structure of a graph neural network model training device provided in an exemplary embodiment of the present disclosure.

[0039] Figure 27 This is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. Detailed Implementation

[0040] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0041] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0042] Application Overview

[0043] In this embodiment of the disclosure, a vehicle in assisted driving or autonomous driving mode can select the correct traffic light by determining the relationship between the traffic light and the lane, and then perform the corresponding response operation according to the signal emitted by the traffic light.

[0044] In related technologies, the relationship between traffic lights and lanes can be determined in real time on the vehicle side, or offline in the cloud, and then this relationship can be sent back to the vehicle side. Typically, the relationship between traffic lights and lanes can be determined manually by designing threshold parameters and algorithms. However, this method requires a significant amount of manual editing and verification, which is both labor-intensive and inefficient.

[0045] To address the issue of low efficiency in determining the relationship between traffic lights and lanes, this disclosure provides a method for predicting the relationship between road signs and lanes. This method uses a graph neural network model to predict the relationship between preset types of road signs and lanes. Since the graph neural network model is a pre-trained model, it eliminates the need for manual design of threshold parameters and algorithm processes when determining the relationship. Therefore, it can improve the efficiency and accuracy of determining the relationship between traffic lights and lanes, ensuring safe driving.

[0046] Exemplary System

[0047] Figure 1 This is a schematic diagram illustrating a scenario of a vehicle traveling at a traffic intersection, provided by an exemplary embodiment of this disclosure. Figure 1 As shown, since the traffic intersection 1 is a crossroads, vehicles can enter the intersection from four directions. Therefore, the driving scenarios of traffic intersection 1 can include the scenario of vehicles entering the intersection from south to north, the scenario of vehicles entering the intersection from east to west, the scenario of vehicles entering the intersection from north to south, and the scenario of vehicles entering the intersection from west to east.

[0048] This embodiment of the disclosure uses a scenario where a vehicle enters a traffic intersection from south to north as an example for illustration. Figure 1 As shown, the traffic signs include traffic lights 11, 13, and 15. In a scenario where vehicles enter the intersection from south to north, the traffic sign corresponding to the left-turn lane is traffic light 11, the traffic sign corresponding to the straight-ahead lane is traffic light 13, and the traffic sign corresponding to the right-turn lane is traffic light 15. For example... Figure 1 As shown, the traffic signs also include traffic lights 16 and 17, which are used to indicate whether pedestrians can cross the road perpendicular to lane 10.

[0049] For example, consider a vehicle traveling in lane 10 and about to enter intersection 1. Lane 10 is a left-turn lane, lane 12 is a straight-ahead lane, and lane 14 is a right-turn lane. Since lane 10 is a left-turn lane, vehicles traveling in lane 10 only need to pay attention to the signal emitted by traffic light 11, and do not need to pay attention to the signals emitted by traffic lights 13 and 15. Therefore, the relationship between lane 10 and traffic light 11 is associated, while the relationship between lane 10 and traffic lights 13 and 15 is not associated.

[0050] For example, consider a vehicle traveling in lane 10 and about to enter intersection 1. Lane 10 is a left-turn and straight-ahead lane, lane 12 is a straight-ahead lane, and lane 14 is a right-turn lane. Since lane 10 is a left-turn and straight-ahead lane, vehicles in lane 10 only need to pay attention to the signals from traffic lights 11 and 13, but not to traffic light 15. Therefore, the relationship between lane 10 and traffic lights 11 and 13 is associated, while the relationship between lane 10 and traffic light 15 is not associated.

[0051] For example, taking the association between lane 10 and traffic lights 11 and 13 as an example, when a vehicle is traveling in lane 10 and is about to enter the traffic intersection 1, it can obtain the signal issued by traffic light 11 or traffic light 13; and perform the corresponding response operation according to the signal issued by traffic light 11 or traffic light 13.

[0052] In some examples, the response actions performed by a vehicle based on the signal emitted by the traffic light associated with the lane in which the vehicle is located include, but are not limited to: turning left, turning right, going straight, and braking.

[0053] In some embodiments, if lane 10 is not associated with traffic light 15, it may mean that when a vehicle is traveling in lane 10 and about to enter traffic intersection 1, it does not need to obtain the signal issued by traffic light 15.

[0054] Exemplary methods

[0055] Figure 2 This is a schematic flowchart illustrating a method for predicting the association between road signs and lanes, provided in an exemplary embodiment of this disclosure. The method for predicting the association between road signs and lanes provided in this embodiment can be applied to... Figure 1 In the traffic intersection scenario shown, such as Figure 2 As shown, the method includes steps 201 to 203.

[0056] Step 201: Determine the driving environment data collected during vehicle operation.

[0057] For example, the driving environment data may include data on at least one static object and / or dynamic object in the vehicle driving environment. For instance, static objects in the vehicle driving environment may include at least one of static objects such as green belts, lane lines, preset type signs, road barriers, and ground arrows; dynamic objects in the vehicle driving environment may include at least one of dynamic objects such as other vehicles, pedestrians, and cyclists.

[0058] For example, driving environment data can be acquired using different types of sensors installed on the vehicle. These different types of sensors may include, but are not limited to, image sensors, attitude sensors, and point cloud sensors. In some examples, the image sensor may include a camera, the attitude sensor may include an inertial measurement unit (IMU), and the point cloud sensor may include an infrared sensor, an ultrasonic sensor, a millimeter-wave radar sensor, and a lidar sensor.

[0059] In some examples, when a vehicle is about to enter or exit a traffic intersection and the distance between the vehicle and the intersection is within a preset distance range (e.g., within 50 meters (m) before or after entering the intersection), driving environment data can be collected by various types of sensors on the vehicle.

[0060] Step 202: Based on driving environment data, determine the input graph structure data used to represent the initial relationship between road signs and lanes.

[0061] For example, road signs include preset type signs and environmental signs. Preset type signs include signs indicating the drivable state of the road, such as traffic signs including traffic lights. Environmental signs include signs indicating the road environment. Environmental signs may include static environmental elements and dynamic environmental elements. Static environmental elements may include, but are not limited to, sidewalks and stop lines. Dynamic environmental elements may include, but are not limited to, pedestrians.

[0062] In some embodiments, the input graph structure data refers to the graph structure data of the graph neural network model to be input. The input graph structure data includes multiple nodes and multiple edges, and the edge between any two nodes is used to represent the relationship between the two nodes.

[0063] For example, the input graph structure data is used to represent the initial relationship between road signs and lanes, and the nodes in the input graph structure data may include sign nodes for representing road signs and lane nodes for representing lanes.

[0064] In some examples, since road signs include preset type signs and environmental signs, the nodes in the input graph structure data may include preset type nodes for representing preset type signs, environmental nodes for representing environmental signs, and lane nodes for representing lanes. Edge information (such as the presence or absence of an edge) between any two nodes in the input graph structure data is used to indicate whether a relationship may exist between these two nodes.

[0065] For example, if there is an edge between a preset type node and a lane node in the input graph structure data, it indicates that there may be a relationship between the preset type marker corresponding to the preset type node and the lane corresponding to the lane node. If there is no edge between a preset type node and a lane node in the input graph structure data, it indicates that there may be no relationship between the preset type marker corresponding to the preset type node and the lane corresponding to the lane node.

[0066] Figure 3 This is a schematic diagram of input graph structure data provided in an exemplary embodiment of this disclosure. Combined with... Figure 1 ,like Figure 3 As shown, the input graph structure data includes lane node 301 representing lane 10, lane node 302 representing lane 12, lane node 303 representing lane 14, preset type node 304 representing traffic light 11, preset type node 305 representing traffic light 13, preset type node 306 representing traffic light 15, preset type node 307 representing traffic light 16, preset type node 308 representing traffic light 17, and environment node 309 representing environmental signs.

[0067] For example, since the initial relationship between lane 10 and traffic lights 11, 13, 15, 16, and 17 is uncertain, the initial relationship between lane 10 and these traffic lights can be set to an associated relationship. Therefore, as... Figure 3 As shown, there are edges between lane node 301 and preset type nodes 304 to 308.

[0068] like Figure 3As shown, since the relationships between lane nodes 301 and 303, between preset type nodes 304 and 306, and between preset type nodes 307 and 308 do not need to be predicted, the nodes that do not need to be predicted are connected with solid lines, while the nodes that need to be predicted are connected with dashed lines or dotted lines.

[0069] Step 203: Process the input graph structure data based on the graph neural network model to obtain the association between the preset type of road signs and the lanes.

[0070] The graph neural network model is used to process the input graph structure data to predict whether there is an edge between the pre-defined type node in the marker node and the lane node, and then to determine whether there is a correlation between the predicted pre-defined marker and the lane.

[0071] In some examples, the processing of input graph structure data by the graph neural network model may include aggregating information from the nodes in the input graph structure data and predicting the association between predefined type signs and lanes after aggregation.

[0072] In some implementations, the graph neural network model processes the input graph structure data to obtain output graph structure data. This output graph structure data represents the association between preset type signs and lanes. In some examples, the association between preset type signs and lanes can be determined based on the edge information between preset type nodes and lane nodes in the output graph structure data. The association between preset type signs and lanes includes both associated and unassociated relationships.

[0073] Figure 4 This is a schematic diagram of an output graph structure data provided by an exemplary embodiment of this disclosure. For example... Figure 4 As shown, taking lane node 401 as the node aggregated from lane node 301 and preset type node 404 as the node aggregated from preset type node 304 as an example. Because... Figure 4 In the output graph structure data shown, there is an edge 4014 between the lane node 401 representing lane 10 and the preset type node 404 representing traffic light 11. Therefore, it can be determined that lane 10 and traffic light 11 are associated. Since... Figure 4 In the output graph structure data shown, there are no edges between lane node 401 representing lane 10 and preset type nodes representing other traffic lights. Therefore, it can be determined that lane 10 is not associated with other traffic lights (e.g., traffic lights 13, 15 to 17).

[0074] In some examples, a graph neural network model can be trained in the cloud and downloaded to the vehicle. The vehicle can then predict the relationship between preset type signs and lanes in real time. In other examples, after training the graph neural network model in the cloud, the model can be stored there. The vehicle then uploads real-time driving environment data to the cloud, processes it using the cloud-based graph neural network model, and sends the predicted relationship back to the vehicle in real time. This disclosure does not specifically limit the method for determining the relationship between preset type signs and lanes.

[0075] In this embodiment, input graph structure data representing the initial relationship between road signs and lanes is determined based on driving environment data collected during vehicle operation. Then, a graph neural network model processes the input graph structure data to obtain the association relationship between preset type signs and lanes. Since the graph neural network model is pre-trained, the processing of the input graph structure data does not require manual design of threshold parameters and algorithm flows. Therefore, it improves the efficiency and accuracy of determining the association relationship between preset type signs and lanes, ensuring safe driving.

[0076] Because the input graph structure data includes lane nodes, preset type nodes, and environment nodes, which are of different types, the dimensions of the data corresponding to these nodes may be different, resulting in these nodes residing in different vector spaces. Since different vector spaces may require different computational processing methods, it is not possible to directly perform computational processing on lane nodes, preset type nodes, and environment nodes.

[0077] To ensure that the graph neural network model can correctly predict the relationship between traffic lights and lanes, this embodiment of the disclosure can aggregate the different types of nodes using the graph neural network model, so that the aggregated nodes are in the same vector space. Then, the nodes in the same vector space are processed to predict the relationship between preset type of signs and lanes.

[0078] In some embodiments, such as Figure 5 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 203 may include the following steps 2031 and 2032:

[0079] Step 2031: Aggregate multiple nodes in the input graph structure data based on at least one aggregation layer in the graph neural network model to obtain intermediate graph structure data.

[0080] For example, the aggregation layer is used to perform feature aggregation on preset type nodes, environment nodes and lane nodes in different vector spaces in the input graph structure data, so as to transform the preset type nodes, environment nodes and lane nodes into nodes in the same vector space.

[0081] For example, a graph neural network model may include one aggregation layer or multiple aggregation layers. If the graph neural network model includes multiple aggregation layers, these aggregation layers may be connected sequentially. For example, the output of the first aggregation layer is connected to the input of the second aggregation layer, the output of the second aggregation layer is connected to the input of the third aggregation layer, and so on, until the last aggregation layer. This disclosure does not limit the number of aggregation layers included in the graph neural network model; the number of aggregation layers is related to the prediction accuracy of the graph neural network. In some examples, the number of aggregation layers included in the graph neural network model may be the minimum number of layers required to ensure the prediction accuracy of the graph neural network.

[0082] In some implementations, the aggregation layer can aggregate multiple nodes by combining neighboring nodes and connecting edge information. Neighboring nodes can include nodes connected to the node to be updated via an edge (neighboring nodes with depth 1), or nodes connected to the node to be updated via multiple edges (neighboring nodes with depth greater than or equal to 1). Edges connecting nodes can include edges directly connected to the node to be updated (edges with depth 1), or edges indirectly connected to the node to be updated (edges with depth greater than or equal to 1).

[0083] In some implementations, the intermediate graph structure data may be aggregated data of the nodes in the input graph structure data, and the intermediate graph structure data includes the updated nodes.

[0084] Step 2032: Process the intermediate graph structure data based on the decoding layer in the graph neural network model to obtain the association between the preset type of sign and the lane.

[0085] For example, the decoding layer is used to perform calculations on the nodes in the intermediate graph structure data to obtain the edge information between preset type nodes and lane nodes.

[0086] In some implementations, the decoding layer can be a fully connected neural network layer. By processing preset type nodes and lane nodes in the same vector space through a fully connected neural network layer, the correlation between the preset type markers and lanes can be predicted.

[0087] In this embodiment of the disclosure, multiple nodes in the input graph structure data are aggregated through at least one aggregation layer in the graph neural network. This not only reduces the dimensionality of the data but also transforms the multiple nodes into the same vector space. This avoids the inability to predict the association between preset type markers and lanes due to multiple nodes not being in the same vector space, ensuring the accuracy of the graph neural network model's predictions.

[0088] like Figure 6 As shown above, in the above Figure 5 Based on the illustrated embodiment, step 2031 may include the following steps 601 to 603:

[0089] Step 601: Based on the first encoding layer in the aggregation layer, perform local information aggregation on multiple nodes to obtain the first subgraph structure data.

[0090] For example, the first encoding layer can be a single-layer graph neural network used for local information aggregation in a graph neural network, which can be called a graph neural network (GNN) block. Local information aggregation can be the aggregation of information about some nodes and some edges in the input graph structure data. For example, the information of neighboring nodes with a depth of 1 and edges with a depth of 1 can be aggregated.

[0091] In some implementations, the first coding layer can aggregate local information from multiple nodes through spatial domain models such as Graph Sage and Aggregate (Graph Sage), Learnable Graph Convolutional Networks (LGCN), and Mixture Model Networks (MoNet).

[0092] Step 602: Based on the second encoding layer in the aggregation layer, perform global information aggregation on multiple nodes after local information aggregation in the first subgraph structure data to obtain the second subgraph structure data.

[0093] For example, the second encoding layer is a single-layer graph neural network used for global information aggregation in the graph neural network, which can be called a Transformer block. Global information aggregation can be the aggregation of information about all nodes and all edges in the graph structure data.

[0094] The second encoding layer can perform global information aggregation on multiple nodes in the first subgraph structure data through a Transformer architecture with a multi-head attention mechanism.

[0095] Step 603: Determine the intermediate graph structure data based on the second subgraph structure data.

[0096] For example, when the graph neural network model includes an aggregation layer (i.e., the graph neural network model includes a first encoding layer and a second encoding layer), intermediate graph structure data can be obtained through the above steps 601 and 602.

[0097] In some examples, when the graph neural network model includes an aggregation layer, step 603 can be considered as determining the second subgraph structure data output by the aggregation layer as intermediate graph structure data.

[0098] For example, when the graph neural network model includes multiple aggregation layers (i.e., each aggregation layer includes a first encoding layer and a second encoding layer), the second subgraph structure data output by an aggregation layer can be obtained through the above steps 601 and 602, and the second subgraph structure data is input into the next aggregation layer, and so on, until the last aggregation layer outputs the intermediate graph structure data.

[0099] In some examples, when the graph neural network model includes multiple aggregation layers, step 603 can be considered as processing the subgraph structure data output by the previous aggregation layer sequentially through other aggregation layers in the graph neural network model (such as aggregation layers other than the first aggregation layer in the graph neural network model) until the last aggregation layer outputs subgraph structure data, and then determining the subgraph structure data output by the last aggregation layer as the intermediate graph structure data. The way any aggregation layer in the graph neural network model processes the input data is similar to steps 601 and 602 above.

[0100] In this embodiment of the disclosure, when aggregating multiple nodes in the input graph neural network through the aggregation layer of the graph neural network, a two-stage aggregation method of first aggregating local information and then aggregating global information is adopted, which is beneficial to improving the accuracy of the graph neural network model in predicting the correlation relationship.

[0101] like Figure 7 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 202 may include the following steps 2021 and 2022.

[0102] Step 2021: Based on driving environment data, determine the first graph structure data for representing the initial relationship between lanes, the second graph structure data for representing the initial relationship between each preset type of sign, and the environmental nodes for representing environmental signs in the road signs.

[0103] For example, the initial relationships between lanes are used to represent the inherent relationships between lanes in the vehicle's driving environment. For instance, in the first graph structure data, there is an edge between the two lane nodes corresponding to two adjacent lanes, and no edge between two non-adjacent lane nodes. The initial relationships between preset type signs are used to represent whether each preset type sign in the vehicle's driving environment belongs to the same group (e.g., whether it is set on the same traffic light bracket). For instance, in the second graph structure data, there is an edge between preset type nodes corresponding to preset type signs belonging to the same group, and no edge between preset type nodes corresponding to preset type signs not belonging to the same group.

[0104] Figure 8 This is a schematic diagram of a first graph structure data provided in an exemplary embodiment of this disclosure. Combined with... Figure 1 ,like Figure 8 As shown, the first graph structure data includes lane node 801 representing lane 10, lane node 802 representing lane 12, and lane node 803 representing lane 14. Since lane 12 is located between lane 10 and lane 14, and is adjacent to both lane 10 and lane 14, there is an edge 8012 between lane node 802 and lane node 801, and an edge 8023 between lane node 802 and lane node 803.

[0105] Figure 9 This is a schematic diagram of a second graph structure data provided in an exemplary embodiment of this disclosure. Continuing in conjunction with... Figure 1 ,like Figure 9 As shown, since traffic lights 11, 13, and 15 are located on the same traffic light bracket, they belong to the same group. Therefore, there is an edge 9012 between the preset type node 901 representing traffic light 11 and the preset type node 902 representing traffic light 13, and an edge 9023 between the preset type node 902 representing traffic light 13 and the preset type node 903 representing traffic light 15. For example, traffic lights 16 and 17 are located on another traffic light bracket; therefore, there is an edge 9045 between the preset type node 904 representing traffic light 16 and the node 905 representing traffic light 17.

[0106] Step 2022: Determine the input graph structure data based on the first graph structure data, the second graph structure data, and the environment nodes.

[0107] For example, initial graph structure data can be determined based on each lane node and the edges between lane nodes in the first graph structure data, each preset type node and the edges between preset type nodes in the second graph structure data, and environment nodes. Then, the possible relationships between different types of nodes in the initial graph structure data are predicted, and initial edges are set or not set between different types of nodes according to the prediction results to obtain input graph structure data.

[0108] In some examples, the potential relationships between different types of nodes in the same graph structure data can be predicted based on the vehicle's driving environment. For instance, based on the potential relationships between lanes and preset type signs in the vehicle's driving environment, the potential relationships between any lane node and any preset type node can be predicted. As another example, based on the potential relationships between environmental signs and lanes and / or preset type signs in the vehicle's driving environment, the potential relationships between environmental nodes and any lane node and / or any preset type node can be predicted.

[0109] In some examples, if the prediction results include possible relationships between nodes of different types, initial edges are set between nodes of different types. For example, if the prediction results include possible relationships between any lane node and any preset type node, an initial edge is set between that lane node and that preset type node. As another example, if the prediction results include possible relationships between an environment node and any lane node and / or any preset type node, an initial edge is set between the environment node and any lane node and / or any preset type node.

[0110] In some examples, if the prediction results indicate that there is no possible association between different types of nodes, then no initial edge is set between nodes of that type.

[0111] For example, combined Figure 3 ,like Figure 8 and Figure 9 As shown, Figure 8 Lane nodes 801, 802, and 803, edge 8012, and edge 8023 are included. Figure 9The preset type nodes 901, 902, 903, 904, 905, edges 9012, 9023, and 9045, and environment node 309 are placed in the same graph structure data. Based on the possible relationships between environment node 309 and any other type node in the same graph structure data (such as lane nodes 801 to 803, preset type nodes 901 to 905); and the possible relationships between lane nodes 801 to 803 and any other preset type node in the same graph structure data (such as preset type nodes to 905), initial edges can be set between environment node 309 and any other type node in the same graph structure data, and between lane nodes 801, 802, and 803 and any other preset type node in the same graph structure data, to obtain the following... Figure 3 The input graph structure data is shown.

[0112] In this embodiment, since the first graph structure data, the second graph structure data, and the environmental nodes are all determined based on the vehicle's driving environment data, the first graph structure data, the second graph structure data, and the environmental nodes can accurately reflect the vehicle's current driving environment. Consequently, the input graph structure data determined based on the first graph structure data, the second graph structure data, and the environmental nodes can also accurately reflect the current vehicle's driving environment. This is beneficial for accurately predicting the relationship between preset type markers and lanes based on the input graph structure data, thereby improving driving safety.

[0113] like Figure 10 As shown above, in the above Figure 7 Based on the illustrated embodiment, step 2021 may include the following steps 1001 and 1002:

[0114] Step 1001: Based on the driving environment data, determine the static scene data used to represent the driving environment of the vehicle.

[0115] To facilitate the description of the specific implementation method for determining static scene data, in this embodiment of the disclosure, the vehicle is referred to as the "self-vehicle," and the lane in which the self-vehicle travels is referred to as the "self-lane."

[0116] For example, static scene data includes data obtained by encoding, reconstructing, and logically reasoning about object data of multiple static objects collected by the vehicle at multiple times. In some examples, the multiple static objects may include multiple static objects of different types. For example, the multiple static objects may include predefined type signs, ground arrows, and lane lines.

[0117] For example, the driving environment data may include object data of static objects included in the vehicle driving environment at each of multiple time points, and static scene data may be determined based on the object data of static objects included in the vehicle driving environment at each time point.

[0118] In some examples, determining static scene data based on data of static objects included in the vehicle's driving environment at each time point may include the following steps S1 to S5:

[0119] Step S1: Obtain object data of static objects included in the vehicle's driving environment at each time point.

[0120] For example, object data of static objects at each moment can be obtained from image frames at each moment in the driving environment data. In some examples, static objects included in the driving environment of the vehicle at each moment can be identified from the image frames at each moment to obtain object data of the static objects.

[0121] In some implementations, static objects at different times can be identified from each image frame using a static object recognition model. In some examples, if the static objects include preset type markers, environmental static elements, lane lines, and ground arrows, then the static object recognition model includes, but is not limited to, a preset type marker recognition model, an environmental static element recognition model, a lane line recognition model, and a ground arrow recognition model.

[0122] Step S2: Encode the static object data at each time step by encoding the object data of the static object at each time step.

[0123] For example, the encoding process in S2 may include converting the object data of each static object into encoded data of each static object, represented by multiple explicit vectors. In some examples, an explicit vector is a vector that has physical meaning and whose numbers represent the magnitude of the physical quantity. Multiple explicit vectors representing the same type of encoded data can form an explicit vector group; for example, the encoded data of preset type markers at multiple times can form an explicit vector group.

[0124] In some embodiments, the object data of static objects at each time step can be encoded using an encoding algorithm to obtain encoded data of static objects at each time step, represented by display vector groups. This encoding algorithm includes, but is not limited to, ordinary neural network models and classical rule-based algorithms.

[0125] Step S3: Reconstruct the encoded data of static objects at each time point.

[0126] The encoded data of static objects at each time point are aggregated and reconstructed using an aggregation and reconstruction algorithm to obtain reconstructed data of static objects corresponding to the vehicle's driving environment. Aggregation and reconstruction algorithms include, but are not limited to, ordinary neural networks and other classic rule-based algorithms.

[0127] Step S4: Perform logical reasoning on the reconstructed data of the static objects obtained after reconstruction to obtain the logical layer data and the logical relationship between the reconstructed data of each static object and the logical layer data.

[0128] For example, the reconstruction data of static objects may include lane line reconstruction data, preset type sign reconstruction data, environmental static element reconstruction data, and ground arrow reconstruction data. Step S4 may include performing logical reasoning on the lane line reconstruction data using a logical reasoning algorithm to obtain lane data, and performing logical reasoning on the lane data and preset type sign reconstruction data using a logical reasoning algorithm to obtain intersection area information.

[0129] In some examples, for ease of description, the reconstructed data of static objects obtained through logical reasoning can be referred to as static object data. For example, the reconstructed data of preset type markers, the reconstructed data of environmental static elements, and the reconstructed data of ground arrows can be referred to as preset type marker data, environmental static element data, and ground arrow data, respectively.

[0130] Step S5: Static scene data is constructed based on the logical layer data and the logical relationship between the reconstructed data of each static object and the logical layer data.

[0131] For example, static scene data may include, but is not limited to, ground arrow data, lane data, adjacency relationships between lanes, preset type sign data, preset type sign group, and environmental static element data.

[0132] Step 1002: Based on the static scene data, determine the first graph structure data, the second graph structure data, and the environment nodes.

[0133] In some implementations, step 1002 may include: determining first map structure data, preset type marker data and preset type marker groups based on ground arrow data, lane data and connection relationships between lanes in static scene data, determining second map structure data and determining environmental nodes based on environmental static element data in static scene data.

[0134] In this embodiment, the lane nodes in the first graph structure data, the preset type nodes in the second graph structure data, and the environment nodes are all data represented by implicit vectors. Implicit vectors are vectors that are the opposite of explicit vectors; they have no physical meaning, and the magnitude of the physical quantity represented by the number in the vector cannot be determined from the number in the vector.

[0135] In some examples, the process of transforming data represented by explicit vectors into data represented by implicit vectors is called embedding. By performing deep feature extraction on the implicit vectors obtained after embedding explicit vectors, implicit vectors used to represent nodes can be obtained. Exemplarily, both embedding and deep feature extraction can be implemented through a feature extraction module. For example, lane nodes can be obtained by embedding and extracting features from lane data using a feature extraction module.

[0136] For example, environmental static element data in static scene data can be embedded and feature extracted using a feature extraction module to obtain environmental nodes. In some examples, the feature extraction module may include and is limited to multilayer perceptrons, stochastic long short-term memory networks (LSTM), and gated recurrent units (GRU).

[0137] In this embodiment of the disclosure, since the environmental nodes are determined based on the static environmental element data in the static scene data, and the static environmental element data is determined based on the collected driving environment data, the environmental nodes can accurately reflect the environmental markers included in the actual driving environment of the vehicle.

[0138] In some embodiments, determining the implementation method of the first graph structure data may include: determining the second arrow data based on the first arrow data in the static scene data; and determining the first graph structure data based on the lane data in the static scene data, the adjacency relationship between lanes in the static scene data, and the second arrow data.

[0139] For example, the first arrow data may include ground arrow data corresponding to ground arrows included in the vehicle's driving environment (the ground arrow data can be represented by a corresponding explicit vector). The second arrow data may be arrow data obtained by cleaning the first arrow data and filtering out arrow data that is not related to preset type markers.

[0140] In some implementations, multiple lane nodes for representing lanes can be determined based on lane data and second arrow data; and first graph structure data can be determined based on adjacency relationships and each lane node.

[0141] Since the driving environment of a vehicle includes multiple lanes, the static scene data corresponds to multiple lane data (corresponding to multiple explicit vectors). Therefore, the first graph structure data includes multiple corresponding lane nodes. For example, refer to... Figure 1 and Figure 8 As shown, in the scenario where a car is about to enter traffic intersection 1 from south to north, there are three lanes: lane 10, lane 12, and lane 14. The corresponding lane nodes in the structure data of the first figure are lane node 801, lane node 802, and lane node 803.

[0142] Furthermore, factors affecting the association between each lane and preset type of signage include the arrows on each lane. Therefore, the corresponding arrow information needs to be embedded in the lane node corresponding to each lane.

[0143] For example, multiple lane data can be embedded and feature extracted first to obtain multiple corresponding lane nodes. Then, multiple explicit vector representations of second arrow data can be embedded and feature extracted to obtain multiple arrow information represented by multiple implicit vectors. Then, each arrow information represented by each implicit vector is embedded to represent the lane node of its corresponding lane.

[0144] In some examples, if the lane includes a ground arrow, the corresponding arrow information is embedded in the lane node representing that lane. If the lane does not include an arrow, no arrow information is embedded in the lane node representing that lane. For example, see reference... Figure 1 and Figure 8 As shown, in a scenario where a vehicle is about to enter intersection 1 from south to north, if lane 10 includes a left-turn ground arrow, lane 12 includes a straight-ahead ground arrow, and lane 14 includes a right-turn ground arrow, then the left-turn arrow information corresponding to the left-turn ground arrow needs to be embedded in lane node 801, the straight-ahead arrow information corresponding to the straight-ahead ground arrow needs to be embedded in lane node 802, and the right-turn arrow information corresponding to the right-turn ground arrow needs to be embedded in lane node 803. For example, continue to refer to... Figure 1 and Figure 8 As shown, in the scenario where a vehicle is about to enter traffic intersection 1 from south to north, if only lane 12 includes a straight ground arrow, and lanes 10 and 14 do not include ground turning arrows, then only the straight arrow information corresponding to the straight ground arrow needs to be embedded in lane node 802, while no arrow information needs to be embedded in lane nodes 801 and 803.

[0145] For example, similar to the implementation of embedding and feature extraction of static environmental element data, the second arrow data can also be embedded and feature extracted through a feature extraction module to obtain arrow information represented by multiple implicit vectors.

[0146] In some implementations, after determining multiple lane nodes, an edge may or may not be set between the two lane nodes corresponding to the two lanes, depending on the adjacency relationship between the two lanes. For example, if two lanes are adjacent, an edge is set between the two lane nodes corresponding to the two lanes. If two lanes are not adjacent, no edge is set between the two lane nodes corresponding to the two lanes. Specifically, refer to the above. Figure 1 , Figure 3 and Figure 8 As shown, it will not be elaborated further here.

[0147] In this embodiment of the disclosure, since the lane nodes are determined based on lane data and second arrow data, and lane information and arrow information are superimposed in the lane nodes, the lane information represented by the lane nodes can accurately reflect the lanes in the driving environment. Consequently, the first graph structure data determined based on the lane nodes and adjacency relationships can also accurately reflect each lane in the vehicle driving environment and the adjacency relationships between each lane.

[0148] In this embodiment of the disclosure, by cleaning the first arrow data to obtain the second arrow data, irrelevant arrow data can be filtered out, which can reduce the workload of subsequent embedding and feature extraction, and at the same time reduce the dimensionality of the lane nodes, which is beneficial to simplifying the first graph structure data.

[0149] In some implementations, determining the implementation method of the second graph structure data may include: determining multiple preset type nodes for representing preset type markers based on preset type marker data in static scene data; and determining the second graph structure data based on the groups of each preset type node and each preset type marker in the static scene data.

[0150] For example, the driving environment of a vehicle includes multiple preset type markers, and the static scene data correspondingly includes multiple preset type marker data. Therefore, the second graph structure data includes multiple corresponding preset type nodes. (See reference...) Figure 1 and Figure 9 As shown, in the scenario where a car is about to enter traffic intersection 1 from south to north, there are 5 traffic lights: traffic light 11, traffic light 13, traffic light 15, traffic light 16 and traffic light 17. The structure data in the second figure corresponds to 5 lane nodes, from preset type node 901 to preset type node 905.

[0151] In some examples, the determination of multiple preset type nodes is achieved by embedding and extracting features from multiple preset type marker data represented by explicit vectors using a feature extraction module, resulting in multiple preset type nodes represented by implicit vectors. The feature extraction module can be the same as described above, and will not be repeated here.

[0152] In some implementations, after determining multiple preset type nodes, edges can be set between preset type nodes corresponding to preset type markers in the same group, and between preset type nodes corresponding to preset type markers in different groups, based on the group of each preset type marker. Specifically, refer to the above. Figure 1 , Figure 3 and Figure 9 As shown, it will not be elaborated further here.

[0153] In this embodiment, since multiple preset type nodes are determined based on preset type marker data, and the preset type marker data is determined based on driving environment data, the preset type nodes can accurately represent the preset type markers in the driving environment. Furthermore, the second graph structure data determined based on the groups of each preset type node and each preset type marker in the static scene data can also accurately represent each preset type marker in the driving environment and the positional relationships between them.

[0154] In this embodiment of the disclosure, static scene data is determined through driving environment data. Therefore, the static scene data can accurately reflect each static object in the driving environment and the positional relationships between them. Furthermore, the first graph structure data, the second graph structure data, and the environment nodes determined based on the static scene data can also accurately reflect each static object in the driving environment and the positional relationships between them.

[0155] like Figure 11 As shown above, in the above Figure 7 Based on the illustrated embodiment, step 2022 may include the following steps 1101 to 1103:

[0156] Step 1101: Determine the first initial relationship between the environment node and each lane node in the first graph structure data, and the second initial relationship between the environment node and each preset type node in the second graph structure data.

[0157] For example, the first initial relationship can correspond to the predicted association between each lane and environmental signs in the driving environment. The second initial relationship can correspond to the predicted association between each preset type of sign and each environmental sign in the driving environment.

[0158] In one instance, to shorten the path length of information transmission when performing feature aggregation on nodes later, the environment node can be used as a relay virtual node, and corresponding edges can be set between it and all other nodes except the environment node.

[0159] refer to Figure 3As shown, corresponding edges are set between node 309 and other nodes in the input graph structure data (such as nodes 301, 302, 303, 304, 305, 306, 307 and 308).

[0160] Step 1102: Determine the third initial relationship between each lane node and each preset type node.

[0161] For example, the third initial relationship can correspond to the predicted association between each lane and each preset type of sign. Figure 3 The description is the same. Since the relationship between each lane and each preset type of sign is uncertain, a corresponding edge is set between each lane node and each preset type node.

[0162] Continue to refer to Figure 3 As shown, nodes 301, 302, and 303 are each connected to a corresponding edge between them and each of the preset type nodes (such as nodes 304, 305, 306, 307, and 308).

[0163] Step 1103: Determine the input graph structure data based on the first initial relation, the second initial relation, and the third initial relation.

[0164] For example, step 1103 may include setting predicted edges between nodes based on the first graph structure data, the second graph structure data, and the environment nodes to form each node and each inherent edge in the input graph structure data, and based on the first initial relationship, the second initial relationship, and the third initial relationship, so as to obtain the input graph structure data.

[0165] In some examples, the nodes in the input graph structure data include lane nodes in the first graph structure data, preset type nodes in the second graph structure data, and environment nodes. The inherent edges in the input graph data include the inherent edges between lane nodes in the first graph structure data and the inherent edges between preset type nodes in the second graph structure data. The inherent edges between lane nodes include the edges between the two lane nodes corresponding to adjacent lanes. For example, refer to... Figure 8 As shown, the inherent edges between lane nodes include edge 8012 and edge 8023. The inherent edges between preset type nodes include the edges between multiple preset type nodes corresponding to multiple preset type markers in the same group. For example, refer to... Figure 9 As shown, the inherent edges between nodes of each preset type include edge 9012, edge 9023 and edge 9045.

[0166] In some examples, based on each node and inherent edge in the input graph structure data, and based on the first and second initial relations, corresponding predicted edges can be set between the environment nodes and each lane node, as well as between each preset type node, to obtain, based on the third initial relation, corresponding predicted edges between each node and each preset type node, as shown below. Figure 3 The input graph structure data is shown.

[0167] In this embodiment, the first initial relationship between the environmental nodes and the lane nodes in the first graph structure data can accurately represent the initial association between environmental signs and lanes in the vehicle's driving environment; the second initial relationship between the environmental nodes and the preset type nodes in the second graph structure data can accurately represent the initial association between environmental signs and preset type signs in the vehicle's driving environment; and the third initial relationship between each lane node and each preset type node can accurately represent the initial association between each lane and each preset type sign in the vehicle's driving environment. Therefore, the input graph structure data determined based on the first, second, and third initial relationships can accurately represent the initial relationship between road signs and lanes, which is beneficial for accurately determining the association between preset type signs and lanes and improving driving safety.

[0168] To improve the prediction accuracy of the graph neural network model, the graph neural network model can be pre-trained to obtain the graph neural network model used in the aforementioned embodiments. This disclosure also provides a graph neural network model training method, such as... Figure 12 As shown, the method includes the following steps 1201 to 1203:

[0169] Step 1201: Determine the sample association relationship between the sample preset type signs and sample lanes in the sample road signs corresponding to the multiple sets of sample input map structure data and sample input map structure data.

[0170] Among them, the sample input graph structure data is used to represent the initial relationship between sample road signs and sample lanes.

[0171] For example, multiple sets of sample input graph structure data can be multiple sets of input graph structure data determined based on driving environment data collected by multiple vehicles within a preset time period (e.g., 10 days). In some examples, multiple sets of sample input graph structure data can be determined by selecting from the driving environment data collected by multiple vehicles within the preset time period and based on the driving environment data collected by multiple vehicles within a preset distance range before entering each traffic intersection (e.g., within 50 meters (m) before and after entering the traffic intersection).

[0172] In some implementations, such as Figure 13 As shown, the method for determining multiple sets of sample input graph structure data may include steps 1301 to 1303.

[0173] Step 1301: Determine multiple sets of sample driving environment data collected from the sample vehicles.

[0174] For example, the sample vehicles may include at least one vehicle. In some examples, the sample vehicles may be all vehicles in a certain area.

[0175] For example, a portion of the driving environment data collected by at least one vehicle within a preset time period can be determined as multiple sets of sample driving data. In some examples, the driving environment data collected by at least one vehicle within a preset distance range when entering each traffic intersection can be determined as multiple sets of sample driving environment data.

[0176] In some implementations, the driving environment data collected by a vehicle when it enters a traffic intersection from one direction can be defined as a set of sample driving environment data.

[0177] Step 1302: Based on the driving environment data of each group of samples, determine the first sample graph structure data for representing the initial relationship between each sample lane, the second sample graph structure data for representing the initial relationship between each sample preset type marker, and the sample environment node for representing the sample environment marker.

[0178] For example, the method for determining the first sample graph structure data, the second sample graph structure data, and the sample environment nodes is similar to the method for determining the first graph structure data, the second graph structure data, and the group environment nodes, and will not be described again here.

[0179] Step 1303: Based on multiple sets of first sample graph structure data, second sample graph structure data, and sample environment nodes, determine multiple sets of sample input graph structure data.

[0180] For example, a set of sample input graph structure data can be determined based on a set of first sample graph structure data, second sample graph structure data, and sample environment nodes. The method for determining a set of sample input graph structure data is similar to that for determining input graph structure data, and will not be described again here.

[0181] In this embodiment, first sample graph structure data, second sample graph structure data, and sample environment nodes are determined using each set of sample driving environment data. Therefore, each set of first sample graph structure data, second sample graph structure data, and sample environment nodes accurately reflects the sample driving environment of each sample vehicle. Furthermore, the multiple sets of sample input graph structure data determined based on these multiple sets of first sample graph structure data, second sample graph structure data, and sample environment nodes also accurately reflect the sample driving environment of each sample vehicle. This is beneficial for subsequent training of the graph neural network model based on the multiple sets of sample input graph structure data, thereby improving the performance of the graph neural network model.

[0182] In some embodiments, reference Figure 14 As shown, step 1302 may include steps 1401 and 1402.

[0183] Step 1401: Based on the driving environment data of each group of samples, determine the sample static scene data used to represent the driving environment of the sample vehicles.

[0184] For example, sample static scene data is similar to static scene data, and can be data obtained by encoding, reconstructing and logically reasoning the object data of multiple sample static objects collected from a sample vehicle at multiple times.

[0185] For example, the method for determining the static scene data of the sample is similar to the method for determining the static scene data, and can be found in steps S1 to S5, which will not be repeated here.

[0186] Step 1402: Based on the static scene data of the samples, determine the first sample graph structure data, the second sample graph structure data, and the sample environment nodes.

[0187] For example, the methods for determining the first sample graph structure data, the second sample graph structure data, and the sample environment nodes are similar to those for determining the first graph structure data, the second graph structure data, and the environment nodes, respectively. See step 1002 for details, which will not be repeated here.

[0188] In this embodiment of the disclosure, sample static scene data is determined through each group of sample driving environment data. Therefore, this sample static scene data can accurately reflect each sample static object in the sample driving environment and the positional relationship between each sample static object. Furthermore, the first sample graph structure data, the second sample graph structure data, and the sample environment nodes determined based on this sample static scene data can also accurately reflect each sample static object in the sample driving environment and the positional relationship between each sample static object.

[0189] In some embodiments, reference Figure 15 As shown, step 1302 may include steps 1501 and 1502.

[0190] Step 1501: Based on the driving environment data of each group of samples, determine the static scene data and dynamic scene data of the samples used to represent the driving environment of the sample vehicles.

[0191] For example, sample dynamic scene data can be data obtained by encoding, reconstructing, and logically reasoning about object data of multiple sample dynamic objects collected from a sample vehicle at multiple times. The multiple sample dynamic objects can include different types of sample dynamic objects. For example, the multiple sample dynamic objects can include sample vehicles and sample pedestrians.

[0192] In some examples, data processing can be performed on the driving environment data of each group of samples to obtain static scene data and dynamic scene data of the samples. This data processing may include, but is not limited to, encoding, reconstruction, logical reasoning, and scene segmentation.

[0193] For example, step 1501 may include steps S11 to S15.

[0194] Step S11: Determine the sample vehicle corresponding to each group of sample driving environment data, and obtain the object data of the sample static objects and the sample dynamic objects (including the sample vehicle itself) included in the driving environment of the sample vehicle at multiple times.

[0195] For example, object data of static objects included in the driving environment of the sample vehicle at multiple moments can be obtained by acquiring image frames at multiple moments during the driving process of the sample vehicle. Object data of dynamic objects included in the driving environment of the sample vehicle at multiple moments can be obtained by acquiring point cloud data and / or vehicle attitude data at multiple moments during the driving process of the vehicle.

[0196] In some examples, the method of obtaining the object data of sample static objects from image frames is similar to the method of obtaining the object data of static objects, and will not be repeated here.

[0197] In some examples, a dynamic object recognition model can be used to identify sample dynamic objects at each time step from point cloud data and / or vehicle attitude data, thus obtaining object data for the sample dynamic objects at each time step. In some examples, if the sample dynamic objects include sample vehicles and sample pedestrians, the dynamic object trajectory model can include a sample vehicle trajectory recognition model and a sample pedestrian trajectory recognition model.

[0198] Step S12: Encode the object data of the static objects and the object data of the dynamic objects included in the driving environment of the sample vehicle at multiple times to obtain the encoded data of the static objects and the encoded data of the dynamic objects included in the driving environment of the sample vehicle at multiple times.

[0199] For example, step S12 may include converting the object data of the sample static objects and the object data of the sample dynamic objects included in the driving environment of the sample vehicle at each time moment into the encoded data of the sample static objects and the encoded data of the sample dynamic objects represented by explicit vector groups, respectively.

[0200] In some examples, the object data of a sample vehicle may include four physical quantities with different physical meanings: attitude, position, acceleration, and velocity. The relative positions of these four physical quantities within the object data of the sample vehicle can be arbitrary. For example, an explicit vector I1 representing the object data of the sample vehicle at a single moment may include, but is not limited to, [attitude, position, acceleration, velocity], [attitude, acceleration, position, velocity], and [attitude, position, velocity, acceleration]. The attitude, position, acceleration, and velocity in this explicit vector I1 are used to represent the attitude value of the sample vehicle, the position value of the sample vehicle, the acceleration value representing the magnitude of the acceleration of the sample vehicle, and the velocity value representing the magnitude of the velocity of the sample vehicle, respectively.

[0201] In some examples, where the attitude, position, acceleration, and velocity values ​​are all represented by three-dimensional numerical values ​​in the vehicle coordinate system of the sample, the explicit vector I1 is a 12-dimensional vector. For example, the explicit vector I1 can be [a1 a2 a3a4a5 a6 a7 a8 a9 a10 a11 a12].

[0202] Let's take the explicit vector I1, representing the relative position as [attitude, position, acceleration, velocity], as an example. In the explicit vector I1, a1, a2, and a3 represent the attitude values ​​of the sample vehicle in the X, Y, and Z directions in the sample vehicle coordinate system; a4, a5, and a6 represent the position values ​​of the sample vehicle in the X, Y, and Z directions in the sample vehicle coordinate system; a7, a8, and a9 represent the acceleration values ​​of the sample vehicle in the X, Y, and Z directions in the sample vehicle coordinate system; and a10, a11, and a12 represent the velocity values ​​of the sample vehicle in the X, Y, and Z directions in the sample vehicle coordinate system. In some examples, the origin of the sample vehicle coordinate system can be the midpoint between the two rear tires of the sample vehicle at a selected moment from multiple moments.

[0203] For example, the object data of sample dynamic objects at each time step can be encoded using an encoding algorithm to obtain encoded data of the sample dynamic objects represented by display vector groups. In some examples, the encoding algorithm includes, but is not limited to, ordinary neural network models and classical rule-based algorithms.

[0204] Step S13: Reconstruct the encoded data of sample static objects included in the driving environment of the vehicle at multiple time points to obtain the reconstructed data of sample static objects included in the driving environment of the vehicle.

[0205] For example, step S13 is implemented in a similar way to step S3, and will not be described again here.

[0206] Step S14: Perform logical reasoning on the reconstructed data of the sample static objects included in the driving environment of the sample vehicle to obtain the sample logic layer data and the logical relationship between the reconstructed data of the sample static objects and the sample logic layer data.

[0207] For example, step S14 is implemented in a similar way to step S4, and will not be described again here.

[0208] Step S15: Based on the sample logic layer data and the logical relationship between the reconstructed data of the sample static objects and the sample logic layer data, scene segmentation is performed on the reconstructed data of the sample static objects and the encoded data of the sample dynamic objects included in the driving environment of the sample vehicles at multiple time points to obtain sample static scene data and sample dynamic scene data.

[0209] Step 1502: Based on the static scene data and dynamic scene data of the samples, determine the first sample graph structure data, the second sample graph structure data, and the sample environment nodes.

[0210] In some examples, for ease of description, the reconstructed data of static objects obtained after scene segmentation can be referred to as static object data. For instance, the reconstructed data of sample preset type markers, the reconstructed data of sample environmental static elements, and the reconstructed data of sample ground arrows can be referred to as sample preset type marker data, sample environmental static element data, and sample ground arrow data, respectively. The encoded data of dynamic objects obtained after scene segmentation is referred to as the trajectory data of dynamic objects.

[0211] In some implementations, step 1502 may include: determining first sample graph structure data, determining second sample graph structure data, and determining multiple sets of environmental nodes based on the trajectory data of sample vehicles at multiple times in the dynamic scene data of each group of samples, the sample ground arrow data, sample lane data, and the connection relationships between sample lanes in the static scene data of each group of samples, the sample ground arrow data, sample lane data, and the connection relationships between sample lanes.

[0212] In this embodiment of the disclosure, sample static scene data and sample dynamic scene data are determined through each set of sample driving environment data. Therefore, the sample static scene data and dynamic scene data can accurately reflect the sample static objects, the positional relationships of the sample static objects, and the trajectories of each sample dynamic object in the sample driving environment. Furthermore, the first sample graph structure data, the second sample graph structure data, and the sample environment nodes determined based on the sample static scene data and dynamic scene data can also accurately reflect the sample static objects, the positional relationships of each sample static object, and the trajectories of each sample dynamic object in the sample driving environment.

[0213] In some embodiments, reference Figure 16 As shown, step 1502 includes steps 1601 and 1602.

[0214] Step 1601: Based on the first sample arrow data in the static scene data and the first sample trajectory data in the dynamic scene data, determine the second sample arrow data and the second sample trajectory data.

[0215] For example, the first sample trajectory data may include trajectory data of multiple sample vehicles collected by at least one vehicle at various times, and other trajectory data that may be obviously unrelated to signals emitted by a particular type of marker.

[0216] The second sample trajectory data can be obtained by cleaning the first sample trajectory data and filtering out trajectory data that is obviously irrelevant to the signals emitted by preset type markers. In some examples, cleaning the first sample trajectory data may not only filter out illegal driving trajectory data and erroneously associated data, but also need to filter out vehicle trajectory data in traffic jam scenarios based on dynamic obstacles.

[0217] For example, the second sample trajectory data may include trajectory data of multiple sample vehicles collected by at least one vehicle at various times.

[0218] For example, since the method for determining the second sample arrow data is similar to the method for determining the second arrow data, it will not be described again here.

[0219] Step 1602: Based on the sample lane data in the sample static scene data, the adjacency relationship between sample lanes in the sample static scene data, the second sample arrow data, and the second sample trajectory data, determine the first sample graph structure data.

[0220] In some embodiments, multiple sample lane nodes for representing multiple sample lanes can be determined based on sample lane data, second sample trajectory data, and second sample arrow data; multiple sets of first sample graph structure data can be determined based on the adjacency relationship between sample lanes and multiple sample lane nodes.

[0221] For example, in order to improve the performance of neural network models, sample arrow information and trajectory information can be embedded in the sample lane nodes.

[0222] For example, the sample lane data can first be embedded and feature extracted to obtain the corresponding sample lane nodes. Then, the second sample arrow data represented by explicit vectors and the second driving trajectory data represented by explicit vector groups are embedded and feature extracted respectively to obtain the sample arrow information and trajectory information represented by implicit vectors. Then, the sample arrow information and trajectory information represented by implicit vectors are embedded into the sample lane nodes representing the corresponding sample lanes.

[0223] In some implementations, the method for determining the sample arrow information is the same as the method for determining the arrow information, and will not be described again here.

[0224] In some implementations, embedding and feature extraction of trajectory data for a single vehicle represented by an explicit vector group to obtain a single trajectory information represented by an implicit vector can be achieved as follows: If the vehicle's trajectory data is a 12-dimensional explicit vector group consisting of t explicit vectors of dimension 12 with a temporal order (one 12-dimensional explicit vector corresponds to the vehicle's trajectory data at one moment), then before embedding and feature extraction of this 12-dimensional explicit vector group, the temporal order information of the t explicit vectors can be incorporated into the 12-dimensional explicit vector group, resulting in an explicit vector group with increased dimension. For example, a 14-dimensional explicit vector group can be obtained. Then, this 14-dimensional explicit vector group is embedded and subjected to deep feature extraction using a feature extraction module with a multi-head attention mechanism to obtain an implicit vector representing the trajectory of a single vehicle.

[0225] For example, the feature extraction module may include at least two layers of feature extraction networks. For instance, the feature extraction module may include feature extraction networks of layers 1 to k. The first layer is used to embed a 14-dimensional explicit vector group, transforming it into an implicit vector group. The second to k-th layers are used to perform deep feature extraction on the implicit vector group generated by the first layer, obtaining an implicit vector representing the vehicle's trajectory. Multi-head attention may be used in this deep feature extraction to learn attention weights to combat noise and improve generalization. K is an integer greater than or equal to 2.

[0226] In some examples, the feature extraction module may include and is limited to multilayer perceptrons, stochastic long short-term memory networks (LSTM), and gated recurrent units (GRU).

[0227] For example, after embedding and extracting the trajectory data of each sample vehicle, the trajectory information of vehicles in the same lane can be aggregated to obtain the trajectory information to be embedded for each lane.

[0228] In some examples, feature aggregation methods may include, but are not limited to, using LSTM, concatenation, average pooling, and max pooling.

[0229] Step 1603: Based on the static scene data and dynamic scene data of the samples, determine the second sample graph structure data and sample environment nodes.

[0230] For example, sample preset type nodes representing sample preset type markers can be determined based on sample preset type marker data in sample static scene data; second sample graph structure data can be determined based on the sample preset type nodes and the grouping of sample preset type markers in sample static scene data. In some examples, the method for determining the second sample graph structure data is similar to the method for determining the second graph structure data, and will not be described again here.

[0231] For example, sample environment nodes can be determined based on sample environment static element data in sample static scene data and sample environment dynamic element data in sample dynamic scene data.

[0232] In this embodiment of the disclosure, since the sample lane nodes are determined based on sample lane data, second sample trajectory data, and second sample arrow data, and sample lane information, sample trajectory information, and sample arrow information are superimposed in the sample lane nodes, the lane information represented by the sample lane nodes can accurately reflect the sample lanes in the driving environment. Consequently, the first sample graph structure data determined based on the sample lane nodes and adjacency relationships can also accurately reflect each sample lane in the driving environment and the adjacency relationships between each sample lane.

[0233] In this embodiment of the disclosure, by cleaning the first sample arrow data and the first sample trajectory data, the second sample arrow data and the second sample trajectory data are obtained. Irrelevant arrow data and trajectory data can be filtered out, which can reduce the workload of subsequent embedding and feature extraction. At the same time, the dimensionality of the sample lane nodes is reduced, which is beneficial to simplifying the first sample graph structure data.

[0234] like Figure 17As shown, steps S11 and S12 can be executed by the vehicle-side reconstruction module 1701 located at the vehicle end, and the encoded data of the obtained sample dynamic objects can be transmitted to the logic reasoning module 1703. Simultaneously, the encoded data of the obtained sample static objects can be transmitted to the map feature aggregation module 1702. Step S13 is executed by the map feature aggregation module 1702, and the reconstructed data of the obtained static objects can be transmitted to the logic reasoning module 1703. Step S14 is executed by the logic reasoning module 1703, and the obtained sample logic layer data and the logical relationship between the encoded data of each sample static object and the sample logic layer data can be transmitted to the scene segmentation module 1704. The scene segmentation module 1704 executes step S15, and the obtained sample static scene data and sample dynamic scene data can be transmitted to the computational graph embedding module 1705. The graph embedding module 1305 generates first sample graph structure data, second sample graph structure data, and sample environment nodes based on sample static scene data and sample dynamic scene data. It also generates multiple sets of sample input graph structure data based on the first sample graph structure data, second sample graph structure data, and sample environment nodes. Finally, it transmits the obtained multiple sets of sample input graph structure data to the graph neural network module 1706.

[0235] In some examples, the graph neural network module 1706 may include a data validation and editing module and a ground truth annotation module required for model training. The data validation and editing module provides an entry point for manual verification and editing when the graph neural network model's accuracy in predicting the association between preset type markers and lanes is insufficient in the early stages of model training. The ground truth annotation module is used to annotate ground truth data during model training.

[0236] The implementation method for determining multiple sets of first sample graph structure data, second sample graph structure data, and environment nodes based on multiple scenario data in the cloud can refer to the aforementioned step 1002. The implementation method for determining multiple input graph structure data can be found in... Figure 11 The implementation steps 1101 to 1103 of step 2022 shown will not be described again here.

[0237] After determining the sample association relationships between preset type road signs and sample lanes in multiple sets of sample input graph structure data and corresponding sample road signs, each set of sample input graph structure data can be input into an initial graph neural network model to obtain the predicted association relationship between the preset type road signs and sample lanes. Then, based on the predicted association relationship, using this association relationship as supervision information, the initial graph neural network model is iteratively trained to obtain a trained neural network prediction model. This trained graph neural network model can predict the association relationship between preset type road signs and lanes relatively accurately.

[0238] The training process of the graph neural network model is described below through steps 1202 and 1203.

[0239] Step 1202: Process the sample input graph structure data based on the initial graph neural network model to obtain the predicted correlation between the sample preset type markers and the sample lanes.

[0240] For example, the predicted association between sample preset type markers and sample lanes includes the predicted association between each sample preset type marker and each sample lane in each sample input map structure data.

[0241] For example, after obtaining the predicted association between sample preset type markers and sample lanes, the loss value can be determined based on the predicted association between sample preset type markers and sample lanes and the sample association between sample preset type markers and sample lanes. The initial graph neural network model is then iteratively updated based on the loss value to obtain the trained graph neural network model.

[0242] In some embodiments, combined with Figure 5 ,refer to Figure 18 As shown, step 1202 above may include steps 1801 and 1802.

[0243] Step 1801: Aggregate multiple sample nodes in the sample input graph structure data based on at least one aggregation layer in the initial graph neural network model to obtain the predicted intermediate graph structure data.

[0244] Among them, multiple sample nodes include sample sign nodes for representing sample road signs and sample lane nodes for representing sample lanes.

[0245] Step 1802: Based on the decoding layer in the initial graph neural network model, process the predicted intermediate graph structure data to obtain the predicted correlation.

[0246] In some embodiments, step 1801 includes: performing local information aggregation on multiple sample nodes based on a first encoding layer in the aggregation layer to obtain first prediction subgraph structure data; performing global information aggregation on multiple sample nodes after local information aggregation in the first prediction subgraph structure data based on a second encoding layer in the aggregation layer to obtain second prediction subgraph structure data; and determining prediction intermediate graph structure data based on the second prediction subgraph data structure.

[0247] For example, the specific implementation of steps 1801 and 1802 can be referred to steps 2031 and 2032 above, and will not be repeated here.

[0248] Step 1203: Using the predicted association relationship as the initial training output of the initial graph neural network model and the sample association relationship as the supervision information, the initial graph neural network model is iteratively trained to obtain the trained graph neural network model.

[0249] In some embodiments, reference Figure 19 As shown, step 1203 may include steps 1901 to 1903.

[0250] Step 1901: Determine the loss value based on the predicted association and the sample association.

[0251] For example, the difference between a first value representing the predicted association and a second value representing the sample association can be determined as the loss value. In some examples, the first value representing the predicted association and the second value representing the sample association can be determined in a customized manner.

[0252] Step 1902: Determine the penalty function corresponding to the association between the preset type of marker and the lane.

[0253] In some implementation sets, the penalty function can be expressed as:

[0254]

[0255] in, Let L be the penalty function. edge Let BCE(A, A^) be the binary cross-entropy loss function, and L be the value of L. edge Let CE(Y, Y^) be the cross-entropy loss function, and L be the cross-entropy loss function. cross Let M be ||M*N||1; A is the connection matrix between lane nodes and traffic sign nodes, and A^ is the truth value of A; Y is the output graph structure data, and Y^ is the truth value of Y; M is the cross-association matrix constructed from the edge information of the input graph structure data; N is the query matrix with the same dimensions as M when A is expanded.

[0256] Step 1903: Based on the loss value, iteratively update the initial graph neural network model to obtain the trained graph neural network model.

[0257] The graph neural network prediction method provided in this disclosure continuously optimizes the graph neural network model by using a loss value and a penalty function corresponding to the association between preset type signs and lanes, ensuring that the trained graph neural network model can accurately predict the association between preset type signs and lanes.

[0258] In some embodiments of this disclosure, if the above-mentioned multiple sets of sample driving environment data are driving environment data of vehicles in a certain region, then after obtaining the trained graph neural network model through the above-mentioned graph neural network model training method, any vehicle driving in that region can determine the relationship between the lane and the preset type of sign based on the trained graph neural network model.

[0259] In some examples, after obtaining the trained graph neural network model, the association between lanes and preset type signs in the aforementioned area can be determined based on the trained graph neural network model, and this association can be stored in a cloud database. If this association stored in the database can be distributed to the vehicle, the vehicle can search for the association in real time based on the vehicle's driving location and determine the preset type signs associated with the lane the vehicle is traveling in real time.

[0260] In some embodiments, after the association stored in the database is sent to the vehicle, the association between the lane and the preset type of sign can be obtained through other means, and a target association can be determined based on the association obtained through other means and the association determined in real time based on the vehicle's driving position. In some examples, it can be determined whether the association obtained through other means is the same as the association determined in real time based on the vehicle's driving position, and the target association can be determined based on the determination result. If they are the same, the association is determined as the target association; if they are different, an alarm message can be issued or the association can be re-determined.

[0261] In some embodiments, after training an initial graph neural network model based on vehicle driving environment data in a certain region to obtain a trained graph neural network model, the association between the lanes traveled by vehicles in other regions and preset type signs can be predicted based on the trained graph neural network model.

[0262] For example, after training an initial graph neural network model based on the driving environment data of vehicles in region A, the trained graph neural network model can be sent to vehicles in region B, which are stored in the cloud. Vehicles in region B can then use the trained graph neural network model to predict in real time the preset type of markers associated with the lane they are traveling in.

[0263] For example, after training an initial graph neural network model based on the driving environment data of vehicles in region A, the driving environment data of vehicles in region B can be uploaded to the cloud. The trained graph neural network model, stored in the cloud, can then process the driving environment data of vehicles in region B to obtain the association between lanes and preset type markers within region B. If this association for region B is distributed to vehicles in region B, the vehicles can then use their driving location to find the association in real time and determine the preset type markers associated with their lane.

[0264] Exemplary device

[0265] Figure 20 This is a schematic diagram of the structural composition of a device for predicting the relationship between signs and lanes, provided in an exemplary embodiment of this disclosure. Figure 20 As shown, the device 2000 for predicting the relationship between markers and lanes includes a first determining module 2001, a second determining module 2002, and a predicting module 2003.

[0266] The first determining module 2001 is used to determine the driving environment data collected by the vehicle during driving.

[0267] The second determining module 2002 is used to determine, based on driving environment data, the input graph structure data used to represent the initial relationship between road signs and lanes.

[0268] The prediction module 2003 is used to process the input graph structure data based on the graph neural network model to obtain the association between the preset type of road signs and the lanes.

[0269] In some embodiments, such as Figure 21 As shown, Figure 20 The prediction module 2003 includes an aggregation unit 2101 and a processing unit 2102.

[0270] Aggregation unit 2101 is used to aggregate multiple nodes in the input graph structure data based on at least one aggregation layer in the graph neural network model to obtain intermediate graph structure data; the multiple nodes include sign nodes for representing road signs and lane nodes for representing lanes.

[0271] The processing unit 2102 is used to process the intermediate graph structure data based on the decoding layer in the graph neural network model to obtain the association relationship between preset type signs and lanes.

[0272] In some embodiments, such as Figure 22 As shown, Figure 21The aggregation unit 2101 includes a first sub-aggregation unit 2201, a second sub-aggregation unit 2202, and a first sub-determining unit 2203.

[0273] The first sub-aggregation unit 2201 is used to process the intermediate graph structure data based on the decoding layer in the graph neural network model to obtain the association relationship between preset type markers and lanes.

[0274] The second sub-aggregation unit 2202 is used to perform global information aggregation on multiple nodes after local information aggregation in the first subgraph structure data based on the second encoding layer in the aggregation layer, so as to obtain the second subgraph structure data.

[0275] The first sub-determination unit 2203 is used to determine the intermediate graph structure data based on the second sub-graph structure data.

[0276] In some embodiments, such as Figure 23 As shown, the first determining module 2001 includes a first determining unit 2301 and a second determining unit 2302.

[0277] The first determining unit 2301 is used to determine, based on driving environment data, first graph structure data for representing the initial relationship between lanes, second graph structure data for representing the initial relationship between preset type signs, and environmental nodes for representing environmental signs in road signs.

[0278] The second determining unit 2302 is used to determine the input graph structure data based on the first graph structure data, the second graph structure data, and the environment nodes.

[0279] In some embodiments, such as Figure 24 As shown, the first determining unit 2301 includes a second sub-determining unit 2401 and a third sub-determining unit 2402.

[0280] The second sub-determination unit 2401 is used to determine static scene data representing the driving environment of the vehicle based on driving environment data.

[0281] The third sub-determination unit 2402 is used to determine the first graph structure data, the second graph structure data, and the environment nodes based on static scene data.

[0282] In some embodiments, such as Figure 25 As shown, the second determining unit 2302 includes a fourth sub-determining unit 2501, a fifth sub-determining unit 2502, and a sixth sub-determining unit 2503.

[0283] The fourth sub-determination unit 2501 is used to determine the first initial relationship between the environmental node and each lane node in the first graph structure data, and the second initial relationship between the environmental node and each preset type node in the second graph structure data.

[0284] The fifth sub-determination unit 2502 is used to determine the third initial relationship between each lane node and each preset type node.

[0285] The sixth sub-determination unit 2503 is used to determine the input graph structure data based on the first initial relation, the second initial relation, and the third initial relation.

[0286] Figure 26 This is a schematic diagram of the structural composition of a graph neural network model training device provided in an exemplary embodiment of this disclosure. Figure 26 As shown, the neural network model training device 2600 includes a determination module 2601, a prediction module 2602, and a training module 2603.

[0287] The determination module 2601 is used to determine the sample association relationship between sample preset type signs and sample lanes in multiple sets of sample input map structure data and sample road signs corresponding to the sample input map structure data; wherein, the sample input map structure data is used to represent the initial relationship between sample road signs and sample lanes.

[0288] The prediction module 2602 is used to process the sample input graph structure data based on the initial graph neural network model to obtain the predicted correlation between the sample preset type markers and the sample lanes.

[0289] The training module 2603 is used to iteratively train the initial graph neural network model by using the predicted association relationship determined by the prediction module 2602 as the initial training output of the initial graph neural network model and the sample association relationship as the supervision information to obtain the trained graph neural network model.

[0290] In some embodiments, the prediction module 2602 is specifically used to aggregate multiple sample nodes in the sample input graph structure data based on at least one aggregation layer in the initial graph neural network model to obtain the predicted intermediate graph structure data; wherein, the multiple sample nodes include sample sign nodes for representing sample road signs and sample lane nodes for representing sample lanes; and the predicted intermediate graph structure data is processed based on the decoding layer in the initial graph neural network model to obtain the predicted correlation.

[0291] In some embodiments, the training module 2603 is specifically used to determine the loss value based on the predicted association relationship and the sample association relationship; and to iteratively update the initial graph neural network model based on the loss value to obtain the trained graph neural network model.

[0292] Exemplary electronic devices

[0293] Figure 27 This is a structural diagram of an electronic device provided in an exemplary embodiment of the present disclosure, such as... Figure 27 As shown, the electronic device 2700 includes at least one processor 2701 and a memory 2702.

[0294] The processor 2701 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 2300 to perform desired functions.

[0295] The memory 2702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 2701 may execute one or more computer program instructions to implement the methods for predicting the association between markers and lanes, graph neural network model training methods, and / or other desired functions of the various embodiments of this disclosure described above.

[0296] In some examples, the electronic device 2700 may also include an input device 2703 and an output device 2704, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0297] Of course, for the sake of simplicity, Figure 27 Only some of the components of the electronic device 2700 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 2700 may include any other suitable components depending on the specific application.

[0298] Exemplary computer program products and computer-readable storage media

[0299] In addition to the methods and devices described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform steps in the methods for predicting the association between signs and lanes or the graph neural network model training methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0300] Furthermore, embodiments of this disclosure also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform steps in the methods for predicting the association between signs and lanes or the graph neural network model training methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0301] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires; portable disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); optical fiber; compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof.

[0302] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0303] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for predicting the association between a sign and a lane, comprising: Determine the driving environment data collected during vehicle operation; Based on the driving environment data, input graph structure data for representing the initial relationship between road signs and lanes is determined; The input graph structure data is processed based on a graph neural network model to obtain the association between the preset type of road signs and the lanes; The step of determining the input graph structure data for representing the initial relationship between road signs and lanes based on the driving environment data includes: Based on the driving environment data, first graph structure data for representing the initial relationship between lanes, second graph structure data for representing the initial relationship between various preset types of signs, and environmental nodes for representing environmental signs among the road signs are determined; based on the first graph structure data, the second graph structure data, and the environmental nodes, the input graph structure data is determined; The preset type markers include traffic lights; determining the second graph structure data includes: determining static scene data representing the vehicle's driving environment based on the driving environment data; and determining the second graph structure data based on the preset type marker data and the grouping of each preset type marker in the static scene data.

2. The method according to claim 1, wherein, The process of processing the input graph structure data based on a graph neural network model to obtain the association between preset type road signs and lanes includes: At least one aggregation layer in the graph neural network model is used to aggregate multiple nodes in the input graph structure data to obtain intermediate graph structure data; the multiple nodes include sign nodes for representing road signs and lane nodes for representing lanes; The intermediate graph structure data is processed based on the decoding layer in the graph neural network model to obtain the association between the preset type of marker and the lane.

3. The method according to claim 2, wherein, The process of aggregating multiple nodes in the input graph structure data based on at least one aggregation layer in the graph neural network model to obtain intermediate graph structure data includes: Based on the first encoding layer in the aggregation layer, local information aggregation is performed on the multiple nodes to obtain the first subgraph structure data; Based on the second encoding layer in the aggregation layer, global information aggregation is performed on the multiple nodes after local information aggregation in the first subgraph structure data to obtain the second subgraph structure data. The intermediate graph structure data is determined based on the second subgraph structure data.

4. The method according to claim 1, wherein, The step of determining, based on the driving environment data, first graph structure data for representing the initial relationship between lanes, second graph structure data for representing the initial relationship between various preset types of signs, and environmental nodes for representing environmental signs among the road signs includes: Based on the driving environment data, static scene data representing the driving environment of the vehicle is determined; Based on the static scene data, the first graph structure data, the second graph structure data, and the environment nodes are determined.

5. The method according to claim 1, wherein, The step of determining the input graph structure data based on the first graph structure data, the second graph structure data, and the environment nodes includes: Determine a first initial relationship between the environmental node and each lane node in the first graph structure data, and a second initial relationship between the environmental node and each preset type node in the second graph structure data; Determine the third initial relationship between each lane node and each preset type node; The input graph structure data is determined based on the first initial relation, the second initial relation, and the third initial relation.

6. A method for training a graph neural network model, comprising: Determine the sample association relationship between sample preset type signs and sample lanes in multiple sets of sample input map structure data and the sample road signs corresponding to the sample input map structure data; wherein, the sample input map structure data is used to represent the initial relationship between the sample road signs and the sample lanes; The sample input graph structure data is processed based on the initial graph neural network model to obtain the predicted correlation between the sample preset type markers and the sample lanes; Using the predicted association relationship as the initial training output of the initial graph neural network model and the sample association relationship as supervision information, the initial graph neural network model is iteratively trained to obtain the trained graph neural network model; The determination of multiple sets of sample input graph structure data includes: Based on the driving environment data of each set of samples, a first sample graph structure data is determined to represent the initial relationship between each sample lane, a second sample graph structure data is determined to represent the initial relationship between each sample preset type marker, and a sample environment node is determined to represent the sample environment marker in the sample road marker; based on multiple sets of first sample graph structure data, second sample graph structure data, and sample environment nodes, multiple sets of sample input graph structure data are determined. The sample preset type markers include traffic lights; determining the second sample graph structure data includes: determining sample static scene data to represent the sample vehicle's driving environment based on the sample driving environment data; and determining the second sample graph structure data based on the sample preset type marker data and the group of each sample preset type marker in the sample static scene data.

7. The method according to claim 6, wherein, The process of processing the sample input graph structure data based on the initial graph neural network model to obtain the predicted correlation between the sample preset type markers and the sample lanes includes: Based on at least one aggregation layer in the initial graph neural network model, multiple sample nodes in the sample input graph structure data are aggregated to obtain predicted intermediate graph structure data; wherein, the multiple sample nodes include sample sign nodes for representing the sample road signs and sample lane nodes for representing the sample lanes; The predicted intermediate graph structure data is processed based on the decoding layer in the initial graph neural network model to obtain the predicted correlation.

8. The method according to claim 6, wherein, The step of using the predicted association relationship as the initial training output of the initial graph neural network model and the sample association relationship as supervision information to iteratively train the initial graph neural network model to obtain the trained graph neural network model includes: Based on the predicted correlation and the sample correlation, the loss value is determined; Based on the loss value, the initial graph neural network model is iteratively updated to obtain the trained graph neural network model.

9. An apparatus for predicting the relationship between a sign and a lane, comprising: The first determining module is used to determine the driving environment data collected by the vehicle during driving; The second determining module is used to determine, based on the driving environment data, the input graph structure data representing the initial relationship between road signs and lanes; The prediction module is used to process the input graph structure data based on a graph neural network model to obtain the association between the preset type of road signs and the lanes; The second determining module includes: a first determining unit, configured to determine, based on driving environment data, first graph structure data representing the initial relationship between lanes, second graph structure data representing the initial relationship between preset type signs, and environmental nodes representing environmental signs among road signs; a second determining unit, configured to determine input graph structure data based on the first graph structure data, the second graph structure data, and the environmental nodes; wherein, the preset type signs include traffic lights; determining the second graph structure data includes: determining, based on the driving environment data, static scene data representing the driving environment of the vehicle; and determining the second graph structure data based on the preset type sign data and the groups of each preset type sign in the static scene data.

10. A graph neural network model training device, comprising: A determining module is used to determine the sample association relationship between sample preset type signs and sample lanes in multiple sets of sample input map structure data and the sample road signs corresponding to the sample input map structure data; wherein, the sample input map structure data is used to represent the initial relationship between the sample road signs and the sample lanes; determining multiple sets of sample input map structure data includes: determining first sample map structure data to represent the initial relationship between sample lanes, second sample map structure data to represent the initial relationship between sample preset type signs, and sample environment nodes to represent sample environment signs in the sample road signs based on each set of sample driving environment data; determining multiple sets of sample input map structure data based on multiple sets of first sample map structure data, second sample map structure data, and sample environment nodes; wherein, the sample preset type signs include traffic lights; determining the second sample map structure data includes: determining sample static scene data to represent the sample driving environment of sample vehicles based on the sample driving environment data; determining the second sample map structure data based on the sample preset type sign data in the sample static scene data and the group of each sample preset type sign; The processing module is used to process the sample input graph structure data based on the initial graph neural network model to obtain the predicted correlation between the sample preset type markers and the sample lanes; The training module is used to iteratively train the initial graph neural network model using the predicted association relationship as the initial training output of the initial graph neural network model and the sample association relationship as supervision information to obtain the trained graph neural network model.

11. A computer-readable storage medium storing a computer program for performing a method for predicting the association between a signpost and a lane as described in any one of claims 1-5, or for performing a graph neural network model training method as described in any one of claims 6-8.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for predicting the association between markers and lanes as described in any one of claims 1-5, or to implement the graph neural network model training method as described in any one of claims 6-8.

Images