Driving scene classification and labeling method and device based on vehicle interaction

Through the driving scene classification method based on vehicle interaction, the scene space-time tree and graph structure calculation tree are used, combined with dynamic time regularization algorithm, the complexity problem of vehicle interaction scene data annotation in the field of autonomous driving is solved, and the precise classification of driving scenes is realized. Annotation improves the generalization ability of the algorithm.

CN118968428APending Publication Date: 2024-11-15TSINGHUA UNIVERSITY
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410914027.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the field of autonomous driving, it is difficult for the existing technology to effectively classify and label complex vehicle interaction scenario data, resulting in insufficient generalization capabilities of training and verification algorithms, especially in different types of scenarios and environments.

Method used

Through the driving scene classification method based on vehicle interaction, the scene space-time tree and graph structure calculation tree are used, combined with dynamic time regularization algorithm, the vehicle's interaction type and labeling results are determined, and a clear scene classification level and efficient comparison method are constructed.

Benefits of technology

It realizes accurate annotation and classification of driving scenarios, provides efficient data processing methods, can adapt to different types of scenarios and environments, and improves the generalization ability of algorithms and the verification ability of extremely challenging scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968428A_ABST
    Figure CN118968428A_ABST
Patent Text Reader

Abstract

The invention is suitable for the technical field of automatic driving, and discloses a driving scene classification and labeling method, device and equipment based on vehicle interaction and a medium, and the method comprises the steps: determining the interaction type of a vehicle in a driving scene based on the interaction relation of the vehicle movement flow direction in the driving scene; acquiring first sample data of a driving scene; based on a scene space-time tree and the interaction type of the vehicle, slicing and classifying the first sample data of the driving scene to obtain second sample data of the driving scene; and based on a scene measurement standard combining a graph structure calculation tree and dynamic time warping, obtaining a labeling result of each scene in the second sample data of the driving scene. According to the technical scheme, the clear scene classification hierarchy is constructed by defining the basic driving scene type and combining the space-time tree slices; a driving scene graph structure measurement standard is adopted, and an efficient driving scene comparison method is provided, so that the driving scene is accurately marked.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure belongs to the field of autonomous driving technology, and in particular relates to a method, device, equipment and medium for classifying and labeling driving scenes based on vehicle interaction. Background Art

[0002] In the field of autonomous driving, a large amount of scene data is needed in autonomous driving testing, autonomous driving prediction decision model training, etc. In order to make full use of the scene data, it is very important to describe and label the interaction information of vehicles. On the one hand, the module needs to accurately extract the complex patterns and interactions of traffic participants. For training and optimization, appropriate descriptions and labels must be obtained as supervision. On the other hand, the module should adapt to different types of scenes and environments, and scene samples need to be balanced to maintain generalization. Sometimes it is necessary to select extremely challenging scenes with complex interactions to verify the designed algorithm. Therefore, it is necessary to build a clear scene classification hierarchy and an efficient scene comparison method based on vehicle interaction to help classify and label a large amount of scene data. Summary of the invention

[0003] To solve the above problems, the present disclosure provides a method, device, equipment and medium for classifying and labeling driving scenes based on vehicle interaction.

[0004] To achieve the above objectives, the present disclosure adopts the following technical solutions:

[0005] In a first aspect, the present disclosure provides a driving scene classification and labeling method based on vehicle interaction, comprising:

[0006] Determine the interaction type of vehicles in the driving scene based on the interactive relationship between the movement directions of vehicles in the driving scene;

[0007] Acquire first sample data of the driving scene;

[0008] Based on the scene spatiotemporal tree and the interaction type of the vehicle, the first sample data of the driving scene is sliced ​​and classified to obtain the second sample data of the driving scene;

[0009] Based on the scene metric standard combining graph structure computation tree and dynamic time warping, the labeling results of each scene in the second sample data of the driving scene are obtained.

[0010] Furthermore, the types of vehicle interactions in the driving scenario include: dynamic following flow, dynamic oncoming flow, dynamic conflict line, static conflict line, and static conflict line.

[0011] Furthermore, based on the scene spatiotemporal tree and the interaction type of the vehicle, the first sample data of the driving scene is sliced ​​and classified to obtain the second sample data of the driving scene, including:

[0012] Specify the ego vehicle and the spatial range of the ego vehicle in the first sample data of the driving scene;

[0013] Extracting surrounding vehicle data within the spatial range of the vehicle from the first sample data of the driving scene;

[0014] If the vehicle corresponding to the surrounding vehicle data has no interaction relationship with the vehicle itself, filter the surrounding vehicle data;

[0015] Based on the time window method, the interaction status between the surrounding vehicle data and the vehicle itself is determined;

[0016] The surrounding vehicle data having the same interaction type and interaction area as the vehicle itself are merged to obtain the second sample data of the driving scene.

[0017] Furthermore, based on the scene metric standard combining the graph structure computation tree and dynamic time warping, the annotation results of each scene in the second sample data of the driving scene are obtained, including:

[0018] Based on the graph structure calculation tree comparison algorithm, the optimal transmission distance corresponding to each scene in the second sample data of the driving scene is calculated;

[0019] Based on the dynamic time warping algorithm, the dynamic time warping distance of each scene in the second sample data of the driving scene is calculated;

[0020] Based on the optimal transmission distance and the dynamic time warping distance, a cluster analysis is performed on the second sample data of the driving scene to obtain the labeling results of each scene in the second sample data of the driving scene.

[0021] In a second aspect, the present disclosure further provides a driving scene classification and labeling device based on vehicle interaction, comprising:

[0022] A determination module, for determining the interaction type of vehicles in the driving scene based on the interaction relationship of the movement flow of the vehicles in the driving scene;

[0023] An acquisition module, used to acquire first sample data of a driving scene;

[0024] A slicing module, used for slicing and classifying the first sample data of the driving scene based on the scene spatiotemporal tree and the interaction type of the vehicle, to obtain the second sample data of the driving scene;

[0025] The labeling module is used to obtain the labeling results of each scene in the second sample data of the driving scene based on the scene metric standard combined with the graph structure calculation tree and dynamic time warping.

[0026] Furthermore, the types of vehicle interactions in the driving scenario include: dynamic following flow, dynamic oncoming flow, dynamic conflict line, static conflict line, and static conflict line.

[0027] Furthermore, the slicing module is also used to:

[0028] Specify the ego vehicle and the spatial range of the ego vehicle in the first sample data of the driving scene;

[0029] Extracting surrounding vehicle data within the spatial range of the vehicle from the first sample data of the driving scene;

[0030] If the vehicle corresponding to the surrounding vehicle data has no interaction relationship with the vehicle itself, filter the surrounding vehicle data;

[0031] Based on the time window method, the interaction status between the surrounding vehicle data and the vehicle itself is determined;

[0032] The surrounding vehicle data having the same interaction type and interaction area as the vehicle itself are merged to obtain the second sample data of the driving scene.

[0033] Furthermore, the annotation module is also used to:

[0034] Based on the graph structure calculation tree comparison algorithm, the optimal transmission distance corresponding to each scene in the second sample data of the driving scene is calculated;

[0035] Based on the dynamic time warping algorithm, the dynamic time warping distance of each scene in the second sample data of the driving scene is calculated;

[0036] Based on the optimal transmission distance and the dynamic time warping distance, a cluster analysis is performed on the second sample data of the driving scene to obtain the labeling results of each scene in the second sample data of the driving scene.

[0037] In a third aspect, the present disclosure further provides an electronic device, including: a processor and a memory;

[0038] The processor is coupled to the memory;

[0039] The processor is used to read and execute the program or instructions stored in the memory, so that the device executes the method of the first aspect.

[0040] In a fourth aspect, the present disclosure further provides a computer-readable storage medium storing a computer program, which implements the method of the first aspect when executed by a processor.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The technical solution disclosed in the present invention provides a new driving scene classification and annotation method, defines basic driving scene types, classifies driving scenes based on vehicle interactions in combination with spatiotemporal tree slices, converts driving scenes into graph structures, and then converts them into vehicle-to-vehicle interaction calculation trees and vehicle-to-road interaction calculation trees for detailed comparison and annotation. Therefore, the technical solution disclosed in the present invention constructs a clear scene classification hierarchy by defining basic driving scene types in combination with spatiotemporal tree slices; adopts a driving scene graph structure metric standard, provides an efficient driving scene comparison method, and thus achieves accurate annotation of driving scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 The flow chart of a driving scene classification and labeling method based on vehicle interaction in an embodiment of the present disclosure is shown;

[0045] Figure 2 The basic vehicle interaction types in the embodiments of the present disclosure are shown;

[0046] Figure 3 The scene spatiotemporal tree slices and classification scenes in the embodiment of the present disclosure are shown;

[0047] Figure 4 The optimal transmission distance of the scene frame graph computation tree in the embodiment of the present disclosure is shown;

[0048] Figure 5 The time-warping distance of a dynamic scene in an embodiment of the present disclosure is shown;

[0049] Figure 6 The scene refinement annotation of static conflict line interaction of Interaction dataset in the embodiment of the present disclosure is shown;

[0050] Figure 7 The edge risk scenario of the graph structure metric expansion in the embodiment of the present disclosure is shown;

[0051] Figure 8 The structure of a driving scene classification and labeling device based on vehicle interaction in an embodiment of the present disclosure is shown;

[0052] Fig. 9 An electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0054] The present disclosure provides a driving scene classification and labeling method based on vehicle interaction. By defining basic driving scene types and combining them with spatiotemporal tree slices, a clear scene classification hierarchy is constructed. By adopting a driving scene graph structure metric standard, an efficient driving scene comparison method is provided, thereby achieving accurate labeling of driving scenes.

[0055] Figure 1 The process of a driving scene classification and labeling method based on vehicle interaction in an embodiment of the present disclosure is shown. As shown in the figure, the method includes:

[0056] S101, determining an interaction type of vehicles in the driving scene based on an interaction relationship of movement directions of vehicles in the driving scene;

[0057] Vehicle interaction refers to the interaction between vehicles, that is, the driving behavior of vehicles using at least two roads, which is affected by the possibility that vehicles plan to occupy the same spatial area at the same time in the near future. The vehicle interaction mechanism is complex and varies with the driving strategy. However, considering the numerous parameters of the scene will lead to high computational complexity and cannot achieve complete classification. In order to simplify the problem and conform to the experience of human drivers, according to the above definition of vehicle interaction, the vehicle interaction in the driving scene in this scheme refers to the interaction between the selected target vehicle and the vehicles in the adjacent lane, that is, the interaction between the self-vehicle and the surrounding vehicles. Correspondingly, the driving scene in this scheme also refers to the driving scene composed of the selected target vehicle and the vehicles in the adjacent lane, that is, the driving scene composed of the self-vehicle and the surrounding vehicles. When the self-vehicle in the vehicle interaction interacts with the surrounding vehicles, the vehicles participating in the interaction need to check the potential conflict area and collision, and negotiate to determine the right of way, so as to maintain an appropriate balance between safety and efficiency. The conflict area refers to the area where the vehicle conflicts with other vehicles to use the same lane when crossing, merging or leaving the lane. The conflict area in this scheme also refers to the interaction area between the self-vehicle and the surrounding vehicles.

[0058] Based on the simplified interaction process mentioned above, we can summarize the interaction types of vehicles in driving scenarios mainly according to the interactive relationship of vehicle movement flow, and then divide the complex interaction fragments.

[0059] In this scheme, the interaction areas between the ego vehicle and surrounding vehicles are first divided into two types: dynamic interaction areas and static interaction areas. Dynamic interaction areas refer to areas where conflicts and collisions may occur due to various actions of vehicles. These areas are usually local, and their positions and sizes will change with the dynamic movement of targets and vehicles, and the right of way will be dynamically determined as the scene evolves. When the vehicle completes the behavior and interaction process, the dynamic conflict area will also disappear. The vehicle completes the behavior and interaction process here, which refers to the behavior and interaction process that resolves the conflict between vehicles. For example, when the ego vehicle is changing lanes, the vehicle behind in the adjacent lane accelerates to overtake. At this time, the ego vehicle needs to slow down and wait before changing lanes. Static interaction areas are relatively fixed areas where some vehicles need to interact with other vehicles and adjust their movements. Static interaction areas are usually key bottleneck areas caused by changes in road geometry (such as ramps and intersections) or road occupation. Key bottleneck areas, such as ramps and intersections, are natural conflict areas caused by road structure restrictions and are bottlenecks that affect traffic safety and efficiency.

[0060] Furthermore, the basic vehicle interaction processes in the dynamic interaction area can be divided into the following types:

[0061] 1) Dynamic following flow: The following area between the following vehicle pairs is prone to forward collision with the following vehicle in the driving direction.

[0062] 2) Dynamic oncoming flow. The two vehicles are traveling in opposite directions. This situation is mainly caused by vehicles driving in the opposite direction on the road or unclear lane divisions.

[0063] 3) Dynamic conflict line. Generally refers to the flow of vehicles that seize the right of way of other vehicles during driving. The most common behavior is lane change. That is, the target area of ​​the vehicles is the same and the movement flow overlaps. In the interactive decision-making process of dynamic conflict flow, the vehicle needs to interact with the vehicles in the current lane and adjacent lanes, and consider when and where to perform operations. The interactive decision-making process of dynamic conflict flow refers to the decision-making and action response of the vehicle when dealing with conflict relationships.

[0064] The basic vehicle interaction processes in the static interaction area can be divided into the following types:

[0065] 1) Static conflict line. Refers to the vehicle conflict line in the static interaction area. Common behavior is vehicle merging, such as ramps and main lanes. Vehicles in the two lanes drive to the same area, that is, moving from different lanes to the same lane under road constraints. Road constraints such as ramp merging, no space ahead, and mandatory lane changes are required. In the decision-making process from entering the control area to completing the task, the self-vehicle needs to interact with surrounding vehicles with the same target in the conflict area and consider when to pass. For example, when merging, the ramp vehicle and the main lane vehicle have the same driving target area, and the main lane vehicle considers when to pass.

[0066] 2) Static conflict points. Generally, it refers to the intersection of vehicles in the static conflict area. Intersection flow refers to the intersection of vehicle movement flow, such as straight flow and turning flow in different directions at the intersection. During the whole process, the vehicle will interact with the surrounding areas with trajectory intersections in the conflict area and determine the order of passing. Trajectory intersection refers to the intersection of vehicle movement flow in the future period of time, that is, the intersection of vehicle driving trajectories.

[0067] To summarize, the interaction types of vehicles in the driving scenario are determined, including: dynamic following flow, dynamic oncoming flow, dynamic conflict line, static conflict line, and static conflict line.

[0068] S102, obtaining first sample data of a driving scene;

[0069] Scene data includes map environment data and vehicle motion data within a certain time and space range, describing the interaction and decision-making process of the vehicle in the driving environment. Scene data is collected by sensors such as vehicle cameras and radars, as well as roadside cameras, drones and other equipment. The original sensor data is processed by target detection, image segmentation, trajectory filtering, etc., and converted into map data and spatiotemporal data of vehicle motion to form a scene database. The data in the scene database is the first sample data of the driving scene. The data of vehicle motion within a certain time and space range refers to the data of the motion of the vehicle and surrounding vehicles (for example, the motion data of the vehicle and surrounding vehicles within a 50m area around the vehicle in the next 10s).

[0070] S103: Slice and classify the first sample data of the driving scene based on the scene spatiotemporal tree and the interaction type of the vehicle to obtain the second sample data of the driving scene, including:

[0071] Specify the ego vehicle and the spatial range of the ego vehicle in the first sample data of the driving scene;

[0072] Extracting surrounding vehicle data within the spatial range of the vehicle from the first sample data of the driving scene;

[0073] If the vehicle corresponding to the surrounding vehicle data has no interaction relationship with the vehicle itself, filter the surrounding vehicle data;

[0074] Based on the time window method, the interaction status between the surrounding vehicle data and the vehicle itself is determined;

[0075] The surrounding vehicle data having the same interaction type and interaction area as the vehicle itself are merged to obtain the second sample data of the driving scene.

[0076] Exemplary, the details are as follows:

[0077] Slicing and classifying scene data based on vehicle interactions requires solving the following problems:

[0078] 1) Which vehicles? We need to search for vehicles moving in the same spatial area as our vehicle. The same spatial area here refers to an area within a certain spatial range around the current vehicle, such as an area within 50 meters from the vehicle.

[0079] 2) Is it interactive? It is necessary to determine whether the searched vehicle will conflict with the self-vehicle at present or in the near future based on the movement flow.

[0080] 3) When is the interaction? If the vehicle is detected to be interacting, it should be further checked whether the vehicle is in the process of interaction in the current frame, and the start and end time of the interaction should be extracted.

[0081] 4) What kind of interaction? It is necessary to clarify the type of interaction between the surrounding vehicles and the self-vehicle, and whether consecutive frames can be merged into a scene segment.

[0082] These four aspects constitute the properties of the scene fragment.

[0083] Based on the above four aspects, the scene spatiotemporal tree is constructed. These four aspects correspond to the four levels of the scene spatiotemporal tree. Each scene segment needs to be obtained through the four levels of the scene spatiotemporal tree.

[0084] 1) Search layer. In each frame of the scene, the vehicle data around the self-vehicle is extracted within the spatial range determined by the characteristics and distribution of the driving environment, that is, the trajectory data of the surrounding vehicles in the time and space window. When the vehicle density is high and congestion occurs, the vehicle only needs to interact with the adjacent vehicles, and the spatial range can be appropriately reduced; when the vehicle density is low, the spatial range needs to be appropriately expanded. Exemplarily, spatiotemporal information is retrieved from the scene database to extract the surrounding vehicles within a 50m area around the self-vehicle and the movement trajectory for the next 10s. The vehicle data around the self-vehicle searched within a certain spatial range around the self-vehicle needs to be temporarily stored in the memory buffer for fast retrieval and extraction, waiting for the operation judgment of the subsequent layer. The vehicle data around the self-vehicle has the same meaning as the surrounding vehicle data in this scheme. The surrounding vehicle data includes the vehicle's status data and relationship data. The status data includes the vehicle's position, speed, heading angle and other information, and the relationship data includes the distance and orientation relationship information of the vehicle relative to the self-vehicle.

[0085] 2) Interaction filtering layer. Through the extracted surrounding vehicle data and the annotation of the lane area, it is detected whether the target trajectory of the searched vehicle overlaps or intersects with the future flow direction of the ego vehicle. The target trajectory of the vehicle is composed of the vehicle's trajectory points. The trajectory points are a series of discrete (x, y) coordinate points. The line segment intersection detection algorithm is used to determine whether the vehicle's trajectory lines intersect, that is, whether there is an interactive relationship. If there is an intersection, the vehicle will be temporarily retained in the buffer as a vehicle that interacts with the ego vehicle. If not, the vehicle will be removed as a non-interactive vehicle, that is, deleted from the buffer to save space. The annotation of the lane area (which belongs to the characteristic data of the above-mentioned driving environment) indicates the lane where the marked vehicle is located to distinguish whether the vehicle conflict is a dynamic conflict or a static conflict, and further distinguish whether it is a following flow or a conflict line.

[0086] 3) Time filtering layer. This layer determines whether the current vehicle has started or ended the interaction with the ego vehicle. The system can extract the typical state of the vehicle within a time span, such as the lateral speed or direction during the lane change process, the steering angle during the turning process, etc. The typical state is determined by the specific interaction type. For example, in the case of dynamic conflict line interaction, the ego vehicle interacts with other vehicles during the lane change process, and the lateral speed or vehicle orientation angle can be extracted to determine when the interaction starts and ends; another example is static conflict point interaction, where the ego vehicle interacts with other vehicles during the turning process, and the vehicle steering angle and other states can be extracted to determine when the interaction starts and ends. By setting a time window, it is determined whether all state values ​​within the window meet the threshold requirements. The time window method can effectively help determine whether the vehicle starts or ends the corresponding action and the interaction with the ego vehicle. The length of the time window is a preset parameter, which is also determined by characteristics such as vehicle density and flow in the scene database. Within the time window, determine whether the extracted state value meets the threshold requirement. For example, when the vehicle changes lanes in an environment where there are neighboring vehicles or surrounding vehicles, if the lateral speed of the vehicle at this moment is higher than 0.6 m / s, it means that the lane change is in progress and interacting with the surrounding vehicles, and it is marked as interacting. If it is lower than 0.6 m / s, it may be that the lane change has not started or is about to end, and it is marked as no interaction.

[0087] 4) Merging layer. This layer compares the vehicle interactions of consecutive static scene frames based on the interaction information extracted by the above layers, that is, which vehicles, when and what kind of interaction occurred, and then merges the static scene frames into a scene segment. The condition for merging is that the interaction type and corresponding interaction area between the vehicles at the previous and next moments are the same. That is, compare the static scene frames at each moment in the time series to see whether the IDs of the interacting vehicles are consistent, whether the interaction status (interacting or not interacting) is consistent, and whether the interaction type is consistent. They can be merged only if they are consistent. A series of merged static scene frames constitute a dynamic scene. For example, Figure 3 As shown, the static scene frames corresponding to the ellipsis can be merged, where the interacting vehicles and interaction types are the same. Figure 3 (2) and Figure 3(3) In the corresponding scenes, the interacting vehicles are the same, but the interaction types are different, so they cannot be merged. Vehicles are numbered in the order they appear. When the number of interacting vehicles reaches a certain threshold (such as 10), the merging process will switch to a new scene segment. Scene segments also refer to dynamic scenes.

[0088] The scene fragment obtained through the above four-layer operation based on the scene spatiotemporal tree and the interaction type of the vehicle is the second sample data of the driving scene.

[0089] S104, based on the scene metric standard combining the graph structure computation tree and dynamic time warping, obtain the labeling results of each scene in the second sample data of the driving scene, including:

[0090] Based on the graph structure calculation tree comparison algorithm, the optimal transmission distance corresponding to each scene in the second sample data of the driving scene is calculated;

[0091] Based on the dynamic time warping algorithm, the dynamic time warping distance of each scene in the second sample data of the driving scene is calculated;

[0092] Based on the optimal transmission distance and the dynamic time warping distance, a cluster analysis is performed on the second sample data of the driving scene to obtain the labeling results of each scene in the second sample data of the driving scene.

[0093] Exemplary, the details are as follows:

[0094] This solution proposes a scene metric based on a combination of graph structure computation tree and dynamic time warping, compares the dynamic scenes of the same interaction type in the second sample data of the driving scene mentioned above, and compares the distances between the dynamic scenes to achieve detailed comparison and annotation of the scenes. The detailed annotation of the scene is mainly divided into two steps. The first is to connect the scene elements in the corresponding static scene frames in the dynamic scene into a graph structure, compare the graph distances of the static scene frames, and calculate the optimal transmission distance. The second is to calculate the time-warping distance of the dynamic scene. Scene elements include vehicles and road nodes. Static scene frames refer to static scenes at a single moment. A series of static scene frames are arranged in time to form a dynamic scene. The DTW (Dynamic Time Warping) algorithm can calculate the similarity of two time series, and is particularly suitable for time series of different lengths and rhythms.

[0095] For static scene frames, the optimal transmission distance is calculated based on the graph structure calculation tree comparison algorithm, such as Figure 4 As shown, the scene graph structure is first converted into a vehicle-to-vehicle (V2V) computation tree T V2V =(E R ,E V ,R V2V ) and vehicle-to-road interaction (V2N) computation tree T V2N =(E R,E N ,E V ,R V2N ). Where E R The root node of the tree is the car, E V Refers to other vehicles that have an interactive relationship with the self-vehicle, E N Refers to other road nodes that have an interactive relationship with the vehicle, R V2V Refers to the relationship and connection between vehicles, R V2N Refers to the relationships and connections between vehicles and road nodes.

[0096] For a V2V computation tree node u, define It means that the V2V computation tree with u as the root node in the first layer contains the node {u}. As the computation tree of the lth layer, Each vehicle in the V2N is connected to each vehicle that has an interactive relationship with it. For the computation tree node u of V2N, define It means that the V2N computing tree with u as the root node in the first layer contains the nodes {u}. As the computation tree of the lth layer, The vehicle nodes / road nodes of the ego vehicle are connected to other types of nodes that interact with them. The nodes of the V2V computation tree only contain vehicles, and the edges are the relationships between vehicles. In the V2N computation tree, the nodes of the odd-numbered layers are all vehicles, and the nodes of the even-numbered layers are all road network nodes. The edges are the interaction relationships between vehicles and road nodes. The comparison results of each layer in the graph structure computation tree will be summarized to the ego vehicle, so the upper layer represents the explicit interaction related to the ego vehicle, and the lower layer represents the implicit interaction of the surrounding vehicles. Explicit interaction refers to vehicles that intersect with the ego vehicle's movement flow, and have direct edge connections with the ego vehicle in the graph structure. In the computation tree, the ego vehicle is located in the first layer, and the second layer is the surrounding vehicles. Implicit interaction refers to the interaction relationship between the surrounding vehicles that interact with the ego vehicle, and the implicit association that may be generated to the ego vehicle through the conduction between the surrounding vehicles. In the computation tree, it is located in the second and third layers, representing the association between the surrounding vehicles, and the implicit interaction is transmitted to the ego vehicle with the help of the surrounding vehicles.

[0097] The Optimal Transport (OT) algorithm is an optimization algorithm that seeks to find the minimum distance between two probability distributions. The core idea of ​​the OT algorithm is to transform one probability distribution into another probability distribution so that the distance between the transformed distribution and the target distribution is minimized. This distance is usually calculated using the Wasserstein distance, also known as the Earth Mover's Distance (EMD), which is a manifold-based distance metric that can measure the difference between two probability distributions.

[0098] The optimal transmission algorithm is used to compare the above graph structure calculation tree to obtain the comparison distance of each static scene frame in the two compared dynamic scenes, that is, the optimal transmission distance, which is also the optimal transmission distance corresponding to each scene in the second sample data of the driving scene. The specific calculation steps of the optimal transmission distance are as follows:

[0099]

[0100] sC ij =D(u i ,v j )

[0101] g 1 n =1,g T 1 n =1

[0102] Where g is the transport flow that satisfies the constraints, C is the distance matrix between the corresponding nodes of the two compared trees or subtrees, and OT D is the optimal transmission distance under the distance matrix D, u is the root node of the first comparison subtree, v is the root node of the second comparison subtree, Ψ(u) is the subtree with u as the root node, Ψ(v) is the subtree with v as the root node, is a positive real n*n matrix, R n×n is a real n*n matrix, n is the maximum number of nodes in the two compared subtrees, st(subject to) is the condition of the optimization problem, C ij =D(u i ,v j ) is the distance matrix of the two compared subtree nodes paired with each other, u i is the i-th node of the subtree with u as the root node, v j is the jth node of the subtree with v as the root node, g 1 n =1 means the sum of the forward transmission is 1, g T 1 n =1 means the sum of reverse reception is 1.

[0103]

[0104] in, Represents two subtrees for comparison, Ψ(u) is the subtree with u as the root node, Ψ(v) is the subtree with u as the root node, u is the root node of the first comparison subtree, v is the root node of the second comparison subtree, Ψ(0) is a blank node with no content and edges, Ψ(0) n refers to a node set with n blank nodes, and |Ψ(u)|,|Ψ(v)| refer to the number of nodes in each layer.

[0105] Finally, the distance between the two trees is recursively calculated, from the subtree level to the root node, and the distance matrix is ​​updated hierarchically:

[0106]

[0107] Among them, f u ,f v Refers to the characteristics of nodes u,v related to the comparison standard, w l is the weight of the level, is the distance between nodes u and v when the V2V computation tree is summarized to the lth layer. is the u node at level l of the V2V computation tree, is the v node at level l of the V2N computation tree, OT D is the optimal transmission distance under the aforementioned round D matrix, Ψ V2V (u),Ψ V2V (v) is the set of V2V subtree nodes with u,v as root nodes, Ψ V2N (u),Ψ V2N (v) is the set of V2N subtree nodes with u and v as root nodes. The comparison distance of the final scene graph is summarized as:

[0108]

[0109] in, is the graph structure of the ego car u,v, l V2V ,l V2N are the weights of different types of computational trees, is the optimal transmission distance of the corresponding tree, L is the number of expansion layers of the calculation tree, OT D is the optimal transmission distance under the D matrix, Ψ V2V (ego u ) is the set of V2V computation tree nodes with the ego vehicle u as the root node, Ψ V2V (ego v ) is the set of V2V computation tree nodes with the ego vehicle v as the root node, Ψ V2N (ego u ) is the set of V2N computation tree nodes with the ego vehicle u as the root node, Ψ V2N (ego v ) is a set of V2N computation tree nodes with the ego vehicle v as the root node.

[0110] For dynamic scenes, the dynamic time warping distance of each scene in the second sample data of the driving scene is calculated based on the dynamic time warping algorithm. After obtaining the optimal transmission distance between static scene frames, the dynamic time warping method is used to solve the problem of comparing scenes of different time lengths. Figure 5The dynamic scene sequence shown is composed of a series of static scene graphs arranged in time sequence. The algorithm corresponds a time point in the sequence to multiple continuous time points in another sequence, and calculates the distance matrix between each scene frame of the two graph sequences. Further, a shortest path from the upper left corner to the lower right corner is found in the matrix graph, and the sum of the elements along the path is used as the final comparison distance of the two dynamic scenes. The final comparison distance is the dynamic time warping distance of each scene in the second sample data of the driving scene. The final comparison distance can be expressed as:

[0111]

[0112] Where D(x,y) is the distance between the xth frame and the yth frame of the two contrasting scene clips, L min (x,y) is the shortest distance in the scene summed up to position (x,y).

[0113] Based on the scene distance standard, the DBSCAN algorithm is used for clustering, and the marked scenes constitute the marginal risk scene library

[0114] Based on the standard of scene distance, clustering is performed using the DBSCAN algorithm to obtain the annotation results of each scene in the second sample data of the driving scene. The scene annotation results include: common scenes and extreme edge scenes. Since there are many common scenes and they are relatively concentrated, the areas with denser clusters will be relatively common safety scenes, while the sparser areas will be more valuable extreme edge scenes. The standard of scene distance refers to the optimal transmission distance of static scene frames + DTW distance of dynamic scenes.

[0115] A scene database can be formed through classification and labeling of scenes, which can be used in actual autonomous driving research. For example, it can be used as a scene database required for autonomous driving training and testing, provide efficient retrieval and extraction of extreme edge scene data, and automatically classify and label new scene data in combination with interaction types to form new sample data in the scene database.

[0116] To verify the effectiveness of the scheme, we further compare the extreme risk scenario set annotated by the graph structure metric with the scenario set annotated by the classic collision time (TTC) metric. For the TTC metric, we set the threshold to 1.0s to select extreme risk scenarios, use virtual mapping technology to convert two-dimensional interactions into one-dimensional vehicle following interactions, and then perform calculations to reflect the risk. The feature f used for the graph structure metric is calculated as:

[0117]

[0118] where v ego is the speed of the vehicle, v is the speed of other vehicles, and Δr is the relative distance between the vehicles.

[0119] The annotated extreme scenes only account for a small part of the dataset. Figure 6 It can be seen that most scenes are clustered into the central group, and a few scenes are filtered out due to the large scene comparison distance. After comparison with other indicators and manual inspection, the selected samples are exactly edge risk scenes. Comparison results with other indicators, such as TTC (Time-To-Collision), THW (Time-Headway) and other typical numerical calculation indicator results. Manual inspection refers to human experts checking whether the labeled scene samples have relative risk and extremeness. Through the distribution of risk scenes, we summarize the edge scene samples labeled by TTC and graph structure metrics into a complete risk set and analyze their specific proportions. In the end, the overlap ratio of the two annotations exceeded 70%, indicating that the proposed graph structure metric and calculation features can effectively index edge scenes.

[0120] In addition, the graph structure metric proposed in this patent can expand the edge risk scenarios that cannot be accurately extracted by standards such as TTC, such as Figure 7 As shown in the figure, vehicle 50 tries to insert into the gap between vehicles 42 and 49, but it is obvious that vehicle 49 is more likely to occupy the right of way, and the driving behavior of vehicle 50 is too aggressive. Due to the uncertain conflict location, TTC cannot keenly capture potential risks, and graph metrics that better characterize vehicle interaction relationships can better expand similar edge scenarios.

[0121] In summary, compared with the prior art, the present disclosure has the following advantages:

[0122] The technical solution disclosed in the present invention provides a new driving scene classification and annotation method, defines basic driving scene types, classifies driving scenes based on vehicle interactions in combination with spatiotemporal tree slices, converts driving scenes into graph structures, and then converts them into vehicle-to-vehicle interaction calculation trees and vehicle-to-road interaction calculation trees for detailed comparison and annotation. Therefore, the technical solution disclosed in the present invention constructs a clear scene classification hierarchy by defining basic driving scene types in combination with spatiotemporal tree slices; adopts a driving scene graph structure metric standard, provides an efficient driving scene comparison method, and thus achieves accurate annotation of driving scenes.

[0123] Based on the above disclosed content, the present disclosure also provides an electronic device. Fig. 9 As shown, the electronic device of an embodiment of the present disclosure includes at least one electrically connected processor and at least one storage medium, wherein the storage medium is electrically connected to the processor, wherein the storage medium stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method as described above.

[0124] Based on the same inventive concept, the present disclosure further provides a storage medium, which stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method as described above.

[0125] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. An indirect connection method can be applicable to the embodiments of the present disclosure as long as the purpose of the present disclosure is achieved.

[0126] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A driving scene classification and labeling method based on vehicle interaction, characterized in that: include: Determine the interaction type of vehicles in the driving scene based on the interactive relationship between the movement directions of vehicles in the driving scene; Acquire first sample data of the driving scene; Slicing and classifying the first sample data of the driving scene based on the scene spatiotemporal tree and the interaction type of the vehicle to obtain second sample data of the driving scene; Based on a scene metric standard combining a graph structure computation tree and dynamic time warping, a labeling result of each scene in the second sample data of the driving scene is obtained.

2. The driving scene classification and labeling method based on vehicle interaction according to claim 1 is characterized in that: The types of vehicle interactions in the driving scenario include: dynamic following flow, dynamic oncoming flow, dynamic conflict line, static conflict line, and static conflict line.

3. The driving scene classification and labeling method based on vehicle interaction according to claim 1, characterized in that: The step of slicing and classifying the first sample data of the driving scene based on the scene spatiotemporal tree and the interaction type of the vehicle to obtain the second sample data of the driving scene includes: Specifying the ego vehicle and the spatial range of the ego vehicle in the first sample data of the driving scene; Extracting surrounding vehicle data within the spatial range of the vehicle from the first sample data of the driving scene; If the vehicle corresponding to the surrounding vehicle data has no interactive relationship with the vehicle itself, filtering the surrounding vehicle data; Determine the interaction state between the vehicle corresponding to the surrounding vehicle data and the vehicle based on a time window method; The surrounding vehicle data having the same interaction type and interaction area as the own vehicle are merged to obtain second driving scene sample data.

4. The driving scene classification and labeling method based on vehicle interaction according to any one of claims 1 to 3, characterized in that: The scene metric standard based on the graph structure calculation tree combined with dynamic time warping is used to obtain the labeling results of each scene in the second sample data of the driving scene, including: Calculating the optimal transmission distance corresponding to each scene in the second sample data of the driving scene based on a graph structure calculation tree comparison algorithm; Based on a dynamic time warping algorithm, calculating a dynamic time warping distance of each scene in the second sample data of the driving scene; Based on the optimal transmission distance and the dynamic time warping distance, a cluster analysis is performed on the second sample data of the driving scene to obtain a labeling result of each scene in the second sample data of the driving scene.

5. A driving scene classification and labeling device based on vehicle interaction, characterized in that: include: A determination module, for determining the interaction type of vehicles in the driving scene based on the interaction relationship of the movement flow of the vehicles in the driving scene; An acquisition module, used to acquire first sample data of a driving scene; A slicing module, used for slicing and classifying the first sample data of the driving scene based on the scene spatiotemporal tree and the interaction type of the vehicle to obtain second sample data of the driving scene; The labeling module is used to obtain the labeling results of each scene in the second sample data of the driving scene based on the scene metric standard combining the graph structure calculation tree and dynamic time warping.

6. The driving scene classification and labeling device based on vehicle interaction according to claim 5, characterized in that: The types of vehicle interactions in the driving scenario include: dynamic following flow, dynamic oncoming flow, dynamic conflict line, static conflict line, and static conflict line.

7. The driving scene classification and labeling device based on vehicle interaction according to claim 6, characterized in that: The slicing module is also used for: Specifying the ego vehicle and the spatial range of the ego vehicle in the first sample data of the driving scene; Extracting surrounding vehicle data within the spatial range of the vehicle from the first sample data of the driving scene; If the vehicle corresponding to the surrounding vehicle data has no interactive relationship with the vehicle itself, filtering the surrounding vehicle data; Determine the interaction state between the vehicle corresponding to the surrounding vehicle data and the vehicle based on a time window method; The surrounding vehicle data having the same interaction type and interaction area as the own vehicle are merged to obtain second driving scene sample data.

8. The driving scene classification and labeling device based on vehicle interaction according to any one of claims 5 to 7, characterized in that: The annotation module is also used for: Calculating the optimal transmission distance corresponding to each scene in the second sample data of the driving scene based on a graph structure calculation tree comparison algorithm; Based on a dynamic time warping algorithm, calculating a dynamic time warping distance of each scene in the second sample data of the driving scene; Based on the optimal transmission distance and the dynamic time warping distance, a cluster analysis is performed on the second sample data of the driving scene to obtain a labeling result of each scene in the second sample data of the driving scene.

9. An electronic device, characterized in that: include: Processor and memory; The processor is coupled to a memory; The processor is used to read and execute the program or instruction stored in the memory, so that the device executes the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Citation Information

Cited By

  • Track planning method and device and storage medium

    CN120293167A

  • Trajectory planning method, device and storage medium

    CN120293167B

  • Multi-mode man-machine interaction control system for remote driving of mining locomotive

    CN120735789A