Driving behavior modeling method based on spatio-temporal information fusion

Through the driving behavior modeling method of fusion of space-time information, a driving behavior model is constructed, which solves the problem of insufficient understanding of driving scenarios and improves the trajectory control accuracy and simulation test accuracy of autonomous driving vehicles.

CN118940634APending Publication Date: 2024-11-12JILIN UNIVERSITY
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411059841.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing driving behavior model lacks understanding of driving scenarios, which makes it difficult to narrow the gap between virtual simulation testing and real driving environments, affecting the development and testing of autonomous driving technology.

Method used

Using the driving behavior modeling method based on space-time information fusion, the driving behavior model is constructed through the spatial information encoding network, the timing information encoding network, the feature fusion network and the feature decoding network, and the driving behavior model is constructed, and the spatial sample information and timing sample information in the training sample set are obtained, and the training is carried out. Model to determine the future trajectory sequence of the target main vehicle.

Benefits of technology

It improves the driving behavior model's understanding of driving scenarios, improves the trajectory control accuracy, accurately reproduces human driving behavior, and narrows the gap between virtual simulation and real driving environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940634A_ABST
    Figure CN118940634A_ABST
Patent Text Reader

Abstract

The invention discloses a driving behavior modeling method based on spatio-temporal information fusion, and relates to the field of driving behavior simulation, and the method comprises the steps: constructing a driving behavior model which comprises a spatial information coding network, a time sequence information coding network, a feature fusion network and a feature decoding network; the feature fusion network is connected with the spatial information coding network and the time sequence information coding network, and the feature decoding network is connected with the feature fusion network; according to the space information and the time sequence information of the target main vehicle, based on the trained driving behavior model, determining a future trajectory sequence of the target main vehicle at a future moment; the target host vehicle is controlled to travel according to a future trajectory sequence at a future time. According to the method, the space information and the time sequence information are coded and fused, so that the understanding ability of the driving behavior model to the driving scene is improved, and the trajectory control precision of the target main vehicle is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of driving behavior simulation, and in particular to a driving behavior modeling method based on spatiotemporal information fusion. Background Art

[0002] With the advantages of high efficiency, safety, repeatability and low cost, virtual simulation technology has become one of the key technologies to promote the development of autonomous driving safety testing technology. As the most important traffic participants, human drivers' driving behavior directly affects the behavioral decisions of autonomous vehicles. Therefore, the driving behavior model is the key to building a high-fidelity virtual simulation test environment. The lack of authenticity and intelligence of the driving behavior model makes it difficult to further narrow the gap between the virtual simulation test environment and the real driving environment. This has become the main bottleneck restricting the further application of virtual simulation technology in autonomous driving development and testing. There are rich and multimodal data in the driving scene, including multi-source information such as space and time series. These data provide more comprehensive information for the driving behavior model and improve the model's ability to understand the driving scene. However, there are huge differences between different modal data. How to effectively encode and fuse features has become the key to fully understand the driving scene and improve the performance of the driving behavior model. Summary of the invention

[0003] The purpose of this application is to provide a driving behavior modeling method based on spatiotemporal information fusion, which can improve the driving behavior model's ability to understand driving scenarios and thus improve the vehicle's trajectory control accuracy.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a driving behavior modeling method based on spatiotemporal information fusion, comprising:

[0006] Acquire a training sample set; each training sample in the training sample set includes spatial sample information, temporal sample information and a real trajectory sequence of the sample main vehicle;

[0007] Constructing a driving behavior model; the driving behavior model includes a spatial information encoding network, a temporal information encoding network, a feature fusion network and a feature decoding network; the feature fusion network is connected to the spatial information encoding network and the temporal information encoding network respectively, and the feature decoding network is connected to the feature fusion network;

[0008] Training the driving behavior model according to the training sample set to obtain a trained driving behavior model;

[0009] According to the spatial information and temporal information of the target vehicle, based on the trained driving behavior model, a future trajectory sequence of the target vehicle at a future moment is determined; wherein the target vehicle is controlled to travel according to the future trajectory sequence at the future moment; the spatial information includes multiple elements, and the multiple elements are lane lines, historical trajectories of the target vehicle and historical trajectories of its background vehicles; the temporal information includes the historical trajectory of the target vehicle and the historical trajectories of its background vehicles.

[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0011] The present application provides a driving behavior modeling method based on spatiotemporal information fusion. The driving behavior model includes a spatial information encoding network, a temporal information encoding network, a feature fusion network, and a feature decoding network. By encoding and fusing the spatial information and temporal information in the driving scene, the driving behavior model's ability to understand the driving scene is improved. The future trajectory sequence determined by the driving behavior model is more accurate, so that the target vehicle can be better guided to drive, so as to achieve the purpose of accurately reproducing human driving behavior, thereby improving the trajectory control accuracy of the target vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 This is an application environment diagram of a driving behavior modeling method based on spatiotemporal information fusion in one embodiment of the present application;

[0014] Figure 2 A flowchart of a driving behavior modeling method based on spatiotemporal information fusion provided in one embodiment of the present application;

[0015] Figure 3 A flowchart of a driving behavior modeling method based on spatiotemporal information fusion provided in another embodiment of the present application;

[0016] Figure 4 is the cumulative distribution curve of the average trajectory error;

[0017] Figure 5 It is a curve diagram of the average trajectory error changing with the output step length;

[0018] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] The driving behavior modeling method based on spatiotemporal information fusion provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the spatial information and timing information in the driving scene to the server 104. After receiving the spatial information and timing information in the driving scene, the server 104 constructs a driving behavior model based on the spatial information and timing information in the driving scene, and predicts the future trajectory sequence of the main vehicle based on the driving behavior model. The server 104 can feedback the obtained future trajectory sequence of the main vehicle to the terminal 102. In addition, in some embodiments, the driving behavior modeling method based on spatiotemporal information fusion can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly perform driving behavior modeling for the spatial information and timing information in the driving scene, or the server 104 can obtain the spatial information and timing information in the driving scene from the data storage system and perform driving behavior modeling.

[0022] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0023] In an exemplary embodiment, Figure 2 As shown, a driving behavior modeling method based on spatiotemporal information fusion is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 204.

[0024] Step 201, obtaining a training sample set. Each training sample in the training sample set includes spatial sample information, temporal sample information and a real trajectory sequence of the sample host vehicle.

[0025] Step 202: construct a driving behavior model.

[0026] In the present application, the driving behavior model includes a spatial information encoding network, a temporal information encoding network, a feature fusion network and a feature decoding network. The feature fusion network is connected to the spatial information encoding network and the temporal information encoding network respectively, and the feature decoding network is connected to the feature fusion network.

[0027] Step 203: training the driving behavior model according to the training sample set to obtain a trained driving behavior model.

[0028] Step 204, determining a future trajectory sequence of the target vehicle at a future moment according to the spatial information and temporal information of the target vehicle and based on the trained driving behavior model; wherein the target vehicle is controlled to travel according to the future trajectory sequence at the future moment.

[0029] The spatial information includes multiple elements, which are lane lines, historical trajectories of the target vehicle and historical trajectories of its background vehicles. The temporal information includes historical trajectories of the target vehicle and historical trajectories of its background vehicles. The historical trajectories include the position and speed at each moment.

[0030] In a specific example, the sample main vehicle and the target main vehicle can be real vehicles or virtual vehicles in a virtual simulation platform. The sample main vehicle can be an autonomous driving vehicle or a manually driven vehicle. The target main vehicle is an autonomous driving vehicle.

[0031] When the sample main vehicle is a real vehicle, the driving behavior model is trained according to the spatial information and timing information of the vehicle in the real driving scene; when the sample main vehicle is a virtual vehicle, the driving behavior model is trained according to the spatial information and timing information of the vehicle in the virtual simulation test environment.

[0032] When the target vehicle is a real vehicle, the driving behavior model provided by this application is applied in a real driving scene. According to the spatial information and timing information of the target vehicle in the real driving scene, the driving behavior model provided by this application is used to determine the future trajectory sequence of the target vehicle, and then automatically control the driving trajectory of the target vehicle, thereby improving the driving accuracy and safety of the autonomous driving vehicle in the real driving scene; when the target vehicle is a virtual vehicle, the trained driving behavior model is connected to the virtual simulation platform to determine the future trajectory sequence of the target vehicle as a virtual vehicle at the future moment. The virtual simulation platform controls the target vehicle to drive according to the future trajectory sequence at the future moment.

[0033] Specifically, the virtual simulation platform can be any simulation software, such as Tencent TAD Sim autonomous driving simulation test platform, and the corresponding simulation software can be selected according to actual needs. A virtual simulation test environment for autonomous driving vehicles is constructed in Tencent TAD Sim autonomous driving simulation test platform, and lane lines, target main vehicles and background vehicles are set in the virtual simulation test environment. According to the spatial information and timing information of the target main vehicle in the virtual simulation test environment, the driving behavior model provided by this application is used to determine the future trajectory sequence of the target main vehicle, and then the driving trajectory of the target main vehicle is automatically controlled in the virtual simulation test environment, and the operating state of the autonomous driving vehicle is virtually simulated, providing a safety foundation for the subsequent autonomous driving vehicles to be put into real driving scenarios.

[0034] In an exemplary embodiment, step 204 includes steps (1) to (4).

[0035] Step (1), encoding the spatial information through the spatial information encoding network to obtain spatial features. The spatial information encoding network is constructed based on a multi-layer perceptron, a maximum pooling aggregation operation and a self-attention mechanism.

[0036] Spatial information such as road structure and the spatial situation of background vehicles is an important factor affecting driving behavior. Therefore, encoding spatial information is one of the keys to improving the accuracy of driving behavior models. This application divides spatial information into two categories: map elements and trajectory elements. The map elements are lane lines, and the trajectory elements include the historical trajectory of the main vehicle and the historical trajectory of background vehicles. Each lane line, the historical trajectory of the main vehicle, and the historical trajectory of each background vehicle in the driving scene are independently regarded as an element in the spatial information. Each element in the spatial information is vectorized separately and uniformly described as a broken line subgraph consisting of 9 node vectors connected end to end.

[0037] Furthermore, step (1) includes steps (11) to (16).

[0038] Step (11), determining multiple node vectors for each element, wherein each node vector includes a starting position, an end position, an element attribute and an element sequence number.

[0039] Specifically, multiple key points are selected along the vehicle driving direction on the lane line to obtain multiple node vectors of the lane line. According to the historical trajectory of the target vehicle, multiple key points are selected along the time dimension to determine multiple node vectors of the target vehicle. According to the historical trajectory of the background vehicle, multiple key points are selected along the time dimension to determine multiple node vectors of the background vehicle.

[0040] When describing map elements, 10 key points are uniformly selected in sequence along the vehicle's driving direction on each lane line. When describing trajectory elements, historical trajectories within the past 5 seconds are selected, with a sampling interval of 0.5 seconds, and 10 key points are selected in sequence along the time dimension. Adjacent key points are connected in sequence to form a node vector.

[0041] The node vector of the jth element is expressed as:

[0042] v i,j =[x start,i,j ,y start,i,j ,x end,i,j ,y end,i,j ,a j ,j];

[0043] in, The i-th node vector in the line subgraph representing the j-th element, is a set of real numbers, x start,i,j The x-axis coordinate of the starting point of the i-th node vector in the line subgraph of the j-th element, start,i,j The y-axis coordinate of the starting point of the i-th node vector in the line subgraph of the j-th element, x end,i,j The x-axis coordinate of the end point of the i-th node vector in the line subgraph representing the j-th element, and the y-axis coordinate of the end point of the i-th node vector in the line subgraph representing the j-th element. end,i,j Represents the y-axis coordinate of the end point of the i-th node vector in the line subgraph of the j-th element, Indicates the element attribute of the jth element, which includes five categories: main vehicle trajectory, background vehicle trajectory, lane line solid line, lane line dashed line and virtual vehicle trajectory. The five types are encoded and represented in one-hot form, for example, the main vehicle trajectory is represented by 0001, the background vehicle trajectory is represented by 0010, the lane line solid line is represented by 0100, the lane line dashed line is represented by 1000, and the virtual vehicle trajectory is represented by 0000.

[0044] Step (12), for any element, connect the multiple node vectors of the element in sequence to obtain the line subgraph of the element. The line subgraph of the jth element is represented as is composed of {v 1,j ,v 2,j ,...,v 9,j} is a one-way multi-line graph composed of.

[0045] Step (13), based on the multi-layer perceptron and the maximum pooling aggregation operation, the features of the line subgraph of each element are extracted respectively to obtain the feature vector of the line subgraph of each element.

[0046] In order to extract the features of the line subgraph, each line subgraph needs to go through the forward propagation process of the three-layer subgraph feature extraction network, and then the output result of the last layer of subgraph feature extraction network is subjected to the maximum pooling operation, and the spatial elements represented by the line subgraph are represented by a one-dimensional line subgraph feature vector.

[0047] Specifically, the following formula is used to determine the line subgraph feature vector of the jth element:

[0048]

[0049]

[0050] in, The i-th node vector in the line subgraph of the j-th element in the l-th layer subgraph feature extraction network, The i-th node vector in the line subgraph of the j-th element in the l-1-th layer subgraph feature extraction network, is the total node vectors in the line subgraph of the jth element in the lth layer subgraph feature extraction network, l = 1, 2, 3, the node vector in the line subgraph of the jth element in the 0th layer subgraph feature extraction network is the node vector in the line subgraph of the jth element, that is, is the line subgraph feature vector of the jth element, g enc (·) represents the node encoding network, which is implemented by a multi-layer perceptron with the same structure but different parameters in each layer of the subgraph feature extraction network. Indicates the maximum pooling aggregation of all adjacent node vectors. Represents a concatenation operation.

[0051] Step (14), concatenate the line subgraph feature vectors of each element to obtain a spatial feature matrix.

[0052] In the process of obtaining the line subgraph features, only the spatial information of the elements themselves is considered, and the global interaction relationship between the elements is not considered. It is necessary to further build a global graph network to capture the global interaction features. Each line subgraph feature is regarded as a node in the global graph network. In order to fully consider the interaction relationship between each pair of nodes, the global graph network is regarded as a network in which all nodes are interconnected. All the line subgraph feature vectors are spliced ​​together to form a spatial feature matrix

[0053] Each row of the spatial feature matrix represents the line subgraph feature vector of each element. When the number of elements is less than 32, all zero rows are used for padding. It should be noted that the first nine rows of the spatial feature matrix have a fixed order. The first row of the spatial feature matrix is ​​the line subgraph feature vector of the historical trajectory of the main vehicle, and the second to ninth rows are respectively from the line subgraph feature vectors of the historical trajectories of the eight background vehicles around the main vehicle. The eight background vehicles are the vehicles in front of the left of the main vehicle, in front of the right of the main vehicle, on the left side of the main vehicle, on the right side of the main vehicle, behind the left of the main vehicle, behind the main vehicle, and behind the right of the main vehicle. The remaining rows of the spatial feature matrix are the line subgraph feature vectors from the map elements.

[0054] Step (15), using a self-attention mechanism to extract the global interaction features of the spatial feature matrix to obtain a spatial interaction feature matrix.

[0055] The self-attention mechanism enables each element to interact with other elements and adjusts its representation based on the correlation between them. The following formula is used to determine the spatial interaction feature matrix:

[0056]

[0057] P Q =PW Q ;

[0058] P K =PW K ;

[0059] P V =PW V ;

[0060] in, is the spatial interaction feature matrix, sotfmax(·) is the activation function, P Q is the query linear projection of the spatial feature matrix, P K is the key linear projection of the spatial feature matrix, P V is the linear projection of the spatial feature matrix, W Q is the query linear transformation matrix learned during training, W K is the key linear transformation matrix learned during training, WV is the linear transformation matrix of the values ​​learned during training. The superscript T represents the transpose operation.

[0061] Step (16), determining the spatial features based on the spatial interaction feature matrix.

[0062] Specifically, the spatial interaction feature matrix The first nine lines of are taken as the output of the entire spatial information encoding process and recorded as spatial features Each row represents the spatial information encoding results of the main vehicle and the eight surrounding background vehicles.

[0063] Step (2), encoding the time series information through the time series information encoding network to obtain time series features. The time series information encoding network is constructed based on a long short-term memory network (Long Short-Term Memory, LSTM) and a multi-layer perceptron.

[0064] Since driving behavior has obvious time sequence, using time sequence information such as historical trajectory can better understand the movement trend and behavior of each vehicle in the driving scene. When encoding time sequence information, this application adopts different strategies for the main vehicle and background vehicles to provide features from different angles.

[0065] On the one hand, the short-term driving intention of the main vehicle is extracted from the historical trajectory of the main vehicle in the world coordinate system, and on the other hand, the dynamic impact of the movement trend of the background vehicle on the driving scene is extracted from the historical trajectory of the background vehicle in the main vehicle coordinate system. Although the input features of the main vehicle and the background vehicle are the same, they are described based on different coordinate systems. Therefore, two different LSTMs are used to encode the temporal features of the main vehicle and the background vehicle respectively.

[0066] Furthermore, step (2) includes steps (21) to (24).

[0067] Step (21), based on the historical trajectory of the target vehicle, a first long short-term memory network is used to extract the timing characteristics of the target vehicle to obtain the timing characteristics of the vehicle.

[0068] Specifically, the position and speed history trajectory of the target vehicle in the global coordinate system in the past 5 seconds is input into the first long short-term memory network to extract the temporal characteristics of the target vehicle. The sampling interval of the history trajectory of the target vehicle is 0.5 seconds.

[0069] Step (22), based on the historical trajectory of the background vehicle, a second long short-term memory network is used to extract the time series features of the background vehicle to obtain the time series features of the background vehicle.

[0070] Specifically, the position and speed history trajectories of the background vehicles in the main vehicle coordinate system in the past 5 seconds are input into the second long short-term memory network to extract the temporal features of the background vehicles. The sampling interval of the history trajectories of the background vehicles is 0.5 seconds.

[0071] Step (23), splicing the main vehicle time series feature with the background vehicle time series feature to obtain a spliced ​​time series feature.

[0072] Step (24), using a multi-layer perceptron to perform nonlinear dimensionality increase on the spliced ​​time series features to obtain time series features.

[0073] In order to further improve the expressive power of time series features, this application uses a multi-layer perceptron to perform nonlinear dimensionality increase on the spliced ​​time series features to obtain the output of the time series information encoding, which is recorded as the time series feature Among them, each row of the time series feature represents the time series information encoding results of the main vehicle and the surrounding 8 background vehicles in turn.

[0074] The first LSTM network has an input feature shape of (Batch size, 10, 4), an output feature shape of (Batch size, 1, 8), and a structure parameter of LSTM (4, 8, 1). The second LSTM network has an input feature shape of (Batchsize, 10, 4), an output feature shape of (Batchsize, 1, 8), and a structure parameter of LSTM (4, 8, 1). The multi-layer perceptron has an input feature shape of (Batchsize, 9, 8), an output feature shape of (Batchsize, 9, 32), and a structure parameter of Linear (8, 16) + ReLu + Linear (16, 32). Among them, Batchsize is the batch size, LSTM(4,8,1) represents a long short-term memory network layer with an input dimension of 4, 8 hidden units, and 1 layer in the neural network, Linear(8,16) represents a fully connected layer with an input dimension of 4 and an output dimension of 8 in the neural network, Linear(16,32) represents a fully connected layer with an input dimension of 16 and an output dimension of 32 in the neural network, and ReLu is the linear rectification function in the neural network activation function.

[0075] Step (3), fusing the spatial features and the temporal features through the feature fusion network to obtain spatiotemporal fusion features. The feature fusion network is constructed based on a multi-head self-attention mechanism.

[0076] Through steps (1) and (2), the spatial information and temporal information in the driving scene can be encoded into Y Spatial and Y Temporal However, steps (1) and (2) are performed independently, and the encoded Y Spatial and YTemporall There is no effective association between them. The multi-head self-attention mechanism can focus on different feature sources by using multiple attention heads at the same time, which has better diversity, parallelism and generalization ability. Therefore, this application adopts the multi-head self-attention mechanism to realize the fusion of spatiotemporal features.

[0077] First, the spatial feature Y Spatial and the time series feature Y Temporal Stitched together to form the spatiotemporal feature X ST :X ST =Concat(Y Spatial ,Y Temporal ), and then each head in the multi-head self-attention mechanism is used to fuse the spatiotemporal features respectively. Finally, the results of each head are concatenated and multiplied by the weight matrix to obtain the spatiotemporal fusion features.

[0078] The calculation process of each head in the multi-head self-attention mechanism is as follows:

[0079]

[0080] Q z =X ST W z Q ;

[0081] K z =X ST W z K ;

[0082] V z =X ST W z V ;

[0083] Among them, head z represents the self-attention fusion result of the z-th head, z = 1, 2, ..., h, h is the number of heads in the multi-head self-attention mechanism, set to 4, Q z is the spatiotemporal feature X ST The linear projection of the query in the zth head, K z is the spatiotemporal feature X ST The linear projection of the key in the zth head, V z is the spatiotemporal feature X ST Linear projection of the values ​​in the zth head, is the query weight matrix learned in the z-th head, is the key weight matrix learned in the z-th head, is the value weight matrix learned in the z-th head.

[0084] The results of each head are concatenated and multiplied by another weight matrix to get the output of the multi-head self-attention mechanism:

[0085] Y=Concat(head1,head2,head3,head4)W O ;

[0086] Among them, Y is the result of spatiotemporal feature fusion, W O Represents the output weight matrix that needs to be learned in the multi-head self-attention mechanism. The first row of Y is used as the spatiotemporal fusion feature and is recorded as

[0087] Step (4), decoding the spatiotemporal fusion features through the feature decoding network to determine the future trajectory sequence of the target vehicle at a future time. The feature decoding network is constructed based on nonlinear dimensionality reduction operation and long short-term memory network.

[0088] Furthermore, step (4) includes steps (41) to (44).

[0089] Step (41), performing nonlinear dimensionality reduction on the spatiotemporal fusion feature to obtain a first dimensionality reduction feature.

[0090] In order to combine the spatiotemporal features Y ST Decoding into the required future trajectory sequence of the main vehicle first requires the first nonlinear dimensionality reduction to transform the spatiotemporal fusion feature Y ST Initial compression and refinement to reduce the complexity of the data.

[0091] Step (42), reconstructing the time series of the first dimensionality reduction feature, converting the first dimensionality reduction feature into a two-dimensional matrix with a time series relationship, and obtaining sequence data.

[0092] Specifically, the first dimension reduction feature is a one-dimensional vector, which is converted into a two-dimensional matrix with a time series relationship through a matrix shape.

[0093] Step (43), using a third long short-term memory network to perform temporal decoding on the sequence data to obtain decoding features.

[0094] This application uses the powerful modeling ability of LSTM for sequence data to perform temporal decoding of the reintegrated information.

[0095] Step (44) performs nonlinear dimensionality reduction on the decoded features, maps the decoded features into the target space, and obtains a future trajectory sequence of the target vehicle at a future time.

[0096] Specifically, the features after time series decoding by LSTM are subjected to a second nonlinear dimensionality reduction and mapped to a target space with a time series length of 10 and a dimension of 2 to obtain the future trajectory sequence of the target vehicle at the future moment.

[0097] The input feature shape of the first nonlinear dimensionality reduction process is (Batch size, 1, 128), the output feature shape is (Batch size, 1, 120), and the structural parameters are Linear (128, 120) + ReLu. The input feature shape of the time series reconstruction process is (Batch size, 1, 120), and the output feature shape is (Batch size, 10, 12). The input feature shape of the time series decoding process is (Batch size, 10, 12), the output feature shape is (Batch size, 10, 16), and the structural parameters are LSTM (12, 16, 1). The input feature shape of the second nonlinear dimensionality reduction process is (Batch size, 10, 16), the output feature shape is (Batch size, 10, 2), and the structural parameters are Linear (16, 2) + ReLu.

[0098] In another exemplary embodiment, Figure 3 As shown, the driving behavior model constructed in the present application includes a coding layer, a fusion layer and a decoding layer.

[0099] The encoding layer encodes and extracts features from the lane lines and historical trajectories, and uses the spatial information encoding network and the temporal information encoding network to obtain spatial features and temporal features respectively.

[0100] The fusion layer uses a multi-head self-attention mechanism to fuse spatial features and temporal features, and obtains spatiotemporal fusion features that represent the understanding of the current scene.

[0101] The decoder decodes based on the obtained spatiotemporal fusion features to generate future trajectory sequences.

[0102] In an exemplary embodiment, the loss function quantitatively describes the difference between the output of the driving behavior model and its underlying truth value, guiding the learning process of the driving behavior model. The driving behavior model uses a dynamic loss function in training. The core idea is to adaptively adjust the weights of FDE and ADE according to the ratio of the final displacement error (FDE) to the average displacement error (Average Displacement Error, ADE) during the training process, so as to avoid focusing too much on one indicator and ignoring the overall performance during the training process. The loss function of the driving behavior model during training is:

[0103] Loss = FDE / (κ+ω)+ADE;

[0104] κ = FDE / ADE;

[0105] Among them, Loss is the loss function value, FDE is the final trajectory error, which indicates the end point deviation of the trajectory sequence output by the driving behavior model, ADE is the average trajectory error, which indicates the overall deviation between the trajectory sequence output by the driving behavior model and the true trajectory sequence, κ is the ratio coefficient, and ω is the dynamic loss coefficient, which is set to 1.39.

[0106] like Figure 4 and Figure 5 As shown in the figure, when the simulation step size is less than 5 steps, the average trajectory error of the driving behavior model does not change significantly with the output step size. At this time, the error accumulation effect has little effect on the driving behavior model, and the performance is relatively stable when performing short-time domain simulation. When the simulation step size is 5, the average trajectory error is only 0.13 meters. This shows that when the error accumulation effect is small, thanks to the support of spatiotemporal information fusion, the driving behavior model has a more comprehensive understanding of the driving scene, thereby achieving higher prediction accuracy. When the simulation step size is greater than 5, the error of the driving behavior model increases significantly with the increase of the output step size. This is mainly because there are certain errors in each simulation step and they accumulate gradually. In addition, with the increase of the simulation step size, the dynamic uncertainty in the driving scene will also increase. These unavoidable factors make the error of the driving behavior model increase significantly in long-time domain simulation. However, when the simulation step size is 10, the average trajectory error of the driving behavior model is controlled at 0.37 meters. In addition, from the error cumulative distribution curve, when the simulation step size is 10 steps, the driving behavior model has a 90% chance of controlling the average trajectory error below 0.56 meters. Moreover, the error cumulative distribution curve is generally to the left, indicating that the higher the proportion accumulated within a smaller error range, the smaller the error produced by the driving behavior model in most cases.

[0107] This application uses the Adam optimizer to adjust the relevant parameters of the neural network. The initial learning rate is 0.0005. After 100 rounds of training, the learning rate decays to 80%. A total of 600 rounds of training are performed to obtain the final driving behavior model.

[0108] This application independently encodes and globally fuses the spatial information and temporal information in the driving scene, uses the spatial information encoding network and the temporal information encoding network to respectively extract the spatial features and temporal features in the driving scene, and fully fuses the spatial features and temporal features based on the multi-head self-attention mechanism, thereby improving the driving behavior model's ability to understand the driving scene, and proposes a new dynamic loss function to improve the performance of the driving behavior model. The future trajectory sequence determined by the driving behavior model is more accurate, so that the target vehicle can be better guided to drive in the driving scene, narrowing the gap between virtual simulation tests and real driving environments, so as to achieve the purpose of accurately reproducing human driving behavior, thereby improving the accuracy of autonomous driving virtual simulation.

[0109] The present application also provides an application scenario, which applies the above-mentioned driving behavior modeling method based on spatiotemporal information fusion. Specifically: The driving behavior modeling method based on spatiotemporal information fusion provided in this embodiment can be applied in a simulation test scenario of autonomous driving. In each simulation step of the simulation test scenario of autonomous driving, the driving behavior model provided by this embodiment replaces the human driver to drive the vehicle in a virtual simulation environment, interacts with the tested autonomous driving car, predicts the trajectory of the main vehicle, and thus improves the accuracy of the driving behavior simulation and narrows the gap between the virtual simulation test environment and the real driving environment.

[0110] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for spatial information and timing information in the driving scene. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a driving behavior modeling method based on spatiotemporal information fusion is implemented.

[0111] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0112] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0113] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0115] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0116] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0117] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A driving behavior modeling method based on spatiotemporal information fusion, characterized in that: The driving behavior modeling method based on spatiotemporal information fusion includes: Acquire a training sample set; each training sample in the training sample set includes spatial sample information, temporal sample information and a real trajectory sequence of the sample main vehicle; Constructing a driving behavior model; the driving behavior model includes a spatial information encoding network, a temporal information encoding network, a feature fusion network and a feature decoding network; the feature fusion network is connected to the spatial information encoding network and the temporal information encoding network respectively, and the feature decoding network is connected to the feature fusion network; Training the driving behavior model according to the training sample set to obtain a trained driving behavior model; According to the spatial information and temporal information of the target vehicle, based on the trained driving behavior model, a future trajectory sequence of the target vehicle at a future moment is determined; wherein the target vehicle is controlled to travel according to the future trajectory sequence at the future moment; the spatial information includes multiple elements, and the multiple elements are lane lines, historical trajectories of the target vehicle and historical trajectories of its background vehicles; the temporal information includes the historical trajectory of the target vehicle and the historical trajectories of its background vehicles.

2. The driving behavior modeling method based on spatiotemporal information fusion according to claim 1, characterized in that: According to the spatial information and temporal information of the target vehicle, based on the trained driving behavior model, the future trajectory sequence of the target vehicle at a future moment is determined, specifically including: The spatial information is encoded by the spatial information encoding network to obtain spatial features; the spatial information encoding network is constructed based on a multi-layer perceptron, a maximum pooling aggregation operation and a self-attention mechanism; The time series information is encoded by the time series information encoding network to obtain time series features; the time series information encoding network is constructed based on a long short-term memory network and a multi-layer perceptron; The spatial features and the temporal features are fused through the feature fusion network to obtain spatiotemporal fusion features; the feature fusion network is constructed based on a multi-head self-attention mechanism; The spatiotemporal fusion features are decoded by the feature decoding network to determine the future trajectory sequence of the target vehicle at a future moment; the feature decoding network is constructed based on a nonlinear dimensionality reduction operation and a long short-term memory network.

3. The driving behavior modeling method based on spatiotemporal information fusion according to claim 2 is characterized in that: The spatial information is encoded by the spatial information encoding network to obtain spatial features, which specifically includes: Determine multiple node vectors for each element; wherein each node vector includes a starting position, an end position, an element attribute, and an element sequence number; For any element, multiple node vectors of the element are connected in sequence to obtain a line subgraph of the element; Based on the multi-layer perceptron and maximum pooling aggregation operation, the features of the line subgraph of each element are extracted respectively to obtain the feature vector of the line subgraph of each element; The line subgraph feature vectors of each element are concatenated to obtain a spatial feature matrix; A self-attention mechanism is used to extract global interaction features of the spatial feature matrix to obtain a spatial interaction feature matrix; The spatial features are determined according to the spatial interaction feature matrix.

4. The driving behavior modeling method based on spatiotemporal information fusion according to claim 3 is characterized in that: Determine multiple node vectors for each element, including: Selecting a plurality of key points along the vehicle travel direction on the lane line to obtain a plurality of node vectors of the lane line; According to the historical trajectory of the target vehicle, multiple key points are selected along the time dimension to determine multiple node vectors of the target vehicle; According to the historical trajectory of the background vehicle, multiple key points are selected along the time dimension to determine multiple node vectors of the background vehicle.

5. The driving behavior modeling method based on spatiotemporal information fusion according to claim 3 is characterized in that: Use the following formula to determine the line subgraph feature vector of the jth element: in, The i-th node vector in the line subgraph of the j-th element in the l-th layer subgraph feature extraction network, The i-th node vector in the line subgraph of the j-th element in the l-1-th layer subgraph feature extraction network, is the total node vectors in the line subgraph of the jth element in the lth layer subgraph feature extraction network, l = 1, 2, 3, the node vector in the line subgraph of the jth element in the 0th layer subgraph feature extraction network is the node vector in the line subgraph of the jth element, is the line subgraph feature vector of the jth element, g enc (·) represents the node encoding network, which is implemented by a multi-layer perceptron with the same structure but different parameters in each layer of the subgraph feature extraction network. Indicates the maximum pooling aggregation of all adjacent node vectors. Represents a concatenation operation.

6. The driving behavior modeling method based on spatiotemporal information fusion according to claim 2, characterized in that: The time series information is encoded by the time series information encoding network to obtain time series features, which specifically includes: According to the historical trajectory of the target vehicle, a first long short-term memory network is used to extract the time series characteristics of the target vehicle to obtain the time series characteristics of the vehicle; According to the historical trajectory of the background vehicle, a second long short-term memory network is used to extract the time series features of the background vehicle to obtain the time series features of the background vehicle; Splicing the main vehicle time series feature with the background vehicle time series feature to obtain a spliced ​​time series feature; A multi-layer perceptron is used to perform nonlinear dimension upgrading on the spliced ​​time series features to obtain time series features.

7. The driving behavior modeling method based on spatiotemporal information fusion according to claim 2, characterized in that: The spatiotemporal fusion features are decoded by the feature decoding network to determine the future trajectory sequence of the target vehicle at a future time, specifically including: Performing nonlinear dimensionality reduction on the spatiotemporal fusion feature to obtain a first dimensionality reduction feature; Performing time series reconstruction on the first dimensionality reduction feature, converting the first dimensionality reduction feature into a two-dimensional matrix with a time series relationship, and obtaining sequence data; Using a third long short-term memory network to perform temporal decoding on the sequence data to obtain decoding features; The decoded features are subjected to nonlinear dimensionality reduction, and the decoded features are mapped to the target space to obtain a future trajectory sequence of the target vehicle at a future time.

8. The driving behavior modeling method based on spatiotemporal information fusion according to claim 1, characterized in that: The loss function of the driving behavior model during training is: Loss = FDE / (κ+ω)+ADE; κ = FDE / ADE; Among them, Loss is the loss function value, FDE is the final trajectory error, which indicates the end point deviation of the trajectory sequence output by the driving behavior model, ADE is the average trajectory error, which indicates the overall deviation between the trajectory sequence output by the driving behavior model and the true trajectory sequence, κ is the ratio coefficient, and ω is the dynamic loss coefficient.

9. The driving behavior modeling method based on spatiotemporal information fusion according to claim 1, characterized in that: The sample main vehicle and the target main vehicle are both real vehicles.

10. The driving behavior modeling method based on spatiotemporal information fusion according to claim 1, characterized in that: The sample main vehicle is a real vehicle, and the trained driving behavior model is connected to a virtual simulation platform to determine a future trajectory sequence of the target main vehicle as a virtual vehicle at a future moment, wherein the virtual simulation platform controls the target main vehicle to travel according to the future trajectory sequence at the future moment.

Citation Information

Cited By

  • Target long-time trajectory prediction method, system and device based on dynamic graph structure cell long-short-term memory network and medium

    CN119494859A

  • Video description method and system based on video point trajectory constraint

    CN120375376A

  • Video description method and system based on video point trajectory constraints

    CN120375376B

  • High-speed scene intelligent vehicle decision-making method based on time-space fusion attention

    CN120863681A

  • Trajectory tracking method

    FR3170411A1