Heterogeneous multi-vehicle trajectory prediction method and device, electronic equipment and storage medium

Abstract

The invention discloses a heterogeneous multi-vehicle trajectory prediction method and device, electronic equipment and a storage medium, and can be applied to the technical field of vehicle data processing. According to the method, after the heterogeneous vehicle original data is divided into the historical trajectory data and the future trajectory data, the historical trajectory data is preprocessed to obtain the first node feature, the edge feature and the map feature, and then the first node feature is input into the preset encoder architecture constructed through Transform to obtain the historical dynamic feature of the heterogeneous vehicle; then, carrying out attention interaction in combination with the edge features to obtain heterogeneous vehicle interaction features, carrying out attention interaction in combination with the map features to obtain road structure features, and splicing the heterogeneous vehicle historical dynamic features, the heterogeneous vehicle interaction features and the road structure features; and processing is performed based on the preset decreasing teaching learning decoder architecture and the future trajectory data to obtain a heterogeneous vehicle prediction trajectory, so that the accuracy of a heterogeneous multi-vehicle trajectory prediction result can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle data processing, and in particular to a method and device for heterogeneous multi-vehicle trajectory prediction, an electronic device, and a storage medium. Background Art

[0002] Among the related technologies, vehicle trajectory prediction is crucial for traffic flow management, vehicle collision avoidance, and path planning in the context of the rapid development of intelligent transportation systems and autonomous driving technologies. Current trajectory prediction methods are mainly based on statistical models, rule-based heuristic methods, or deep learning-based methods. However, these methods are mainly applicable to single-vehicle trajectory prediction, while the interactions between different vehicles are different, and the vehicle's driving trajectory on the road is also constrained by the road structure, resulting in low accuracy of the prediction results when the existing methods are applied to heterogeneous multi-vehicle trajectory prediction.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a heterogeneous multi-vehicle trajectory prediction method and device, an electronic device and a storage medium, which can effectively improve the accuracy of heterogeneous multi-vehicle trajectory prediction results.

[0005] To achieve the above object, an embodiment of the present application provides a heterogeneous multi-vehicle trajectory prediction method, the method comprising the following steps:

[0006] Acquire raw data of heterogeneous vehicles and divide the raw data of heterogeneous vehicles into historical trajectory data and future trajectory data, wherein the raw data of heterogeneous vehicles includes position information, speed information, vehicle type, and driving scene lane information of the heterogeneous vehicles;

[0007] Preprocessing the historical trajectory data to obtain first node features, edge features, and map features;

[0008] Inputting the first node feature into a preset encoder architecture, and outputting a heterogeneous vehicle historical dynamic feature, wherein the preset encoder architecture is constructed by Transformer;

[0009] Performing attention interaction between the heterogeneous vehicle historical dynamic features and the edge features to obtain heterogeneous vehicle interaction features;

[0010] Performing attention interaction between the heterogeneous vehicle historical dynamic features and the map features to obtain road structure features;

[0011] splicing the heterogeneous vehicle historical dynamic features, the heterogeneous vehicle interaction features and the road structure features to obtain a spliced ​​hidden state;

[0012] The spliced ​​hidden state and the future trajectory data are processed based on a preset decreasing teaching learning decoder architecture to obtain a predicted trajectory of the heterogeneous vehicle.

[0013] In some embodiments, the preprocessing of the historical trajectory data to obtain the first node feature, the edge feature and the map feature includes:

[0014] Screening the heterogeneous vehicle original data to obtain heterogeneous vehicle sample data;

[0015] Using the vehicles in the vehicle sample data as nodes and the relationships between the vehicles as directed edges to construct a first graph structure for each driving scene;

[0016] Features in the first graph structure are extracted to obtain the first node features, the edge features, and the map features.

[0017] In some embodiments, the preset encoder architecture performs the following steps:

[0018] Performing input embedding and time coding introduction on the first node feature to obtain a first embedded state;

[0019] Processing the first embedded state through a self-attention layer to obtain a first self-attention hidden state;

[0020] The first self-attention hidden state is processed through a feed-forward layer to obtain the historical dynamic features of the heterogeneous vehicles.

[0021] In some embodiments, the first node feature is input embedded and time encoded by the following formula:

[0022] H1=tanh(X1W e1 )+E p，i ;

[0023] In the formula, H1 represents the first embedded state, X1 represents the first node feature, and W e1 represents the trainable embedding weight; tanh() is the hyperbolic tangent activation function; E p，i Represents time coding, and the calculation formula is as follows:

[0024] When i is an even number:

[0025] E p，i = sin((p+31) / 10000 i / d);

[0026] When i is an odd number:

[0027] E p，i=cos((p+31) / 10000 i / d);

[0028] In the formula, p is the time index, i is the dimension index, and d is the time encoding dimension.

[0029] In some embodiments, performing attention interaction between the heterogeneous vehicle historical dynamic features and the edge features to obtain heterogeneous vehicle interaction features includes:

[0030] Performing different embedding processing on edge attribute features and edge type features in the edge features to obtain first features of the same dimension;

[0031] Concatenating the first features of the same dimension to obtain an edge embedding state;

[0032] Performing different embedding processing on the heterogeneous vehicle historical dynamic features to obtain a second same-dimensional feature, where the second same-dimensional feature has the same dimension as the first same-dimensional feature;

[0033] Concatenate the second same-dimensional feature and the edge embedding state to obtain an edge attention matrix;

[0034] The edge attribute feature in the edge feature is integrated with the historical dynamic feature of the heterogeneous vehicles to obtain a first joint feature;

[0035] The first joint feature is multiplied element-by-element by the edge attention matrix to obtain the heterogeneous vehicle interaction feature, where the heterogeneous vehicle interaction feature is used to characterize the relationship between vehicles.

[0036] In some embodiments, the performing attention interaction between the heterogeneous vehicle historical dynamic features and the map features to obtain road structure features includes:

[0037] Embedding the global map features in the map features to obtain a third feature of the same dimension, and embedding the local map features to obtain a fourth feature of the same dimension, wherein the third feature of the same dimension and the fourth feature of the same dimension have the same dimension;

[0038] Concatenate the third feature with the same dimension and the fourth feature with the same dimension to obtain a graph embedding state;

[0039] Embedding the heterogeneous vehicle historical dynamic features to obtain a fifth feature of the same dimension, wherein the fifth feature of the same dimension is the same as the dimension of the graph embedding state;

[0040] Concatenate the fifth same-dimensional feature with the graph embedding state to obtain a graph attention matrix;

[0041] The local map feature is combined with the historical dynamic feature of the heterogeneous vehicles to obtain a second joint feature;

[0042] The second joint feature is multiplied element-by-element by the graph attention matrix to obtain the road structure feature, where the road structure feature is used to characterize the relationship between the vehicle and the road.

[0043] In some embodiments, the processing of the concatenated hidden state and the future trajectory data based on a preset decreasing teaching learning decoder architecture to obtain a heterogeneous vehicle predicted trajectory includes:

[0044] The vehicles in the future trajectory data are used as nodes and the relationships between the vehicles are used as directed edges to construct a second graph structure for each driving scenario;

[0045] Extracting features from the second graph structure to obtain second node features;

[0046] Performing input embedding and time coding introduction on the second node feature to obtain a second embedded state;

[0047] Processing the second embedded state through a self-attention layer to obtain a second self-attention hidden state;

[0048] According to the concatenated hidden state, processing the second self-attention hidden state through an interactive attention layer to obtain an interactive attention hidden state;

[0049] Processing the interactive attention hidden state through a feedforward layer to obtain a feedforward hidden state;

[0050] The feed-forward hidden state is processed through a linear layer to obtain the predicted trajectory of the heterogeneous vehicle.

[0051] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a heterogeneous multi-vehicle trajectory prediction device, the device comprising:

[0052] The first module is used to obtain raw data of heterogeneous vehicles and divide the raw data of heterogeneous vehicles into historical trajectory data and future trajectory data, wherein the raw data of heterogeneous vehicles includes position information, speed information, vehicle type and driving scene lane information of heterogeneous vehicles;

[0053] The second module is used to pre-process the historical trajectory data to obtain first node features, edge features and map features;

[0054] A third module is used to input the first node feature into a preset encoder architecture and output a heterogeneous vehicle historical dynamic feature, wherein the preset encoder architecture is constructed by Transformer;

[0055] A fourth module is used to perform attention interaction between the heterogeneous vehicle historical dynamic features and the edge features to obtain heterogeneous vehicle interaction features;

[0056] A fifth module is used to perform attention interaction between the historical dynamic features of the heterogeneous vehicles and the map features to obtain road structure features;

[0057] A sixth module is used to splice the heterogeneous vehicle historical dynamic features, the heterogeneous vehicle interaction features and the road structure features to obtain a spliced ​​hidden state;

[0058] The seventh module is used to process the spliced ​​hidden state and the future trajectory data based on a preset decreasing teaching learning decoder architecture to obtain a predicted trajectory of a heterogeneous vehicle.

[0059] To achieve the above object, another aspect of an embodiment of the present application provides an electronic device, including:

[0060] at least one processor;

[0061] at least one memory for storing at least one program;

[0062] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0063] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0064] The embodiments of the present application include at least the following beneficial effects: The present application provides a heterogeneous multi-vehicle trajectory prediction method and device, an electronic device and a storage medium. The scheme obtains the original data of the heterogeneous vehicles and divides the original data of the heterogeneous vehicles into historical trajectory data and future trajectory data, pre-processes the historical trajectory data to obtain the first node feature, the edge feature and the map feature, and then inputs the first node feature into the preset encoder architecture constructed by Transformer to output the heterogeneous vehicle historical dynamic feature, and then performs attention interaction between the heterogeneous vehicle historical dynamic feature and the edge feature to obtain the heterogeneous vehicle interaction feature, and then performs attention interaction between the heterogeneous vehicle historical dynamic feature and the map feature to obtain the road structure feature, so that the Transformer architecture can be used to capture global information and solve the differences caused by heterogeneous vehicles. Then, the heterogeneous vehicle historical dynamic feature, the heterogeneous vehicle interaction feature and the road structure feature are spliced ​​to obtain the spliced ​​hidden state, and then the spliced ​​hidden state and the future trajectory data are processed based on the preset decreasing teaching learning decoder architecture to obtain the heterogeneous vehicle predicted trajectory, thereby effectively improving the accuracy of the heterogeneous multi-vehicle trajectory prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of a heterogeneous multi-vehicle trajectory prediction method provided in an embodiment of the present application;

[0066] Figure 2 It is a data processing flow chart of a model corresponding to the heterogeneous multi-vehicle trajectory prediction method provided in an embodiment of the present application;

[0067] Figure 3 It is a schematic diagram of the structure of a heterogeneous multi-vehicle trajectory prediction device provided in an embodiment of the present application;

[0068] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0070] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0071] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0073] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0074] In the embodiments of the present application, a heterogeneous multi-vehicle trajectory prediction method and device, an electronic device and a storage medium are provided. The present application can effectively solve the problems of long-distance dependence and insufficient consideration of multiple parameters in vehicle trajectory prediction by inputting the raw data of heterogeneous vehicles into a Transformer-based encoder to extract features; by using a specific type of encoder to extract the historical dynamic features of different types of vehicles, the differences caused by vehicle heterogeneity can be well solved; and by comprehensively considering the impact of vehicle historical dynamic features, vehicle interaction features and road structure features on vehicle driving trajectories, the complex interactive relationships between vehicles and between vehicles and the environment can be better captured, thereby reducing prediction errors and improving the accuracy and robustness of trajectory prediction. In addition, the use of a decreasing teaching learning decoder strategy in the model training stage helps the model gradually learn to use its own predicted values ​​as input, thereby reducing performance degradation in the reasoning stage.

[0075] The heterogeneous multi-vehicle trajectory prediction method provided in the embodiment of the present application relates to the field of vehicle data processing technology. The heterogeneous multi-vehicle trajectory prediction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the heterogeneous multi-vehicle trajectory prediction method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] Figure 1 is an optional flowchart of the heterogeneous multi-vehicle trajectory prediction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S170:

[0078] Step S110, obtaining heterogeneous vehicle raw data and dividing the heterogeneous vehicle raw data into historical trajectory data and future trajectory data, wherein the heterogeneous vehicle raw data includes position information, speed information, vehicle type, and driving scene lane information of the heterogeneous vehicle;

[0079] Step S120, preprocessing the historical trajectory data to obtain first node features, edge features and map features;

[0080] Step S130: input the first node feature into a preset encoder architecture, and output a heterogeneous vehicle historical dynamic feature, wherein the preset encoder architecture is constructed by Transformer;

[0081] Step S140, performing attention interaction on the historical dynamic features and edge features of the heterogeneous vehicles to obtain heterogeneous vehicle interaction features;

[0082] Step S150, performing attention interaction between historical dynamic features of heterogeneous vehicles and map features to obtain road structure features;

[0083] Step S160, splicing the historical dynamic features of heterogeneous vehicles, the interaction features of heterogeneous vehicles and the road structure features to obtain a spliced ​​hidden state;

[0084] Step S170: Process the concatenated hidden state and future trajectory data based on a preset decreasing teaching learning decoder architecture to obtain a predicted trajectory of the heterogeneous vehicle.

[0085] It can be understood that Transformer is a deep learning model architecture. The vehicle's position information includes the horizontal and vertical coordinates of the position, and the speed information includes the lateral and longitudinal speeds, lateral and longitudinal accelerations of the vehicle. The driving scene lane information can be obtained through the scene lane line grayscale map. Among them, the sampling frequency of the acquisition process in this embodiment can be 10Hz, and heterogeneous vehicles refer to different types of vehicles, which can be divided into two categories: motor vehicles and non-motor vehicles. Specifically, this embodiment can obtain 8s of data as the original data of heterogeneous vehicles, and use the first 3s of data as historical trajectory data, and the last 5s of data as future trajectory data. The data can also be divided into a training set and a validation set in a ratio of 8:2.

[0086] In the embodiment of the present application, the process of preprocessing the historical trajectory data to obtain the first node feature, the edge feature and the map feature includes but is not limited to the following steps:

[0087] Step S210, screening the heterogeneous vehicle original data to obtain heterogeneous vehicle sample data;

[0088] Step S220: construct a first graph structure for each driving scene by using vehicles in the vehicle sample data as nodes and the relationships between the vehicles as directed edges;

[0089] Step S230: extract features from the first graph structure to obtain first node features, edge features, and map features.

[0090] It is understandable that the screening process of this embodiment can be to retain the data with a collection time of no less than 8s for this vehicle, and randomly select 8s of continuous data as heterogeneous vehicle sample data. Specifically, for the case where the collection time of surrounding vehicles is less than 8s, the front and back filling method can be adopted to supplement it. After completing the data screening, for each scene, a dedicated first graph structure is constructed, that is, the vehicles in the scene are regarded as nodes, and the relationship between vehicles is regarded as directed edges to generate the first graph structure. Then the graph features extracted from the first graph structure include: node features, edge features, and map features, among which edge features include edge attribute features and edge type features, and map features include global map features and local map features.

[0091] For node features, there are 6 features in total, including the vehicle's horizontal coordinate, vertical coordinate, lateral speed, longitudinal speed, lateral acceleration, and longitudinal acceleration;

[0092] For edge attribute features, it represents the features of directed edges, including the relative position abscissa, relative position ordinate, relative lateral velocity, relative longitudinal velocity, relative lateral acceleration, and relative longitudinal acceleration of the two nodes connected by it, a total of 6 features;

[0093] For the edge type feature, it represents the vehicle category combination of the two nodes it connects, and only includes two categories: motor vehicles and non-motor vehicles. For example, motor vehicles are represented by 1 and non-motor vehicles are represented by 0. The edge type feature from motor vehicles to non-motor vehicles is represented by [1,0];

[0094] For global map features, it represents a fixed-size global map grayscale value matrix; for local map features, it represents a fixed-size local map grayscale value matrix.

[0095] In the embodiments of the present application, Figure 2 As shown, after the first node feature is input into the preset encoder frame, the first node feature is first input embedded and time encoded, and then passes through a self-attention layer and a forward feedback layer, and the encoder output layer outputs the heterogeneous vehicle historical dynamic features. Specifically, the preset encoder frame will perform the following steps based on the first node feature:

[0096] Step S310: embedding the first node feature and introducing time coding to obtain a first embedded state;

[0097] Step S320: Process the first embedded state through the self-attention layer to obtain a first self-attention hidden state;

[0098] Step S330: Process the first self-attention hidden state through the feedforward layer to obtain the historical dynamic characteristics of the heterogeneous vehicles.

[0099] It can be understood that the first node feature X1 is used as the input of the preset encoder architecture, which includes the position coordinates of all vehicles Position ordinate Lateral speed Longitudinal speed Lateral acceleration Longitudinal acceleration Where p∈{-30, -29, …, -1} represents the historical time set, k∈{1, 2, …, n} represents the scene vehicle set, and its total number n is determined by the specific number of vehicles in the scene.

[0100] Then the first node feature X1 is input embedded and time encoded by the following formula:

[0101] H1=tanh(X1W e1 )+E p，i ;

[0102] In the formula, H1 represents the first embedded state, X1 represents the first node feature, and W e1 represents the trainable embedding weight; tanh() is the hyperbolic tangent activation function; E p，i Represents time coding, and the calculation formula is as follows:

[0103] When i is an even number:

[0104] E p，i = sin((p+31) / 10000 i / d);

[0105] When i is an odd number:

[0106] E p，i =cos((p+31) / 10000 i / d);

[0107] In the formula, p is the time index, i is the dimension index, and d is the time encoding dimension.

[0108] The processing process of the self-attention layer on the first embedded state H1 is as follows:

[0109] A1=lnorm(smha(H1, H1, H1)+H1);

[0110] In the formula, A1 represents the first self-attention hidden state, smha() is the scaled multi-head self-attention mechanism, and lnorm() is the layer normalization.

[0111] The process of the feedforward layer processing the first self-attention hidden state A1 is as follows:

[0112] D=lnorm(relu(A1W1)W′1+A1);

[0113] In the formula, D represents the historical dynamic characteristics of heterogeneous vehicles, W1 and W′1 are learnable feedforward weights, relu() is the activation function, and lnorm() is the layer normalization.

[0114] In the embodiment of the present application, after obtaining the heterogeneous vehicle historical dynamic features D, the heterogeneous vehicle historical dynamic features and edge features are subjected to attention interaction to obtain heterogeneous vehicle interaction features that can characterize the relationship between vehicles. The processing process includes but is not limited to the following steps:

[0115] Step S410: Perform different embedding processes on edge attribute features and edge type features in edge features to obtain first same-dimensional features; wherein the first same-dimensional features include dimensional features after the edge attribute features are embedded and dimensional features after the edge type features are embedded, and the dimensions of these two dimensional features are the same;

[0116] Step S420, concatenate the first features of the same dimension to obtain an edge embedding state;

[0117] Step S430: Perform different embedding processes on the historical dynamic features of the heterogeneous vehicles to obtain second features of the same dimension, wherein the second features of the same dimension have the same dimension as the first features of the same dimension;

[0118] Step S440: concatenate the second same-dimensional feature and the edge embedding state to obtain the edge attention matrix a ed ;

[0119] Step S450: fusing the edge attribute feature in the edge feature with the heterogeneous vehicle historical dynamic feature D to obtain a first joint feature;

[0120] Step S460: Combine the first joint feature with the edge attention matrix a ed Perform element-by-element multiplication to obtain the heterogeneous vehicle interaction feature I.

[0121] In the embodiment of the present application, after obtaining the heterogeneous vehicle interaction feature I, the attention interaction is performed on the heterogeneous vehicle historical dynamic features and the map features to obtain a road structure feature M that can characterize the relationship between the vehicle and the road, and the processing process includes but is not limited to the following steps:

[0122] Step S510: embedding the global map features in the map features to obtain a third feature of the same dimension, and embedding the local map features to obtain a fourth feature of the same dimension, wherein the third feature of the same dimension and the fourth feature of the same dimension have the same dimension;

[0123] Step S520, concatenate the third feature with the same dimension and the fourth feature with the same dimension to obtain a graph embedding state;

[0124] Step S530: embedding the heterogeneous vehicle historical dynamic features D according to different vehicle types to obtain fifth features of the same dimension, wherein the fifth features of the same dimension have the same dimension as the embedding state of the graph;

[0125] Step S540: concatenate the fifth same-dimensional feature with the graph embedding state to obtain the graph attention matrix a amd ;

[0126] Step S550, fusing the local map feature with the historical dynamic feature D of the heterogeneous vehicles to obtain a second joint feature;

[0127] Step S560: Combine the second joint feature with the graph attention matrix a amd Perform element-by-element multiplication to obtain the road structure feature M.

[0128] In the embodiment of the present application, after obtaining the heterogeneous vehicle historical dynamic features D, the heterogeneous vehicle interaction features I and the road structure features M, these features are spliced ​​to obtain the spliced ​​hidden state He ,like Figure 2 As shown, the hidden state H is concatenated e As k and v of the Transformer decoder, they are used by the interactive attention layer. At the same time, the concatenated features are decoded using the decremental teaching and learning decoder architecture built on the Transformer to obtain the predicted vehicle trajectory. Specifically, this embodiment uses the future trajectory data X2 as the input of the decoder, where the future trajectory data X2 includes the horizontal coordinates of the positions of all vehicles Position ordinate Lateral speed Longitudinal speed Lateral acceleration Longitudinal acceleration Where q∈{1, 2, ..., 50} represents a future time set, k∈{1, 2, ..., n} represents a scene vehicle number set, and the total number n is determined by the specific number of vehicles in the scene. It can be understood that the process of processing based on the future trajectory data X2 in this embodiment includes but is not limited to the following steps:

[0129] Step S610: construct a second graph structure for each driving scene by using vehicles in the future trajectory data X2 as nodes and the relationships between vehicles as directed edges;

[0130] Step S620: extract features in the second graph structure to obtain second node features;

[0131] Step S630, input embedding and time coding introduction are performed on the second node feature to obtain a second embedded state H2;

[0132] Step S640: Process the second embedded state through the self-attention layer to obtain a second self-attention hidden state;

[0133] Step S650: According to the concatenated hidden state, the second self-attention hidden state is processed by the interactive attention layer to obtain the interactive attention hidden state;

[0134] Step S660: Process the interactive attention hidden state through the feedforward layer to obtain a feedforward hidden state;

[0135] The feed-forward hidden states are processed through a linear layer to obtain the predicted trajectories of heterogeneous vehicles.

[0136] Specifically, the process of input embedding and time encoding introduction of the second node feature is as follows:

[0137] H2=tanh(X2W e2 )+E p，i ;

[0138] In the formula, H2 represents the second embedded state; W e2 represents the trainable embedding weight, tanh() is the hyperbolic tangent activation function; E p，i Represents time coding, and the calculation formula is as follows:

[0139] When i is an even number:

[0140] E p，i = sin((p+31) / 10000 i / d);

[0141] When i is an odd number:

[0142] E p，i =cos((p+31) / 10000 i / d);

[0143] In the formula, p is the time index, i is the dimension index, and d is the time encoding dimension.

[0144] The process of the self-attention layer processing the second embedded state is as follows:

[0145] A2=lnorm(smha(H2, H2, H2)+H2);

[0146] In the formula, A2 represents the second self-attention hidden state, smha() is the scaled multi-head self-attention mechanism, and lnorm() is the layer normalization. Since the future state is not visible during prediction, the attention matrix obtained in this embodiment is masked when performing self-attention calculation to cover up the data after the current moment.

[0147] The process of the interactive attention layer processing the second self-attention hidden state is as follows:

[0148] A3=lnorm(smha(A2,H e , H e )+A2);

[0149] In the formula, A3 represents the hidden state of interactive attention, H e represents the concatenated hidden state obtained by the encoder, smha() is the scaled multi-head self-attention mechanism, and lnorm() is the layer normalization. Since the future state is not visible during prediction, the attention matrix obtained in this embodiment is masked when performing interactive attention calculation to cover up the data after the current moment.

[0150] The process of the feedforward layer processing the interactive attention hidden state is as follows:

[0151] H3=lnorm(relu(A3W2)W′2+A3);

[0152] In the formula, H3 represents the feedforward hidden state, W2 and W′2 are learnable feedforward weights, relu() is the activation function, and lnorm() is the layer normalization.

[0153] The process of the linear layer processing the feedforward hidden state is as follows:

[0154] O=H3W3;

[0155] In the formula, O represents the predicted trajectory of heterogeneous vehicles, including the horizontal coordinates of the positions of all vehicles Position ordinate Lateral speed Longitudinal speed Lateral acceleration Longitudinal acceleration j∈{1,2,…,50} represents the future time set, k∈{1,2,…,n} represents the number of scene vehicles, and the total number n is determined by the specific number of vehicles in the scene.

[0156] In the embodiment of the present application, since the model architecture can use future true values ​​as the input of the decoder during training, but can only use the predicted values ​​of the previous time step during real prediction, the performance of the model may be reduced when making real predictions; therefore, the embodiment of the present application uses a Transformer decoder architecture with decreasing teaching and learning. During training, the decoder input of the next 5s gradually transitions from the true value to the predicted value, that is, the probability of using the true value gradually transitions from 1 to 0, so that the model architecture gradually adapts to using its own predicted value as input, thereby improving the performance of the reasoning stage.

[0157] In summary, the method provided by the embodiment of the present application can significantly improve the trajectory prediction accuracy of heterogeneous multiple vehicles. Specifically, this embodiment uses a Transformer architecture that integrates time features to construct an encoder, so that it can better capture global information; and uses specific types of encoders for different types of vehicles to extract their historical dynamic features, which can well solve the differences caused by vehicle heterogeneity; at the same time, the three-channel network design that integrates historical dynamic features, vehicle interaction features, and road structure features in the model can better analyze the relationship between vehicles and various influencing factors, and obtain more accurate trajectory prediction results; and the decoder is constructed using a Transformer architecture with decremental teaching learning, so that the decoding input of the training process gradually changes from the true value to the predicted value, thereby making the model better adapted to using its own prediction results in the inference stage, thereby improving the robustness and prediction accuracy of the model.

[0158] Reference Figure 3, the embodiment of the present application provides a heterogeneous multi-vehicle trajectory prediction device, the device comprising:

[0159] The first module 710 is used to obtain heterogeneous vehicle raw data and divide the heterogeneous vehicle raw data into historical trajectory data and future trajectory data, wherein the heterogeneous vehicle raw data includes position information, speed information, vehicle type and driving scene lane information of the heterogeneous vehicle;

[0160] The second module 720 is used to pre-process the historical trajectory data to obtain first node features, edge features and map features;

[0161] The third module 730 is used to input the first node feature into a preset encoder architecture, and output the heterogeneous vehicle historical dynamic feature, wherein the preset encoder architecture is constructed by Transformer;

[0162] The fourth module 740 is used to perform attention interaction between the historical dynamic features of the heterogeneous vehicles and the edge features to obtain heterogeneous vehicle interaction features;

[0163] The fifth module 750 is used to perform attention interaction between the historical dynamic features of heterogeneous vehicles and the map features to obtain road structure features;

[0164] The sixth module 760 is used to splice the historical dynamic features of heterogeneous vehicles, the interaction features of heterogeneous vehicles and the road structure features to obtain a spliced ​​hidden state;

[0165] The seventh module 770 is used to process the spliced ​​hidden state and future trajectory data based on a preset decreasing teaching learning decoder architecture to obtain a predicted trajectory of the heterogeneous vehicle.

[0166] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0167] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned heterogeneous multi-vehicle trajectory prediction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0168] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] See also Figure 4 , Figure 4The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 810 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 820 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 820 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 820, and the processor 810 calls and executes the heterogeneous multi-vehicle trajectory prediction method of the embodiment of this application;

[0172] Input / output interface 830, used to implement information input and output;

[0173] The communication interface 840 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0174] bus 850 , which transmits information between the various components of the device (e.g., processor 810 , memory 820 , input / output interface 830 , and communication interface 840 );

[0175] The processor 810 , the memory 820 , the input / output interface 830 and the communication interface 840 are connected to each other in communication within the device via a bus 850 .

[0176] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned heterogeneous multi-vehicle trajectory prediction method is implemented.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0180] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0189] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A heterogeneous multi-vehicle trajectory prediction method, characterized in that: The method comprises the following steps: Acquire raw data of heterogeneous vehicles and divide the raw data of heterogeneous vehicles into historical trajectory data and future trajectory data, wherein the raw data of heterogeneous vehicles includes position information, speed information, vehicle type, and driving scene lane information of the heterogeneous vehicles; Preprocessing the historical trajectory data to obtain first node features, edge features, and map features; Inputting the first node feature into a preset encoder architecture, and outputting a heterogeneous vehicle historical dynamic feature, wherein the preset encoder architecture is constructed by Transformer; Performing attention interaction between the heterogeneous vehicle historical dynamic features and the edge features to obtain heterogeneous vehicle interaction features; Performing attention interaction between the heterogeneous vehicle historical dynamic features and the map features to obtain road structure features; splicing the heterogeneous vehicle historical dynamic features, the heterogeneous vehicle interaction features and the road structure features to obtain a spliced ​​hidden state; The spliced ​​hidden state and the future trajectory data are processed based on a preset decreasing teaching learning decoder architecture to obtain a predicted trajectory of the heterogeneous vehicle.

2. The method according to claim 1, characterized in that The preprocessing of the historical trajectory data to obtain first node features, edge features and map features includes: Screening the heterogeneous vehicle original data to obtain heterogeneous vehicle sample data; Using the vehicles in the vehicle sample data as nodes and the relationships between the vehicles as directed edges to construct a first graph structure for each driving scene; Features in the first graph structure are extracted to obtain the first node features, the edge features, and the map features.

3. The method according to claim 1, characterized in that: The preset encoder architecture performs the following steps: Performing input embedding and time coding introduction on the first node feature to obtain a first embedded state; Processing the first embedded state through a self-attention layer to obtain a first self-attention hidden state; The first self-attention hidden state is processed through a feed-forward layer to obtain the historical dynamic features of the heterogeneous vehicles.

4. The method according to claim 3, characterized in that The first node feature is input embedded and time encoded by the following formula: h1=tanh(x1w e1 )+e p,i ; In the formula, H1 represents the first embedded state, X1 represents the first node feature, and W e1 represents the trainable embedding weight; tanh() is the hyperbolic tangent activation function; E p,i Represents time coding, and the calculation formula is as follows: When i is an even number: AND p,i =sin((p+31) / 10000 i / d); When i is an odd number: It is p,i =cos((p+31) / 10000 i / d); In the formula, p is the time index, i is the dimension index, and d is the time encoding dimension.

5. The method according to claim 1, characterized in that The performing attention interaction between the heterogeneous vehicle historical dynamic features and the edge features to obtain heterogeneous vehicle interaction features includes: Performing different embedding processing on edge attribute features and edge type features in the edge features to obtain first features of the same dimension; Concatenating the first features of the same dimension to obtain an edge embedding state; Performing different embedding processing on the heterogeneous vehicle historical dynamic features to obtain a second same-dimensional feature, where the second same-dimensional feature has the same dimension as the first same-dimensional feature; Concatenate the second same-dimensional feature and the edge embedding state to obtain an edge attention matrix; The edge attribute feature in the edge feature is integrated with the historical dynamic feature of the heterogeneous vehicles to obtain a first joint feature; The first joint feature is multiplied element-by-element by the edge attention matrix to obtain the heterogeneous vehicle interaction feature, where the heterogeneous vehicle interaction feature is used to characterize the relationship between vehicles.

6. The method according to claim 1, characterized in that The step of performing attention interaction between the heterogeneous vehicle historical dynamic features and the map features to obtain road structure features includes: Embedding the global map features in the map features to obtain a third feature of the same dimension, and embedding the local map features to obtain a fourth feature of the same dimension, wherein the third feature of the same dimension and the fourth feature of the same dimension have the same dimension; Concatenate the third feature with the same dimension and the fourth feature with the same dimension to obtain a graph embedding state; Embedding the heterogeneous vehicle historical dynamic features to obtain a fifth feature of the same dimension, wherein the fifth feature of the same dimension is the same as the dimension of the graph embedding state; Concatenate the fifth same-dimensional feature with the graph embedding state to obtain a graph attention matrix; The local map feature is combined with the historical dynamic feature of the heterogeneous vehicles to obtain a second joint feature; The second joint feature is multiplied element-by-element by the graph attention matrix to obtain the road structure feature, where the road structure feature is used to characterize the relationship between the vehicle and the road.

7. The method according to claim 1, characterized in that The method of processing the spliced ​​hidden state and the future trajectory data based on a preset decreasing teaching learning decoder architecture to obtain a heterogeneous vehicle predicted trajectory includes: The vehicles in the future trajectory data are used as nodes and the relationships between the vehicles are used as directed edges to construct a second graph structure for each driving scenario; Extracting features from the second graph structure to obtain second node features; Performing input embedding and time coding introduction on the second node feature to obtain a second embedded state; Processing the second embedded state through a self-attention layer to obtain a second self-attention hidden state; According to the concatenated hidden state, processing the second self-attention hidden state through an interactive attention layer to obtain an interactive attention hidden state; Processing the interactive attention hidden state through a feedforward layer to obtain a feedforward hidden state; The feed-forward hidden state is processed through a linear layer to obtain the predicted trajectory of the heterogeneous vehicle.

8. A heterogeneous multi-vehicle trajectory prediction device, characterized in that: The device comprises: The first module is used to obtain raw data of heterogeneous vehicles and divide the raw data of heterogeneous vehicles into historical trajectory data and future trajectory data, wherein the raw data of heterogeneous vehicles includes position information, speed information, vehicle type and driving scene lane information of heterogeneous vehicles; The second module is used to pre-process the historical trajectory data to obtain first node features, edge features and map features; A third module is used to input the first node feature into a preset encoder architecture and output a heterogeneous vehicle historical dynamic feature, wherein the preset encoder architecture is constructed by Transformer; A fourth module is used to perform attention interaction between the heterogeneous vehicle historical dynamic features and the edge features to obtain heterogeneous vehicle interaction features; A fifth module is used to perform attention interaction between the historical dynamic features of the heterogeneous vehicles and the map features to obtain road structure features; A sixth module is used to splice the heterogeneous vehicle historical dynamic features, the heterogeneous vehicle interaction features and the road structure features to obtain a spliced ​​hidden state; The seventh module is used to process the spliced ​​hidden state and the future trajectory data based on a preset decreasing teaching learning decoder architecture to obtain a predicted trajectory of the heterogeneous vehicle.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images