A trajectory prediction processing method and apparatus

Abstract

Embodiments of the present application relate to a trajectory prediction processing method and device, the method comprising: receiving a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence; and performing trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector; receiving a center line key point sequence of each first lane as a corresponding first key point sequence; and performing lane line feature vector conversion on each first key point sequence to generate a corresponding first lane line feature vector; identifying the association features of any first target and all other first targets to generate a corresponding first association feature set; identifying the association features of any first target and all first lanes to generate a corresponding second association feature set; and predicting the trajectory of each first target according to the first and second association feature sets corresponding to each first target to generate a corresponding first predicted trajectory. The prediction accuracy can be improved by the present application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a trajectory prediction processing method and device. BACKGROUND

[0002] The prediction module is an important module of automatic driving. The role of the prediction module is to predict the motion trajectory of other targets around the ego vehicle, for example, to predict the driving trajectory of other vehicles around the ego vehicle and generate corresponding prediction trajectory output to the downstream planning module. The downstream planning module further receives the target prediction trajectory to plan the ego vehicle driving trajectory as an avoidance reference. The conventional prediction module has relatively high accuracy when processing trajectory prediction of a single target vehicle, but the accuracy is not ideal when processing trajectory prediction of multiple targets in a complex traffic environment with multiple lanes and multiple target vehicles. The reason is that the conventional prediction module only superimposes single target prediction results based on the number of targets to achieve multi-target prediction in a complex traffic environment, without considering the correlation between target vehicles and between target vehicles and the road, so that the prediction accuracy is not enough. SUMMARY

[0003] The present application aims to overcome the defects of the prior art and provides a trajectory prediction processing method, device, electronic equipment and computer readable storage medium. The trajectory feature vector of a target is constructed based on gMLPs (based on MLPs with gating), the feature vector of a lane is constructed from a lane center line key point sequence, the correlation features between targets and between a target and a lane are identified based on the target trajectory feature vector and the lane feature vector using a Multi-Head Attention network, and the trajectory of each target is predicted based on the correlation feature set of each target. Through the present application, the trajectories of each target vehicle can be predicted in a complex traffic environment by considering the correlation between target vehicles and between target vehicles and the road, thereby further improving the prediction accuracy.

[0004] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present application provides a trajectory prediction processing method, which comprises:

[0005] receiving a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence; and performing trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector;

[0006] receiving a center line key point sequence of each first lane as a corresponding first key point sequence; and performing lane line feature vector conversion on each first key point sequence to generate a corresponding first lane line feature vector;

[0007] According to all the first trajectory feature vectors, the association features of any first target and other all first targets are identified to generate a corresponding first association feature set;

[0008] According to any first trajectory feature vector and all first lane line feature vectors, the association features of any first target and all first lanes are identified to generate a corresponding second association feature set;

[0009] According to the first and second association feature sets corresponding to each first target, the trajectory of each first target is predicted to generate a corresponding first predicted trajectory.

[0010] Preferably, the first historical trajectory sequence includes a plurality of first trajectory point data; the first trajectory point data at least includes a horizontal coordinate, a vertical coordinate, a speed, an orientation angle and an acceleration;

[0011] The first key point sequence includes a plurality of first key point data; the first key point data at least includes a horizontal coordinate, a vertical coordinate and a control state; the control state at least includes a left turn state, a right turn state and a straight ahead state;

[0012] The first association feature set includes a plurality of first association feature data;

[0013] The second association feature set includes a plurality of second association feature data.

[0014] Preferably, the trajectory feature vector conversion of each first historical trajectory sequence to generate a corresponding first trajectory feature vector, specifically includes:

[0015] Each first historical trajectory sequence is vector converted to generate a corresponding first vector; the first vector is composed of a plurality of second vectors, each second vector corresponding to a first trajectory point data; the shape of the second vector is X*1, X being the vector dimension; the shape of the first vector is Y*X*1, Y being the total number of first trajectory point data;

[0016] A first tensor is composed of all the first vectors of the first target; the shape of the first tensor is Z*Y*X*1, Z being the total number of first targets;

[0017] The first tensor is input into a gMLPs network to obtain a corresponding second tensor; the second tensor includes Z third vectors; each third vector corresponds to a first target;

[0018] The last first trajectory point data of each first historical trajectory sequence is vector converted to obtain a corresponding fourth vector;

[0019] The third and fourth vectors corresponding to each of the first target form a corresponding first trajectory feature vector.

[0020] Preferably, the lane line feature vector conversion is performed on each of the first key point sequence to generate a corresponding first lane line feature vector, specifically including:

[0021] The first lane line feature vector is formed by the fifth vector obtained by performing vector conversion on each of the first key point data.

[0022] Preferably, the association feature between any first target and all other first targets is identified according to all the first trajectory feature vectors to generate a corresponding first association feature set, specifically including:

[0023] Any one of the first trajectory feature vectors is selected as a first target vector, and any other first trajectory feature vector is selected as a second target vector; and the total number of the second target vectors is counted to generate a total number n.

[0024] The first target vector is used as a Q vector of a multi-head attention network; any second target vector is used as a K vector and a V vector of the multi-head attention network; and n input vector groups are formed by the Q vector and n pairs of K and V vectors; each input vector group includes a pair of K and V vectors and the Q vector.

[0025] n input vector groups are input into the multi-head attention network respectively, and n feature extraction results are obtained by the multi-head attention network based on each input vector group.

[0026] Each feature extraction result is used as a corresponding first association feature data, and a corresponding first association feature set is formed by n first association feature data.

[0027] Preferably, the association feature between any first target and all first lanes is identified according to any first trajectory feature vector and all first lane line feature vectors to generate a corresponding second association feature set, specifically including:

[0028] The current first trajectory feature vector is used as a third target vector; each first lane line feature vector is used as a first lane vector; and the total number of the first lane vectors is counted to generate a total number m.

[0029] inputting m input vector groups into the multi-head attention network respectively, performing attention feature extraction based on each of the input vector groups by the multi-head attention network to obtain m feature extraction results;

[0030] inputting m input vector groups into the multi-head attention network respectively, performing attention feature extraction based on each of the input vector groups by the multi-head attention network to obtain m feature extraction results;

[0031] inputting m input vector groups into the multi-head attention network respectively, performing attention feature extraction based on each of the input vector groups by the multi-head attention network to obtain m feature extraction results;

[0032] Preferably, the first prediction trajectory corresponding to each first target is generated by predicting the trajectory of each first target according to the first and second correlation feature sets corresponding to each first target, and specifically includes:

[0033] inputting m input vector groups into the multi-head attention network respectively, performing attention feature extraction based on each of the input vector groups by the multi-head attention network to obtain m feature extraction results;

[0034] The second aspect of the embodiment of the application provides a device for implementing the trajectory prediction processing method of the first aspect, and the device comprises a historical trajectory data processing module, a lane data processing module, an interaction feature extraction module and a trajectory prediction module.

[0035] The historical trajectory data processing module is configured to receive a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence, and perform trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector.

[0036] The lane data processing module is configured to receive a center line key point sequence of each first lane as a corresponding first key point sequence, and perform lane line feature vector conversion on each first key point sequence to generate a corresponding first lane line feature vector.

[0037] The interaction feature extraction module is configured to identify the correlation features of any first target and all other first targets according to all first trajectory feature vectors to generate a corresponding first correlation feature set, and identify the correlation features of any first target and all first lanes according to any first trajectory feature vector and all first lane line feature vectors to generate a corresponding second correlation feature set.

[0038] The trajectory prediction module is configured to predict a trajectory of each first target to generate a corresponding first predicted trajectory according to the first and second associated feature sets corresponding to each first target.

[0039] The third aspect of the embodiment of the present application provides an electronic device, comprising a memory, a processor and a transceiver.

[0040] The processor is configured to be coupled with the memory, read and execute instructions in the memory to realize the method steps of the first aspect.

[0041] The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transmission and reception.

[0042] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer instructions, when the computer instructions are executed by a computer, the computer instructions make the computer execute the method of the first aspect.

[0043] The embodiment of the present application provides a trajectory prediction processing method and device, an electronic device and a computer readable storage medium, a trajectory feature vector of a target is constructed based on a gMLPs network, a feature vector of a lane is constructed based on a lane center line key point sequence, and a multi-head attention network is used to identify associated features between targets and between targets and lanes based on the trajectory feature vector of the target and the feature vector of the lane, and the trajectory of each target is predicted based on the associated feature set of each target. Through the present application, the trajectories of each target vehicle can be predicted in a complex traffic environment by combining the associated relationship between target vehicles and between target vehicles and roads, and the prediction accuracy of multi-target trajectory prediction is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A trajectory prediction processing method provided by the first embodiment of the present application is shown in the figure;

[0045] Figure 2 A module structure diagram of a trajectory prediction processing device provided by the second embodiment of the present application is shown in the figure;

[0046] Figure 3 A structure schematic diagram of an electronic device provided by the third embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0048] The embodiment one of the present application provides a trajectory prediction processing method, which comprises the following steps. Figure 1 The embodiment one of the present application provides a trajectory prediction processing method, which comprises the following steps.

[0049] Step 1, receiving a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence; and performing trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector.

[0050] Specifically, step 11 comprises: receiving a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence.

[0051] The first historical trajectory sequence comprises a plurality of first trajectory point data; and the first trajectory point data at least comprises a horizontal coordinate, a vertical coordinate, a speed, an orientation angle and an acceleration.

[0052] The first target herein is other vehicles around the ego vehicle in the driving environment of the ego vehicle. There can be multiple targets around the ego vehicle, and therefore, the historical trajectory information of each target before the current time, i.e., the first historical trajectory sequence, needs to be obtained in advance before the trajectory prediction of the multiple targets. The first historical trajectory sequence is composed of a plurality of first trajectory point data in chronological order, each first trajectory point data corresponds to a set of trajectory information at a time point, which comprises a horizontal coordinate, a vertical coordinate, a speed, an orientation angle and an acceleration, or any combination of the horizontal coordinate, the vertical coordinate, the speed, the orientation angle and the acceleration. In the first historical trajectory sequence, the time corresponding to the last first trajectory point data is the current time, and the last first trajectory point data is a set of current horizontal coordinate, vertical coordinate, speed, orientation angle and acceleration information of the corresponding target.

[0053] Step 12, performing trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector.

[0054] Specifically, step 121 comprises: performing vector conversion on each first historical trajectory sequence to generate a corresponding first vector; the first vector is composed of a plurality of second vectors, each second vector corresponds to a first trajectory point data.

[0055] Wherein, the shape of the second vector is X*1, X is the vector dimension; the shape of the first vector is Y*X*1, Y is the total number of the first trajectory point data;

[0056] Here, when performing vector conversion on each first historical trajectory sequence, 5-dimensional vector conversion of X=5 is performed based on 5 trajectory information (abscissa, ordinate, speed, orientation angle and acceleration) of each first trajectory point data to obtain a corresponding second vector, and Y second vectors are spliced to form a first vector;

[0057] Step 122, forming a first tensor by all first vectors of the first target;

[0058] Wherein, the shape of the first tensor is Z*Y*X, Z is the total number of the first target;

[0059] Here, it is actually to splice all first vectors into a tensor with a shape of Z*Y*X;

[0060] Step 123, inputting the first tensor into the gMLPs network to obtain a corresponding second tensor;

[0061] Wherein, the second tensor includes Z third vectors; each third vector corresponds to a first target; the gMLPs network includes Z parallel L modules, each L module corresponds to a first target, and the input of each L module corresponds to a first vector and the output corresponds to a third vector;

[0062] Specifically, it includes: the gMLPs network extracts multiple first vectors from the input first tensor; and inputs each first vector into the corresponding L module to obtain a corresponding third vector; and splices the obtained multiple third vectors into a second tensor output;

[0063] Here, the gMLPs network is a neural network structure realized by a gating gating module to realize attention association disclosed by Google, and the implementation of the gMLPs network is described in detail in the public literature of Google team “Pay Attention to MLPs” (Author: Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le); As can be seen from the public literature, the gMLPs network is composed of multiple parallel L modules, each L module has a gating gating module, that is, the Spatial Gating Unit unit in the literature; Through the Spatial Gating Unit unit, all L modules can add global correlation features to the extracted features while extracting features of the input first vector, that is, the third vectors output by each L module have association with each other;

[0064] Step 124, vector conversion is performed on the last first trajectory point data of each first historical trajectory sequence to obtain a corresponding fourth vector;

[0065] Specifically, the second vector corresponding to the last first trajectory point data of each first historical trajectory sequence is recorded as a current second vector, and the horizontal coordinate and the vertical coordinate of the current second vector are linearly mapped using a preset Muti-Layer Perception (MLP) to obtain a high-dimensional vector and record it as a fourth vector;

[0066] Here, the linear mapping of the Muti-Layer Perception is actually a coordinate system conversion of the target current position coordinates, so that it can be in the same coordinate system as the lane key points used in the subsequent steps;

[0067] Step 125, the third and fourth vectors corresponding to each first target are combined to obtain a corresponding first trajectory feature vector.

[0068] Here, the third and fourth vectors are spliced to obtain the first trajectory feature vector.

[0069] Step 2, receiving the center line key point sequence of each first lane as the corresponding first key point sequence; and performing lane line feature vector conversion on each first key point sequence to generate a corresponding first lane line feature vector;

[0070] Specifically, step 21, receiving the center line key point sequence of each first lane as the corresponding first key point sequence;

[0071] Among them, the first key point sequence includes a plurality of first key point data; the first key point data at least includes horizontal coordinates, vertical coordinates, and control states; the control state at least includes left turn state, right turn state and straight state, or any combination of left turn state, right turn state and straight state;

[0072] Here, in the driving environment of the ego vehicle, there can be multiple lanes, each lane being a first lane; each lane has a center line, and the points on the center line are called center line key points, which are arranged in sequence according to the driving direction of the lane; before performing multi-target trajectory prediction on the driving environment of the ego vehicle, the center line key point information of each first lane in the driving environment of the ego vehicle, i.e., the first key point sequence, needs to be obtained in advance; the first key point sequence is composed of a plurality of first key point data according to the driving direction of the lane, and each first key point data corresponds to an information set of a center line key point, which at least includes the position information of the center line key point, i.e., the horizontal coordinates and the vertical coordinates, and also includes the lane control state information corresponding to the center line key point, such as left turn, right turn, and straight state;

[0073] Step 22, lane line feature vector conversion is performed on each first key point sequence to generate a corresponding first lane line feature vector;

[0074] Specifically, it includes: vector conversion is performed on each first key point data to generate a corresponding fifth vector; and the first lane line feature vector is composed of the plurality of fifth vectors obtained;

[0075] Wherein, the shape of the fifth vector is W*1, W is the vector dimension; the shape of the first lane line feature vector is H*W*1, H is the total number of first key point data.

[0076] Here, when vector conversion is performed on each first key point data, 3-dimensional vector conversion is performed based on 3 key point information (abscissa, ordinate, control state) of each first key point data to obtain a corresponding fifth vector, and the first lane line feature vector is composed of H fifth vectors.

[0077] Step 3, according to all first trajectory feature vectors, the association features of any first target and all other first targets are identified to generate a corresponding first association feature set;

[0078] Wherein, the first association feature set includes a plurality of first association feature data;

[0079] Here, the first association feature set actually reflects the association features between a specified target and other targets in the self-vehicle driving environment, each first association feature data corresponds to one other target, and the first association feature data is the association feature of the specified target and the corresponding other target;

[0080] Here, the current step is actually to extract the two-by-two association features of each first target and any other first target in the self-vehicle driving environment using a multi-head attention (Multi-Head Attention) network, and to construct a first association feature set corresponding to each first target reflecting the vehicle-to-vehicle association relationship according to the extraction result;

[0081] Specifically, it includes: step 31, selecting one of all first trajectory feature vectors as a first target vector, and selecting any other first trajectory feature vector as a second target vector; and the total number of second target vectors is counted to generate a total number n;

[0082] For example, there are 3 targets in the self-driving environment, all the first trajectory feature vectors include first trajectory feature vectors 1, 2, 3; when the first trajectory feature vector 1 is selected as the first target vector, the second target vectors include first trajectory feature vectors 2, 3, n = 2; when the first trajectory feature vector 2 is selected as the first target vector, the second target vectors include first trajectory feature vectors 1, 3, n = 2; when the first trajectory feature vector 3 is selected as the first target vector, the second target vectors include first trajectory feature vectors 1, 2, n = 2;

[0083] Step 32, the first target vector is taken as the Q vector of the multi-head attention network, and any second target vector is taken as the K vector and the V vector of the multi-head attention network; and n input vector groups are formed by the Q vector and n pairs of K, V vectors;

[0084] Each input vector group includes a pair of K, V vectors and a Q vector.

[0085] Here, subsequent steps of the embodiment of the application will perform pairwise correlation feature extraction on the first target vector and all other second target vectors based on the multi-head attention network. The multi-head attention network is actually formed by connecting multiple single-head attention networks in parallel. The multi-head attention network divides the input single input vector group into multiple sub-input vector groups, respectively sends the sub-input vector groups into the single-head attention networks for operation to obtain corresponding single-head output data, and splices the obtained multiple single-head output data to form multi-head data output. The detailed structure and operation mechanism of the multi-head attention network and the attention network can be obtained from public technical documents, and will not be described further here. Only one point needs to be explained here, that is, the multi-head attention network needs to obtain an input vector group composed of K, V and Q vectors before operation. The current step is to prepare these input vector groups.

[0086] The multi-head attention network is used here to calculate the correlation features of the first target vector and any second target vector. During calculation, the multi-head attention network takes the first target vector as the query vector (Q vector) and the corresponding second target vector as the key-value vector pair (K-V vector pair). Therefore, when preparing input data for the multi-head attention network corresponding to n second target vectors, n input vector groups need to be prepared. The Q vector of each input vector group is the same, which is the current first target vector, and the K-V vector pair is different, which corresponds to different second target vectors.

[0087] Step 33, the n input vector groups are respectively input into the multi-head attention network, and the multi-head attention network performs attention feature extraction based on each input vector group to obtain n feature extraction results.

[0088] Here, the n input vector groups are input into the multi-head attention network respectively, and the multi-head attention network calculates the association features of the corresponding first target vector and the corresponding second target vector based on the corresponding K-V vector pair and Q vector pair to obtain the corresponding feature extraction result, and finally n feature extraction results are obtained.

[0089] Step 34, each feature extraction result is taken as a corresponding first association feature data, and the n first association feature data obtained form a corresponding first association feature set.

[0090] For example, there are 3 first targets in the self-vehicle driving environment, and the corresponding all first trajectory feature vectors are first trajectory feature vectors 1-3.

[0091] When the first trajectory feature vector 1 is selected as the first target vector through step 31, the second target vectors include the first trajectory feature vectors 2 and 3, and n=2; the first association feature set of the first target 1 obtained through steps 32-34 is: first association feature data 目标1-目标2 , and first association feature data 目标1-目标3 .

[0092] When the first trajectory feature vector 2 is selected as the first target vector through step 31, the second target vectors include the first trajectory feature vectors 1 and 3, and n=2; the first association feature set of the first target 2 obtained through steps 32-34 is: first association feature data 目标2-目标1 , and first association feature data 目标2-目标3 .

[0093] When the first trajectory feature vector 3 is selected as the first target vector through step 31, the second target vectors include the first trajectory feature vectors 1 and 2, and n=2; the first association feature set of the first target 3 obtained through steps 32-34 is: first association feature data 目标3-目标1 , and first association feature data 目标3-目标2 .

[0094] Step 4, the association features of any first target and all first lanes are identified to generate a corresponding second association feature set according to any first trajectory feature vector and all first lane feature vectors;

[0095] The second association feature set includes a plurality of second association feature data.

[0096] Here, the second association feature set actually embodies the association features between a specified target and all lanes in the self-vehicle driving environment, each second association feature data corresponds to a first lane, and the second association feature data is the association feature between the specified target and the corresponding lane.

[0097] Here, the current step is actually to extract the two-by-two association features of each first target and any first lane in the self-vehicle driving environment by using the multi-head attention network, and to construct a second association feature set corresponding to each first target reflecting the vehicle-lane association relationship, that is, reflecting the vehicle-semantic map association relationship according to the extraction result;

[0098] Specifically, step 41, the current first trajectory feature vector is taken as a third target vector, each first lane line feature vector is taken as a first lane vector, and the total number of first lane vectors is counted to generate a total number m;

[0099] For example, there are 2 first targets and 3 first lanes in the self-vehicle driving environment, corresponding to first trajectory feature vectors 1, 2 and first lane line feature vectors 1, 2, 3; when the first trajectory feature vector 1 is taken as the third target vector, the first lane vector includes the first lane line feature vectors 1, 2, 3, and m = 3; when the first trajectory feature vector 2 is taken as the third target vector, the first lane vector also includes the first lane line feature vectors 1, 2, 3, and m = 3;

[0100] Step 42, the third target vector is taken as the Q vector of the multi-head attention network, and any first lane vector is taken as the K vector and the V vector of the multi-head attention network; and m input vector groups are formed by the Q vector and the K, V vectors;

[0101] Each input vector group includes a pair of K, V vectors and a Q vector;

[0102] Here, the multi-head attention network is used to calculate the association features of the third target vector and any first lane vector; when calculating, the multi-head attention network takes the third target vector as the query vector, that is, the Q vector, and takes the corresponding first lane vector as the key-value vector pair, that is, the K-V vector pair; therefore, when inputting data to the multi-head attention network corresponding to the m first lane vectors, m input vector groups need to be prepared, and the Q vector of each input vector group is the same, that is, the current third target vector, and the K-V vector pair is different, corresponding to different first lane vectors;

[0103] Step 43, m input vector groups are respectively input into the multi-head attention network, and the multi-head attention network performs attention feature extraction based on each input vector group to obtain m feature extraction results;

[0104] Here, m input vector groups are respectively input into the multi-head attention network, and the multi-head attention network performs operation on the association features of the third target vector and the corresponding first lane vector based on the corresponding K-V vector pair and the Q vector pair to obtain the corresponding feature extraction result, and finally m feature extraction results are obtained;

[0105] Step 44, each feature extraction result is taken as corresponding second associated feature data, and a corresponding second associated feature set is formed by the obtained m first associated feature data.

[0106] For example, there are 2 first targets and 3 first lanes in the self-vehicle driving environment, and all the first trajectory feature vectors are first trajectory feature vectors 1-2, and all the first lane line feature vectors are first lane line feature vectors 1-3.

[0107] When the first trajectory feature vector 1 is selected as the third target vector through step 41, the first lane vector includes the first lane line feature vector 1-3, and m=3; through steps 42-44, the second associated feature set of the first target 1 obtained is: second associated feature data 目标1-车道1 , second associated feature data 目标1-车道2 , and second associated feature data 目标1-车道3 .

[0108] When the first trajectory feature vector 2 is selected as the third target vector through step 41, the first lane vector includes the first lane line feature vector 1-3, and m=3; through steps 42-44, the second associated feature set of the first target 2 obtained is: second associated feature data 目标2-车道1 , second associated feature data 目标2-车道2 , and second associated feature data 目标2-车道3 .

[0109] Step 5, according to the first and second associated feature sets corresponding to each first target, the trajectory of each first target is predicted to generate a corresponding first predicted trajectory.

[0110] Specifically, it includes: inputting the first and second associated feature sets corresponding to the current first target into a multimodal probability trajectory prediction model to obtain a plurality of predicted trajectories and corresponding prediction probabilities; and selecting the predicted trajectory with the maximum prediction probability as the corresponding first predicted trajectory.

[0111] Here, the multi-mode probability trajectory prediction model can predict multiple possible trajectories of a specified target in a future time period based on a first set of associated features reflecting interaction characteristics of the specified target with other targets around it, and a second set of associated features reflecting interaction characteristics of the specified target with adjacent lanes, and give a corresponding prediction probability for each predicted trajectory. The multi-mode probability trajectory prediction model is composed of a trajectory probability unit and a trajectory prediction unit; the trajectory probability unit can be implemented by a multi-layer perception network, and the trajectory prediction unit can be implemented based on a Long Short-Term Memory (LSTM) artificial neural network; the trajectory probability unit performs branch trajectory probability calculation based on the input first and second sets of associated features to obtain multiple branch trajectory probabilities, i.e., prediction probabilities; the trajectory prediction unit performs branch trajectory prediction based on the input first and second sets of associated features to obtain multiple branch trajectories, i.e., predicted trajectories; each predicted trajectory corresponds to a prediction probability.

[0112] Figure 2 A module structure diagram of a trajectory prediction processing device provided for the second embodiment of the present application is shown in FIG. 2. The device is a terminal device or a server for implementing the method embodiments described above, or a device capable of enabling the terminal device or the server to implement the method embodiments described above, such as a device or a chip system of the terminal device or the server. As shown in FIG. 2, the device includes a historical trajectory data processing module 101, a lane data processing module 102, an interaction feature extraction module 103, and a trajectory prediction module 104. Figure 2

[0113] The historical trajectory data processing module 101 is configured to receive a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence, and perform trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector.

[0114] The lane data processing module 102 is configured to receive a center line key point sequence of each first lane as a corresponding first key point sequence, and perform lane line feature vector conversion on each first key point sequence to generate a corresponding first lane line feature vector.

[0115] The interaction feature extraction module 103 is configured to identify the associated features of any first target with all other first targets based on all first trajectory feature vectors to generate a corresponding first set of associated features, and identify the associated features of any first target with all first lanes based on any first trajectory feature vector and all first lane line feature vectors to generate a corresponding second set of associated features.

[0116] The trajectory prediction module 104 is configured to predict the trajectory of each first target based on the corresponding first and second sets of associated features of each first target to generate a corresponding first predicted trajectory.​

[0117] The trajectory prediction processing apparatus provided by the embodiment of the present application can execute the method steps in the method embodiments, and has similar implementation principles and technical effects, which will not be repeated here.

[0118] It should be noted that the division of each module of the above apparatus is only a logical division of functions, and all or part of the modules can be integrated into one physical entity or physically separated when actually implemented. Moreover, all the modules can be implemented in the form of software invoked by a processing element; all the modules can also be implemented in the form of hardware; or some modules are implemented in the form of software invoked by a processing element, and some modules are implemented in the form of hardware. For example, the historical trajectory data processing module can be a separately established processing element, or can be integrated into a chip of the above apparatus, in addition, the historical trajectory data processing module can also be stored in the form of program code in the memory of the above apparatus, and the functions of the historical trajectory data processing module can be invoked and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of the modules can be integrated together or independently implemented. The processing element described herein can be an integrated circuit having a signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0119] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of program code invoked by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of invoking program code. For another example, the modules can be integrated together to implement in the form of a system on a chip (SOC).

[0120] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0121] Figure 3 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of the embodiments of the present invention. Figure 3 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0122] exist Figure 3The system bus 305 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the above-mentioned system bus 305, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as the client, the read-write library and the read-only library). The memory can include Random Access Memory (RAM), and can also include Non-Volatile Memory, such as at least one disk memory.

[0123] The processor described above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0124] It should be noted that the embodiments of the present application also provide a computer readable storage medium, which stores instructions, when the instructions are run on a computer, the computer executes the method and process provided in the above embodiments.

[0125] The embodiments of the present application also provide a chip for running instructions, which is used to execute the processing steps described in the foregoing method embodiments.

[0126] The embodiments of the present application provide a trajectory prediction processing method and device, electronic equipment and computer readable storage medium, the trajectory feature vector of the target is constructed based on the gMLPs network, the feature vector of the lane is constructed by the lane center line key point sequence, and the association features between the targets and the targets, and the targets and the lanes are identified based on the target trajectory feature vector and the lane feature vector using the multi-head attention network, and the trajectory of each target is predicted based on the association feature set of each target. Through the present application, the association relationship between the target vehicles and the association relationship between the target vehicles and the road can be combined to predict the trajectory of each target vehicle in a complex traffic environment, and the prediction accuracy of multi-target trajectory prediction is improved.

[0127] Those skilled in the art should further appreciate that the elements and algorithms described in connection with the examples disclosed herein can be embodied in electronic hardware, computer software, or in combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their general functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0128] The steps of a method or algorithm described in connection with the examples disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0129] The specific implementation described above is for the purpose of further illustrating the objects, technical solutions, and advantages of the present application. It should be understood that the above description is merely a specific implementation of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.

Claims

1. A trajectory prediction processing method characterized by comprising: The method comprises: receiving a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence; and performing trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector; receiving a centerline key point sequence of each first lane as a corresponding first key point sequence; and performing lane line feature vector conversion on each first key point sequence to generate a corresponding first lane line feature vector; identifying the association features of any first target and all other first targets according to all first trajectory feature vectors to generate a corresponding first association feature set; identifying the association features of any first target and all first lanes according to any first trajectory feature vector and all first lane line feature vectors to generate a corresponding second association feature set; predicting the trajectory of each first target according to the first and second association feature sets corresponding to each first target to generate a corresponding first predicted trajectory; wherein the trajectory feature vector conversion on each first historical trajectory sequence to generate a corresponding first trajectory feature vector specifically comprises: vector conversion on each first historical trajectory sequence to generate a corresponding first vector, the first vector being composed of multiple second vectors, each second vector corresponding to a first trajectory point data; a first tensor being composed of the first vectors of all first targets; inputting the first tensor into a gMLPs network to obtain a corresponding second tensor, the second tensor including Z third vectors, Z being the total number of first targets; vector conversion on the last first trajectory point data of each first historical trajectory sequence to obtain a corresponding fourth vector; the third and fourth vectors corresponding to each first target being composed of a corresponding first trajectory feature vector; the identification of the association features of any first target and all other first targets according to all first trajectory feature vectors to generate a corresponding first association feature set specifically comprises: selecting any one of all first trajectory feature vectors as a first target vector, and selecting any other first trajectory feature vector as a second target vector; counting the total number of second target vectors to generate a total number n; taking the first target vector as a Q vector of a multi-head attention network; taking any second target vector as a K vector and a V vector of the multi-head attention network; and composing n input vector groups from the Q vector and n K and V vectors; each input vector group including a pair of K and V vectors and the Q vector; inputting n input vector groups into the multi-head attention network respectively, and performing attention feature extraction based on each input vector group by the multi-head attention network to obtain n feature extraction results; taking each feature extraction result as a corresponding first association feature data, and composing a corresponding first association feature set from the obtained n first association feature data; The trajectory of each first target is predicted to generate a corresponding first predicted trajectory according to the first and second associated feature sets corresponding to each first target, specifically including: inputting the first and second associated feature sets corresponding to the current first target into a multimodal probability trajectory prediction model to obtain multiple predicted trajectories and corresponding prediction probabilities; and selecting the predicted trajectory with the maximum prediction probability as the corresponding first predicted trajectory. The last first trajectory point data of each first historical trajectory sequence is vector converted to obtain a corresponding fourth vector, specifically including: using an MLP model to perform linear mapping on the horizontal coordinates and vertical coordinates of the second vector corresponding to the last first trajectory point data to obtain the corresponding fourth vector. The multimodal probability trajectory prediction model is composed of a trajectory probability unit and a trajectory prediction unit; the trajectory probability unit is implemented by a multilayer perception network, and the trajectory prediction unit is implemented based on a long short-term memory artificial neural network; the trajectory probability unit performs branch trajectory probability calculation based on the input first and second associated feature sets to obtain multiple prediction probabilities; the trajectory prediction unit performs branch trajectory prediction based on the input first and second associated feature sets to obtain multiple predicted trajectories; each predicted trajectory corresponds to a prediction probability.

2. The trajectory prediction processing method according to claim 1, characterized in that: The first historical trajectory sequence includes multiple first trajectory point data; and the first trajectory point data at least includes horizontal coordinates, vertical coordinates, speed, orientation angle and acceleration. The first key point sequence includes multiple first key point data; and the first key point data at least includes horizontal coordinates, vertical coordinates and control state; and the control state at least includes left turn state, right turn state and straight state. The first associated feature set includes multiple first associated feature data. The second associated feature set includes multiple second associated feature data.

3. The trajectory prediction processing method according to claim 2, characterized in that: The shape of the second vector is X*1, X being a vector dimension; The shape of the first vector is Y*X*1, Y being a total number of the first trajectory point data; The shape of the first tensor is Z*Y*X*1; Each third vector corresponds to one first target.

4. The trajectory prediction processing method according to claim 2, characterized in that, The first lane line feature vector corresponding to each first key point sequence is generated by vector converting each first key point data, specifically including: Each fifth vector corresponding to each first key point data is generated by vector converting each first key point data; and the corresponding first lane line feature vector is composed of multiple fifth vectors obtained.

5. The trajectory prediction processing method according to claim 2, characterized in that, The second associated feature set corresponding to any first target and all first lanes is generated by identifying the associated features according to any first trajectory feature vector and all first lane line feature vectors, specifically including: taking the first trajectory feature vector as a third target vector, taking each first lane line feature vector as a first lane vector, and counting the total number of the first lane vectors to generate a total number m; taking the third target vector as a Q vector of a multi-head attention network, taking any first lane vector as a K vector and a V vector of the multi-head attention network, and generating m input vector groups by the Q vector and m K and V vectors; inputting the m input vector groups into the multi-head attention network respectively, and obtaining m feature extraction results by the multi-head attention network based on each input vector group; taking each feature extraction result as corresponding second association feature data, and generating a corresponding second association feature set by the m first association feature data.

6. An apparatus for implementing the trajectory prediction processing method of any one of claims 1-5, characterized in that, The device comprises a historical trajectory data processing module, a lane data processing module, an interaction feature extraction module, and a trajectory prediction module. The historical trajectory data processing module is configured to receive a historical trajectory sequence of each first target as a corresponding first historical trajectory sequence, and convert the trajectory feature vector of each first historical trajectory sequence to generate a corresponding first trajectory feature vector. The lane data processing module is configured to receive a center line key point sequence of each first lane as a corresponding first key point sequence, and convert the lane line feature vector of each first key point sequence to generate a corresponding first lane line feature vector. The interaction feature extraction module is configured to identify the association features of any first target and all other first targets based on all first trajectory feature vectors to generate a corresponding first association feature set, and identify the association features of any first target and all first lanes based on any first trajectory feature vector and all first lane line feature vectors to generate a corresponding second association feature set. The trajectory prediction module is configured to predict the trajectory of each first target based on the first and second association feature sets of each first target to generate a corresponding first predicted trajectory.

7. An electronic device, comprising: The device comprises a memory, a processor, and a transceiver. The processor is configured to read and execute instructions in the memory to implement the method of any one of claims 1-5. The transceiver is coupled with the processor, and is controlled by the processor to perform message transmission and reception. The computer readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method of any one of claims 1-5.

Images