Obstacle trajectory prediction method, device, storage medium and electronic device

Abstract

This specification discloses an obstacle trajectory prediction method, device, storage medium and electronic device, which can obtain the driving data of the obstacle in the first time period in history, and use the state data at different times as row data and the data of different data dimensions as column data to construct a state matrix. Afterwards, the state matrix is ​​respectively input into the time coding layer and the space coding layer of the trajectory prediction model to obtain the correlation features in the time dimension and the correlation features in the space dimension. Finally, according to the correlation features of the obstacle in the time dimension and the space dimension, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model. The historical driving data of the obstacle is encoded from the time dimension and the space dimension respectively, and the correlation between the various features of the obstacle in the time dimension and the space dimension is learned, thereby improving the accuracy of trajectory prediction.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving technology, and in particular to an obstacle trajectory prediction method, device, storage medium and electronic device. Background Art

[0002] With the development of unmanned driving technology, unmanned driving equipment is being used more and more widely in multiple fields, bringing more convenience to people's lives.

[0003] In order to ensure the safe driving of unmanned driving equipment, when planning the driving trajectory of the unmanned driving equipment, it is often necessary to predict the driving trajectory of surrounding obstacles so that the unmanned driving equipment can avoid various obstacles during driving.

[0004] At present, when predicting the trajectory of an obstacle, we can first obtain the historical driving trajectory of the obstacle, and then encode the historical driving trajectory through the long short-term memory network (LSTM) to obtain the relationship between the driving trajectories at different times. After that, the LSTM decoder is used to decode and predict the driving trajectory of the obstacle within a certain period of time in the future.

[0005] However, based on the relationship between driving trajectories at different times, only the correlation of driving trajectories in the time dimension is considered, resulting in inaccurate trajectory prediction results, which affects the driving safety of unmanned driving equipment. Summary of the invention

[0006] The embodiments of this specification provide an obstacle trajectory prediction method, device, storage medium, and electronic device to partially solve the problems in the prior art.

[0007] The embodiments of this specification adopt the following technical solutions:

[0008] This specification provides an obstacle trajectory prediction method, including:

[0009] Acquire the travel data of the obstacle within a first time period in history, wherein the travel data includes state data at several moments within the first time period;

[0010] Using state data at different times as row data and data of different data dimensions of the state data as column data to construct a state matrix;

[0011] Input the state matrix into the time encoding layer of the pre-trained trajectory prediction model, perform column convolution on the state matrix, fuse the features of the same data dimension at different times, and obtain the associated features of the obstacle in the time dimension;

[0012] Input the state matrix into the spatial encoding layer of the pre-trained trajectory prediction model, perform row convolution on the state matrix, fuse the features of different data dimensions at the same time, and obtain the associated features of the obstacle in the spatial dimension;

[0013] According to the associated features of the obstacle in the time dimension and the space dimension respectively, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model.

[0014] Optionally, before inputting the state matrix into a time coding layer and / or a space coding layer of a pre-trained trajectory prediction model, the method further comprises:

[0015] For each moment in the first time period, determining a correlation matrix corresponding to the moment according to the state data of the obstacle corresponding to the moment and the state data of the obstacle corresponding to each other moment, wherein the correlation matrix represents the correlation between the state data of each other moment and the state data of the moment;

[0016] Determine an attention weight matrix according to the correlation matrix corresponding to each moment in the first duration;

[0017] The state matrix is ​​attention-weighted according to the attention weight matrix to determine a weighted state matrix.

[0018] Optionally, the temporal coding layer includes a standard convolution layer and a dilated convolution layer;

[0019] The state matrix is ​​input into the time encoding layer of the pre-trained trajectory prediction model, and the state matrix is ​​column-convolved to fuse the features of the same data dimension at different times to obtain the associated features of the obstacle in the time dimension, specifically including:

[0020] Input the state matrix into the standard convolution layer of the time encoding layer in the pre-trained trajectory prediction model, perform standard column convolution on the state matrix, fuse the features of the same data dimension at adjacent moments, and obtain the first sub-correlation feature of the obstacle in the time dimension;

[0021] The state matrix is ​​input into the dilated convolution layer of the time encoding layer in the pre-trained trajectory prediction model, and the state matrix is ​​subjected to dilated column convolution to fuse the features of the same data dimension at intervals, thereby obtaining the second sub-correlation features of the obstacle in the time dimension, where the intervals are separated by a preset time length;

[0022] Feature fusion is performed according to the first sub-association feature and the second sub-association feature to obtain the association feature of the obstacle in the time dimension.

[0023] Optionally, the spatial coding layer includes a standard convolution layer and a dilated convolution layer;

[0024] The state matrix is ​​input into the spatial encoding layer of the pre-trained trajectory prediction model, and row convolution is performed on the state matrix to fuse the features of different data dimensions at the same time, thereby obtaining the associated features of the obstacle in the spatial dimension, specifically including:

[0025] Input the state matrix into a standard convolution layer of a spatial encoding layer in a pre-trained trajectory prediction model, perform standard row convolution on the state matrix, fuse features of adjacent data dimensions at the same time, and obtain a first sub-correlation feature of the obstacle in the spatial dimension;

[0026] Input the state matrix into the dilated convolution layer of the spatial coding layer in the pre-trained trajectory prediction model, perform dilated row convolution on the state matrix, fuse the features of the interval data dimension at the same time, and obtain the second sub-correlation feature of the obstacle in the spatial dimension;

[0027] Feature fusion is performed according to the first sub-association feature and the second sub-association feature to obtain the association feature of the obstacle in the spatial dimension.

[0028] Optionally, the trajectory prediction model further includes a full coding layer;

[0029] According to the associated features of the obstacle in the time dimension and the space dimension respectively, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model, specifically including:

[0030] Inputting the state matrix into the full coding layer of the trajectory prediction model to determine the comprehensive correlation characteristics of the obstacle;

[0031] According to the association features of the obstacles in the time dimension, the association features in the space dimension and the comprehensive association features, feature fusion is performed to obtain fusion features;

[0032] The fused features are input into a decoding layer of the trajectory prediction model to determine a predicted driving trajectory of the obstacle.

[0033] Optionally, according to the associated features of the obstacle in the time dimension and the space dimension respectively, determining the predicted driving trajectory of the obstacle through the decoding layer of the trajectory prediction model specifically includes:

[0034] According to the associated features of the obstacles in the time dimension and the space dimension, feature fusion is performed to determine the fusion features;

[0035] Determining the number of encodings of the state matrix, and judging whether the number of encodings reaches a preset value;

[0036] If so, inputting the fused features into a decoding layer of the trajectory prediction model to determine the predicted driving trajectory of the obstacle;

[0037] If not, the fused features are re-used as a new state matrix, and the re-determined state matrix is ​​input into the time coding layer, column convolution is performed on the state matrix to obtain the associated features of the obstacle in the time dimension, and input into the space coding layer, row convolution is performed on the state matrix, and the associated features of the obstacle in the space dimension are determined. Feature fusion is performed according to the associated features of the obstacle in the time dimension and space dimension, respectively, to determine the fused features, until the number of encoding times reaches a preset value.

[0038] Optionally, train a trajectory prediction model, specifically including:

[0039] Acquire travel data of a plurality of obstacles within a first time period and a second time period in history, wherein the time corresponding to the first time period is earlier than the time corresponding to the second time period;

[0040] For each obstacle, state data of the obstacle at several different times within the first time period is used as row data, and data of different data dimensions of the state data is used as column data to construct a state matrix as sample data of training samples, and the training samples are labeled according to the driving data of the obstacle within the second time period;

[0041] For each training sample, the sample data of the training sample is input into the time coding layer of the trajectory prediction model to be trained, and the sample data is column-convolved to fuse the features of the same data dimension at different times, so as to obtain the associated features of the training sample in the time dimension; and the sample data of the training sample is input into the space coding layer of the trajectory prediction model, and the sample data is row-convolved to fuse the features of different data dimensions at the same time, so as to obtain the associated features of the training sample in the space dimension;

[0042] According to the associated features of the training sample in the time dimension and the space dimension respectively, determining the predicted driving trajectory of the training sample within the second time length through the decoding layer of the trajectory prediction model;

[0043] The model parameters in the trajectory prediction model are adjusted with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the label of each training sample. The trajectory prediction model is used to predict the driving trajectory of the obstacle.

[0044] Optionally, with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the annotation of each training sample, adjusting the model parameters in the trajectory prediction model specifically includes:

[0045] For each training sample, determine the key position points corresponding to each key moment in the predicted driving trajectory of the training sample;

[0046] According to each key position point of the training sample, a fitting trajectory of the training sample is connected;

[0047] The model parameters in the trajectory prediction model are adjusted with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the annotation of each training sample, and the difference between the fitted trajectory of each training sample and the annotation of each training sample.

[0048] This specification provides an obstacle trajectory prediction device, including:

[0049] An acquisition module, configured to acquire historical travel data of the obstacle within a first time period, wherein the travel data includes state data at several moments within the first time period;

[0050] A construction module is configured to construct a state matrix using state data at different times as row data and data of different data dimensions of the state data as column data;

[0051] A time association module is configured to input the state matrix into a time encoding layer of a pre-trained trajectory prediction model, perform column convolution on the state matrix, fuse features of the same data dimension at different times, and obtain association features of the obstacle in the time dimension;

[0052] A spatial association module is configured to input the state matrix into a spatial encoding layer of a pre-trained trajectory prediction model, perform row convolution on the state matrix, fuse features of different data dimensions at the same time, and obtain association features of the obstacle in the spatial dimension;

[0053] The prediction module is configured to determine the predicted driving trajectory of the obstacle through the decoding layer of the trajectory prediction model according to the associated features of the obstacle in the time dimension and the space dimension respectively.

[0054] This specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above obstacle trajectory prediction method is implemented.

[0055] The present specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above-mentioned obstacle trajectory prediction method is implemented when the processor executes the program.

[0056] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0057] In this specification, the driving data of the obstacle in the first time period in history can be obtained, and the state data at different times are used as row data, and the data of different data dimensions are used as column data to construct a state matrix. Afterwards, the state matrix is ​​respectively input into the time coding layer and the space coding layer of the trajectory prediction model to obtain the correlation features in the time dimension and the correlation features in the space dimension. Finally, according to the correlation features of the obstacle in the time dimension and the space dimension, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model. The historical driving data of the obstacle is encoded from the time dimension and the space dimension respectively, and the correlation between the various features of the obstacle in the time dimension and the space dimension is learned, thereby improving the accuracy of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0059] Figure 1 A flowchart of an obstacle trajectory prediction method is provided for an embodiment of this specification;

[0060] Figure 2 A network structure diagram of a trajectory prediction model provided in an embodiment of this specification;

[0061] Figure 3 A network structure diagram of a time coding layer provided in an embodiment of this specification;

[0062] Figure 4 A network structure diagram of a spatial coding layer provided in an embodiment of this specification;

[0063] Figure 5 A network structure diagram of a trajectory prediction model provided in an embodiment of this specification;

[0064] Figure 6 A schematic diagram of the structure of an obstacle trajectory prediction device provided in an embodiment of this specification;

[0065] Figure 7 Schematic diagram of an electronic device for implementing an obstacle trajectory prediction method provided in an embodiment of this specification. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] This specification provides an obstacle trajectory prediction method. The following is a detailed description of the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0068] Figure 1 A flowchart of an obstacle trajectory prediction method provided in an embodiment of this specification may include the following steps:

[0069] S100: Obtaining the driving data of the obstacle within the first time period in history.

[0070] S102: constructing a state matrix by using state data at different times as row data and data of different data dimensions of the state data as column data.

[0071] When planning the driving trajectory of an unmanned driving device, it is often necessary to predict the driving trajectory of surrounding obstacles so that the unmanned driving device can avoid various obstacles during driving. Therefore, this specification provides an obstacle trajectory prediction method, which can extract effective correlation features based on the historical behavior trajectory of the obstacle, and then predict the future driving trajectory.

[0072] The obstacle trajectory prediction method can be executed by a backend server or by an unmanned driving device as an execution subject. This specification does not limit this and can be set as needed. For the convenience of description, the following description will take server execution as an example. The obstacle is a dynamic obstacle that can change its position, which can be a vehicle, pedestrian, etc.

[0073] Specifically, the server may first obtain the travel data of the obstacle in the first time period in history. The travel data of the obstacle includes the state data of the obstacle at several moments in the first time period, and may also include the travel track, size (length, width, etc.) of the obstacle. The state data includes data of several different data dimensions, such as position coordinates, travel direction, speed, acceleration, etc.

[0074] In order to make the trajectory prediction effect more accurate, the first time length can select the most recent time range, such as the driving data 5 minutes ago, to predict the driving trajectory for the next period of time.

[0075] It should be noted that the historical driving data of obstacles obtained in this specification can be obtained by detecting dynamic obstacles in the surrounding environment through the camera, laser radar and other sensor devices configured by the vehicle during the historical driving process. Based on the collected environmental images or laser point clouds of each frame, the driving data of each dynamic obstacle in each frame is determined.

[0076] Afterwards, the server can stitch the collected multiple frames of data into a two-dimensional image format, using the state data of the obstacle at different times as row data and data of different data dimensions of the state data as column data to construct a state matrix.

[0077] For example, assuming that the obstacle is a pedestrian, the first time duration includes T1, T2…Tn. Based on the pedestrian's driving data at time T1, the state data N1 of the pedestrian corresponding to time T1 can be determined. The state data N1 includes k data dimensions such as the position coordinates, speed, acceleration, size, and driving direction of the pedestrian at time T1. Afterwards, the state data at n moments are used as row data, and the data of different dimensions of the state data are used as column data, which are spliced ​​into an n×k state matrix.

[0078] S104: Input the state matrix into the time encoding layer of the pre-trained trajectory prediction model, perform column convolution on the state matrix, fuse the features of the same data dimension at different times, and obtain the associated features of the obstacle in the time dimension.

[0079] S106: Input the state matrix into the spatial encoding layer of the pre-trained trajectory prediction model, perform row convolution on the state matrix, fuse the features of different data dimensions at the same time, and obtain the associated features of the obstacle in the spatial dimension.

[0080] The existing technology uses LSTM to encode historical driving data, which only considers the correlation between features at different times from the time dimension. However, not every feature of each data dimension has the same impact on the trajectory prediction result, and the correlation between each data dimension also varies.

[0081] Therefore, in this specification, the correlation between the data dimensions is called the correlation in the spatial dimension, that is, the correlation between the features of different data dimensions at the same time. When predicting the obstacle trajectory in this specification, the correlation of historical driving data can be learned from the two dimensions of time and space to obtain accurate trajectory prediction results.

[0082] Specifically, the constructed state matrix is ​​input into the time encoding layer of the pre-trained trajectory prediction model, and the state matrix is ​​column-convolved to fuse the features of the same data dimension at different times, so as to obtain the associated features of the obstacle in the time dimension.

[0083] And the state matrix is ​​input into the spatial encoding layer of the pre-trained trajectory prediction model, and row convolution is performed on the state matrix to fuse the features of different data dimensions at the same time, so as to obtain the associated features of the obstacle in the spatial dimension.

[0084] Figure 2 A network structure diagram of a trajectory prediction model provided in an embodiment of this specification, the trajectory prediction model includes a time coding layer, a space coding layer and a decoding layer. The state matrix corresponding to the obstacle can be input into the time coding layer to obtain the associated features of the obstacle in the time dimension, and input into the space coding layer to obtain the associated features of the obstacle in the space dimension.

[0085] Among them, in the trajectory prediction model shown in the embodiment of this specification, the time coding layer and the space coding layer are parallel network structures, and the state matrix corresponding to the obstacle can be input into the time coding layer and the space coding layer respectively at the same time. This specification does not limit the order of inputting the state matrix.

[0086] Furthermore, in the time coding layer of the trajectory prediction model, the time coding layer can be composed of a standard convolution layer. Since a row of data in the state matrix represents the state data at the same time, and a column of data represents the data of the same data dimension at different times. Therefore, when extracting the correlation between the features in the time dimension, different rows of data can be fused. The state matrix corresponding to the obstacle is input into the standard convolution layer of the time coding layer, and a standard column convolution is performed through a one-dimensional convolution kernel to fuse the features of the same data dimension at adjacent times to obtain the first sub-correlation feature of the training sample in the time dimension. Among them, the one-dimensional convolution kernel can be 3×1, 5×1, etc.

[0087] Furthermore, in order to fuse the feature information of other similar moments, the time coding layer may also include a dilated convolution layer. When extracting the correlation between the features in the time dimension, the state matrix corresponding to the obstacle can be input into the dilated convolution layer of the time coding layer, and a dilated column convolution is performed through one-dimensional convolution to fuse the features of the same data dimension at the interval moments, and obtain the second sub-correlation feature of the training sample in the time dimension. Among them, the dilated convolution kernel can be selected as 3×1, 5×1, etc., and the preset time length between the interval moments is related to the expansion rate of the dilated convolution layer. The expansion rate of the dilated convolution layer can be set as needed, and this manual does not limit this. For example, the expansion rate can be set to 2 to fuse the features of similar moments with a spacing of one moment.

[0088] Furthermore, in order to fuse the features between moments of different spacings, in this specification, a variety of different convolution kernels can be used to perform feature fusion from different spacing dimensions such as close spacing moments, medium spacing moments, and long spacing moments.

[0089] In one embodiment of the present specification, the time coding layer can be set with three convolution layers, which are respectively used to fuse the features between the close-distance moments, the medium-distance moments, and the long-distance moments. Among them, the close-distance moments are standard convolution layers, the medium-distance moments and the long-distance moments are hole convolution layers with different convolution kernel sizes, and the convolution kernel size of the long-distance moment is larger than the convolution kernel size of the medium-distance moment. The state matrix corresponding to the obstacle is input into the convolution layer corresponding to the close-distance moment to obtain the sub-correlation features of the close-distance moment, input into the convolution layer corresponding to the medium-distance moment to obtain the sub-correlation features of the medium-distance moment, and input into the convolution layer corresponding to the long-distance moment to obtain the sub-correlation features of the long-distance moment. Afterwards, the sub-correlation features of the close-distance moment, the medium-distance moment, and the long-distance moment are input into the feature fusion layer to obtain the correlation features of the training sample in the time dimension.

[0090] Assume that the convolution layer corresponding to the close spacing moment is a 3×1 standard convolution layer, the convolution layer corresponding to the medium spacing moment is a 3×1 dilated convolution layer, the convolution layer corresponding to the long spacing moment is a 5×1 dilated convolution layer, and the feature fusion layer is a 1×1 standard convolution layer. Figure 3 As shown in the figure, the state matrix corresponding to the obstacle can be input into the 3×1 standard convolution layer, the 3×1 dilated convolution layer, and the 5×1 dilated convolution layer to obtain the sub-correlation features at the close-distance moment, the medium-distance moment, and the far-distance moment. After that, each sub-correlation feature is input into the 1×1 standard convolution layer to obtain the fused correlation features in the time dimension.

[0091] Similarly, in the spatial coding layer of the trajectory prediction model, the spatial coding layer can be composed of a standard convolution layer. Since a row in the state matrix represents the state data at the same time, and a column represents the data of the same data dimension at different times. Therefore, when extracting the correlation between the features in the spatial dimension, the data in different columns can be fused. The state matrix corresponding to the obstacle is input into the standard convolution layer of the spatial coding layer, and a standard row convolution is performed through a one-dimensional convolution kernel to fuse the features of adjacent data dimensions at the same time, and obtain the first sub-correlation feature of the obstacle in the spatial dimension. Among them, the one-dimensional convolution kernel can be 1×3, 1×5, etc.

[0092] Furthermore, in order to fuse the features between other data dimensions, the spatial coding layer may also include a dilated convolution layer. When extracting the correlation between the features in the spatial dimension, the state matrix corresponding to the obstacle may be input into the dilated convolution layer of the spatial coding layer, and a dilated row convolution may be performed through a one-dimensional convolution kernel to fuse the features of the interval data dimension at the same time, thereby obtaining the second sub-correlation feature of the obstacle in the spatial dimension. The dilated convolution kernel may be selected as 1×3, 1×5, etc., and the interval distance of the interval data dimension is related to the expansion rate of the dilated convolution layer.

[0093] Furthermore, in order to associate the relationship between features of different data dimensions, in this specification, a variety of different convolution kernels can be used to perform feature fusion from the close-distance data dimension, the medium-distance data dimension, the long-distance data dimension, etc.

[0094] Assume that the convolution layer corresponding to the close-spacing data dimension is a 1×3 standard convolution layer, the convolution layer corresponding to the medium-spacing data dimension is a 1×3 dilated convolution layer, the convolution layer corresponding to the long-spacing data dimension is a 1×5 dilated convolution layer, and the feature fusion layer is a 1×1 standard convolution layer. Figure 4 As shown in the figure, the state matrix corresponding to the obstacle can be input into the 1×3 standard convolution layer, the 1×3 dilated convolution layer, and the 1×5 dilated convolution layer to obtain the sub-correlation features of the close-spacing data dimension, the medium-spacing data dimension, and the far-spacing data dimension. After that, each sub-correlation feature is input into the 1×1 standard convolution layer to obtain the correlation features in the fused spatial dimension.

[0095] In addition, before the state matrix is ​​input into the time coding layer and the space coding layer, in order to improve the trajectory prediction effect, the attention mechanism can be used to enhance the part of the state matrix that needs attention. Specifically, the state data of the obstacle corresponding to each moment in the first time length can be determined from the state matrix. Afterwards, for each moment in the first time length, the association matrix corresponding to the moment is determined according to the state data of the obstacle corresponding to the moment and the state data of the obstacle corresponding to other moments. Among them, the association matrix represents the correlation between the state data of other moments and the state data of the moment. Finally, according to the association matrix corresponding to each moment in the first time length, the attention weight matrix is ​​determined, and according to the attention weight matrix, the state matrix is ​​weighted to determine the weighted state matrix.

[0096] Assuming that the first time duration includes T1, T2…Tn, the state data of the obstacle corresponding to each moment is Nn, and each state data includes k data dimensions such as position coordinates, speed, acceleration, size, and driving direction. Then the state matrix spliced ​​by the state data at each moment can be expressed as Each row represents the state data at the corresponding time Tn. Then, the attention weight matrix Q is determined according to the state data of the obstacle at each time in the state matrix. Among them:

[0097]

[0098] The correlation matrix corresponding to each moment is: n =[NnN1 NnN2 … NnNn]

[0099] Then, each row of data in the attention weight matrix Q is normalized by the normalized exponential function softmax, and the normalized result is multiplied by the original state matrix to obtain the weighted state matrix.

[0100] S108: Determine the predicted driving trajectory of the obstacle through the decoding layer of the trajectory prediction model according to the associated features of the obstacle in the time dimension and the space dimension respectively.

[0101] After encoding the state matrix from the time dimension and the space dimension respectively, trajectory prediction can be performed based on the extracted correlation features of the time dimension and the correlation features of the space dimension.

[0102] Specifically, the associated features of the obstacle in the time dimension are fused with the associated features of the obstacle in the space dimension to obtain fused features. Figure 2 As shown, the fusion feature is input into the decoding layer of the trajectory prediction model to determine the predicted driving trajectory of the obstacle in the second time period. The decoding layer is composed of several fully connected layers (FC). The predicted driving trajectory includes the position points of the obstacle at each moment in the second time period.

[0103] Furthermore, in order to fully integrate local features and reduce the amount of data calculation, when performing feature fusion, a dilated convolution layer can be used to first fuse local features of the associated features of the time dimension and the space dimension, and at the same time achieve the effect of dimensionality reduction. After that, the features after dimensionality reduction are improved through a standard convolution layer to obtain fused features. Among them, the dilated convolution layer can use a 3×3 dilated convolution kernel, and the standard convolution layer can use a 1×1 standard convolution kernel.

[0104] based on Figure 1 The obstacle trajectory prediction method shown can obtain the driving data of the obstacle in the first time period in history, and use the state data at different times as row data and the data of different data dimensions as column data to construct a state matrix. After that, the state matrix is ​​input into the time encoding layer and the space encoding layer of the trajectory prediction model respectively to obtain the correlation features in the time dimension and the correlation features in the space dimension. Finally, according to the correlation features of the obstacle in the time dimension and the space dimension respectively, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model. The historical driving data of the obstacle is encoded from the time dimension and the space dimension respectively, and the correlation between the various features of the obstacle in the time dimension and the space dimension is learned, thereby improving the accuracy of trajectory prediction.

[0105] In one embodiment of the present specification, in order to fully retain the characteristic information of historical driving data, the trajectory prediction model is also provided with a full coding layer, and the state matrix of the obstacle can be input into the full coding layer, and the features between different data dimensions at different times are associated to obtain the comprehensive correlation features of the obstacle. Among them, the full coding layer can be a standard convolution layer, and the convolution kernel can be set to 1×1. Afterwards, according to the correlation features of the training sample in the time dimension, the correlation features in the space dimension, and the comprehensive correlation features, feature fusion is performed to obtain fused features. Finally, the fused features are input into the decoding layer of the trajectory prediction model to determine the predicted driving trajectory of the obstacle within the second time length.

[0106] Figure 5 This is a network structure diagram of a trajectory prediction model provided in an embodiment of this specification. The trajectory prediction model consists of an attention network, a coding layer, and a decoding layer. After the state matrix of the obstacle is input into the trajectory prediction model, the attention of the obstacle can be weighted through the attention network to obtain a weighted state matrix, and then the weighted state matrix is ​​respectively input into the time coding layer, the full coding layer, and the space coding layer to obtain the association features, comprehensive association features, and association features of the obstacle in the time dimension. Then, through the feature fusion layer, feature fusion is performed to obtain fused features. Finally, the fused features are input into the decoding layer to obtain the predicted driving trajectory within the second time length.

[0107] In one embodiment of the present specification, in order to improve the fusion effect of features, multiple iterative encodings can be performed. After the feature fusion is performed according to the associated features of the obstacle in the time dimension and the space dimension, and the fusion features are determined, the number of encodings of the state matrix can be determined, and it can be judged whether the number of encodings reaches the preset value. Among them, the number of encodings refers to the number of times the state matrix obtains the fusion features through the encoding layer. If the number of encodings reaches the preset value, the fusion features are input into the decoding layer to determine the predicted driving trajectory of the obstacle in the second time length. If the number of encodings does not reach the preset value, the fusion features are re-used as the state matrix corresponding to the obstacle, and the re-determined state matrix is ​​encoded again through the methods described in the above steps S104 and S106, and the outputted associated features of the obstacle in the time dimension and the space dimension are subjected to feature fusion to determine the fusion features until the number of encodings reaches the preset value.

[0108] In addition, compared with the existing method of using LSTM time series model for trajectory prediction, it is necessary to perform serial calculations on the position points at each moment, which seriously affects the overall timeliness of the model. In the process of unmanned driving, long calculation delays may cause serious safety problems. The trajectory prediction model in this application uses an efficient convolutional network, which has faster calculation efficiency and shorter trajectory prediction time, greatly improving the timeliness of trajectory prediction.

[0109] In this specification, when training the trajectory prediction model, the following training methods can be used:

[0110] A0: Obtain the driving data of several obstacles in the first time period and the second time period in history.

[0111] A2: For each obstacle, state data of the obstacle at several different times within the first time length is used as row data, and data of different data dimensions of the state data is used as column data to construct a state matrix as sample data of training samples, and the training samples are labeled according to the driving data of the obstacle within the second time length.

[0112] When training the trajectory prediction model, a large amount of historical driving data of obstacles within a first time period and driving data within a second time period can be obtained. The time corresponding to the first time period is earlier than the time corresponding to the second time period. The first time period and the second time period can be set as needed, such as 5 minutes, 30 seconds, etc.

[0113] Afterwards, for each obstacle, the state data of the obstacle at several different moments in the first time period is used as row data, and the data of different data dimensions of the state data is used as column data to construct a state matrix as sample data of the training sample.

[0114] Since the time period corresponding to the second time period is a future time period relative to the time period of the first time period, the training sample can be labeled according to the driving data of the obstacle in the second time period. When labeling the sample, the actual driving trajectory of the obstacle in the second time period and the position points corresponding to each moment on the actual driving trajectory can be labeled according to the driving data of the obstacle in the second time period.

[0115] A4: For each training sample, the sample data of the training sample is input into the time coding layer of the trajectory prediction model to be trained, and the sample data is column-convolved to fuse the features of the same data dimension at different times to obtain the associated features of the training sample in the time dimension, and the sample data of the training sample is input into the space coding layer of the trajectory prediction model, and the sample data is row-convolved to fuse the features of different data dimensions at the same time to obtain the associated features of the training sample in the space dimension.

[0116] A6: Determine the predicted driving trajectory of the training sample within the second time period through the decoding layer of the trajectory prediction model according to the associated features of the training sample in the time dimension and the space dimension respectively.

[0117] Among them, the trajectory prediction model to be trained can be as follows Figure 2 The network structure shown can also be Figure 5 The network structure shown, the feature extraction through the time coding layer and the space coding layer has been described in detail in the above steps S104 to S106, and will not be repeated in this specification.

[0118] A8: With the goal of minimizing the difference between the predicted driving trajectory of each training sample and the label of each training sample, adjust the model parameters in the trajectory prediction model, wherein the trajectory prediction model is used to predict the driving trajectory of the obstacle.

[0119] During the model training process, the model parameters can be adjusted based on the difference between the predicted driving trajectory output by the trajectory prediction model to be trained and the sample annotation.

[0120] Therefore, the average displacement error (ADE) between the predicted driving trajectory of each training sample and the actual driving trajectory marked by each training sample can be determined, and the model parameters in the trajectory prediction model can be adjusted with the goal of minimizing the ADE between the predicted driving trajectory and the actual driving trajectory. The trajectory prediction model can be used to predict the driving trajectory of the obstacle and plan the driving path of the unmanned equipment in a timely and reasonable manner.

[0121] Furthermore, the average displacement error is calculated based on each position point on the predicted driving trajectory, which indicates that the weight of each position point on the error is consistent. In the actual driving process, the starting point, the end point and the position points in the middle often have a greater impact on the trajectory deviation.

[0122] Therefore, for each training sample, the key position points corresponding to each key moment in the predicted driving trajectory of the training sample can be determined. Afterwards, the fitting trajectory of the training sample is connected according to the key position points of the training sample. Finally, according to the ADE between the predicted driving trajectory of each training sample and the actual driving trajectory marked by each training sample, and the ADE between the actual driving trajectory marked by each training sample and the fitting trajectory of each training sample, the loss function is determined, and the model parameters in the trajectory prediction model are adjusted with the goal of minimizing the loss function. Among them, the key moment can be set as needed, such as the starting moment, the end moment and the middle moment of the second duration, and the key position point is the position point where the obstacle is located at the key moment. For example, if the second duration is set to 5s, the key moment can be selected from the moments of 1s, 3s and 5s.

[0123] It should be noted that the reason why the difference between the actual driving trajectory and the fitted trajectory is fitted, rather than the difference between the key position points, is that the constraint of a single position point on the trajectory prediction is not strong, and the fitted trajectory includes more position point information, which brings stronger constraints. In addition, using the fitted estimate for constraint can ensure the smoothness of the predicted driving trajectory output in the end, and effectively avoid the problem of too many abnormal points in the trajectory.

[0124] based on Figure 1 The obstacle trajectory prediction method shown in the figure, the embodiment of this specification also provides a structural schematic diagram of an obstacle trajectory prediction device, such as Figure 6 shown.

[0125] Figure 6 A schematic diagram of the structure of an obstacle trajectory prediction device provided in an embodiment of this specification includes:

[0126] An acquisition module 200 is configured to acquire the travel data of the obstacle within a first time period in history, wherein the travel data includes state data at several moments within the first time period;

[0127] A construction module 202 is configured to construct a state matrix using state data at different times as row data and data of different data dimensions of the state data as column data;

[0128] The time association module 204 is configured to input the state matrix into the time encoding layer of the pre-trained trajectory prediction model, perform column convolution on the state matrix, fuse the features of the same data dimension at different times, and obtain the associated features of the obstacle in the time dimension;

[0129] The spatial association module 206 is configured to input the state matrix into the spatial encoding layer of the pre-trained trajectory prediction model, perform row convolution on the state matrix, fuse the features of different data dimensions at the same time, and obtain the association features of the obstacle in the spatial dimension;

[0130] The prediction module 208 is configured to determine the predicted driving trajectory of the obstacle through the decoding layer of the trajectory prediction model according to the associated features of the obstacle in the time dimension and the space dimension respectively.

[0131] Optionally, the construction module 202 is also used to determine, for each moment within the first time length, a correlation matrix corresponding to the moment based on the state data of the obstacle corresponding to the moment and the state data of the obstacle corresponding to other moments, the correlation matrix representing the correlation between the state data of other moments and the state data of the moment, determine an attention weight matrix based on the correlation matrices corresponding to each moment within the first time length, perform attention weighting on the state matrix based on the attention weight matrix, and determine a weighted state matrix.

[0132] Optionally, the time coding layer includes a standard convolution layer and a hole convolution layer, and the time association module 204 is specifically used to input the state matrix into the standard convolution layer of the time coding layer in the pre-trained trajectory prediction model, perform standard column convolution on the state matrix, fuse the features of the same data dimension at adjacent moments, and obtain the first sub-association feature of the obstacle in the time dimension, input the state matrix into the hole convolution layer of the time coding layer in the pre-trained trajectory prediction model, perform hole column convolution on the state matrix, fuse the features of the same data dimension at interval moments, and obtain the second sub-association feature of the obstacle in the time dimension, the interval moments are separated by a preset time length, and feature fusion is performed according to the first sub-association feature and the second sub-association feature to obtain the association feature of the obstacle in the time dimension.

[0133] Optionally, the spatial coding layer includes a standard convolution layer and a hole convolution layer, and the spatial association module 206 is specifically used to input the state matrix into the standard convolution layer of the spatial coding layer in the pre-trained trajectory prediction model, perform standard row convolution on the state matrix, fuse the features of adjacent data dimensions at the same time, and obtain the first sub-association feature of the obstacle in the spatial dimension, input the state matrix into the hole convolution layer of the spatial coding layer in the pre-trained trajectory prediction model, perform hole row convolution on the state matrix, fuse the features of interval data dimensions at the same time, and obtain the second sub-association feature of the obstacle in the spatial dimension, and perform feature fusion according to the first sub-association feature and the second sub-association feature to obtain the association feature of the obstacle in the spatial dimension.

[0134] Optionally, the trajectory prediction model also includes a full coding layer, and the prediction module 208 is specifically used to input the state matrix into the full coding layer of the trajectory prediction model, determine the comprehensive correlation features of the obstacle, perform feature fusion according to the association features of the obstacle in the time dimension, the association features in the space dimension and the comprehensive association features, obtain fused features, input the fused features into the decoding layer of the trajectory prediction model, and determine the predicted driving trajectory of the obstacle.

[0135] Optionally, the prediction module 208 is specifically used to perform feature fusion according to the associated features of the obstacle in the time dimension and the space dimension, determine the fused features, determine the number of encodings of the state matrix, and judge whether the encoding number reaches a preset value. If so, input the fused features into the decoding layer of the trajectory prediction model to determine the predicted driving trajectory of the obstacle. If not, re-use the fused features as a new state matrix, input the re-determined state matrix into the time encoding layer, perform column convolution on the state matrix, obtain the associated features of the obstacle in the time dimension, and input the state matrix into the space encoding layer, perform row convolution on the state matrix, determine the associated features of the obstacle in the space dimension, and perform feature fusion according to the associated features of the obstacle in the time dimension and the space dimension, determine the fused features, until the encoding number reaches the preset value.

[0136] Optionally, the obstacle trajectory prediction device further includes a model training module 210, wherein the model training module 210 is specifically used to obtain the driving data of several obstacles in the first time period and the second time period in history, wherein the time corresponding to the first time period is earlier than the time corresponding to the second time period, and for each obstacle, the state data of the obstacle at several different times in the first time period is used as row data, and the data of different data dimensions of the state data is used as column data to construct a state matrix as sample data of the training sample, and the training sample is labeled according to the driving data of the obstacle in the second time period, and for each training sample, the sample data of the training sample is input into the time coding layer of the trajectory prediction model to be trained, and the sample data is subjected to the training. Row-column convolution is used to fuse the features of the same data dimension at different times to obtain the associated features of the training sample in the time dimension, and the sample data of the training sample is input into the spatial encoding layer of the trajectory prediction model, and row convolution is performed on the sample data to fuse the features of different data dimensions at the same time to obtain the associated features of the training sample in the spatial dimension. According to the associated features of the training sample in the time dimension and the spatial dimension respectively, the predicted driving trajectory of the training sample within the second time length is determined through the decoding layer of the trajectory prediction model, and the model parameters in the trajectory prediction model are adjusted with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the annotation of each training sample. The trajectory prediction model is used to predict the driving trajectory of the obstacle.

[0137] Optionally, the model training module 210 is specifically used to determine, for each training sample, the key position points corresponding to each key moment in the predicted driving trajectory of the training sample, and connect them into a fitting trajectory of the training sample based on the key position points of the training sample, so as to adjust the model parameters in the trajectory prediction model with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the annotation of each training sample, as well as the difference between the fitting trajectory of each training sample and the annotation of each training sample.

[0138] The present specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The obstacle trajectory prediction method provided.

[0139] according to Figure 1 The obstacle trajectory prediction method shown in the present specification also proposes Figure 7 The schematic structure diagram of the electronic device shown in FIG. Figure 7At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The obstacle trajectory prediction method shown.

[0140] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0141] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many improvements to method flows today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and generate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually generating integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0142] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0143] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0144] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0150] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0151] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0153] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0155] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0156] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for predicting obstacle trajectories, characterized in that: include: Acquire the travel data of the obstacle within a first time period in history, wherein the travel data includes state data at several moments within the first time period; Using state data at different times as row data and data of different data dimensions of the state data as column data to construct a state matrix, wherein the state data includes data of several different data dimensions, including at least position coordinates, driving direction, speed, and acceleration; Input the state matrix into the time encoding layer of the pre-trained trajectory prediction model, perform column convolution on the state matrix, fuse the features of the same data dimension at different times, and obtain the associated features of the obstacle in the time dimension; Input the state matrix into the spatial encoding layer of the pre-trained trajectory prediction model, perform row convolution on the state matrix, fuse the features of different data dimensions at the same time, and obtain the associated features of the obstacle in the spatial dimension; According to the associated features of the obstacle in the time dimension and the space dimension respectively, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model.

2. The method according to claim 1, characterized in that Before inputting the state matrix into a temporal coding layer and / or a spatial coding layer of a pre-trained trajectory prediction model, the method further comprises: For each moment in the first time period, determining a correlation matrix corresponding to the moment according to the state data of the obstacle corresponding to the moment and the state data of the obstacle corresponding to each other moment, wherein the correlation matrix represents the correlation between the state data of each other moment and the state data of the moment; Determine an attention weight matrix according to the correlation matrix corresponding to each moment in the first duration; The state matrix is ​​attention-weighted according to the attention weight matrix to determine a weighted state matrix.

3. The method according to claim 1, characterized in that The time coding layer includes a standard convolution layer and a hole convolution layer; the state matrix is ​​input into the time coding layer of the pre-trained trajectory prediction model, and the state matrix is ​​column-convolved to fuse the features of the same data dimension at different times to obtain the associated features of the obstacle in the time dimension, which specifically includes: Input the state matrix into the standard convolution layer of the time encoding layer in the pre-trained trajectory prediction model, perform standard column convolution on the state matrix, fuse the features of the same data dimension at adjacent moments, and obtain the first sub-correlation feature of the obstacle in the time dimension; The state matrix is ​​input into the dilated convolution layer of the time encoding layer in the pre-trained trajectory prediction model, and the state matrix is ​​subjected to dilated column convolution to fuse the features of the same data dimension at intervals, thereby obtaining the second sub-correlation features of the obstacle in the time dimension, where the intervals are separated by a preset time length; Feature fusion is performed according to the first sub-association feature and the second sub-association feature to obtain the association feature of the obstacle in the time dimension.

4. The method according to claim 1, characterized in that The spatial coding layer includes a standard convolution layer and a hole convolution layer; the state matrix is ​​input into the spatial coding layer of the pre-trained trajectory prediction model, and row convolution is performed on the state matrix to fuse the features of different data dimensions at the same time, so as to obtain the associated features of the obstacle in the spatial dimension, specifically including: Input the state matrix into a standard convolution layer of a spatial encoding layer in a pre-trained trajectory prediction model, perform standard row convolution on the state matrix, fuse features of adjacent data dimensions at the same time, and obtain a first sub-correlation feature of the obstacle in the spatial dimension; Input the state matrix into the dilated convolution layer of the spatial coding layer in the pre-trained trajectory prediction model, perform dilated row convolution on the state matrix, fuse the features of the interval data dimension at the same time, and obtain the second sub-correlation feature of the obstacle in the spatial dimension; Feature fusion is performed according to the first sub-association feature and the second sub-association feature to obtain the association feature of the obstacle in the spatial dimension.

5. The method according to claim 1, characterized in that The trajectory prediction model also includes a full coding layer; According to the associated features of the obstacle in the time dimension and the space dimension respectively, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model, specifically including: Inputting the state matrix into the full coding layer of the trajectory prediction model to determine the comprehensive correlation characteristics of the obstacle; According to the association features of the obstacles in the time dimension, the association features in the space dimension and the comprehensive association features, feature fusion is performed to obtain fusion features; The fused features are input into a decoding layer of the trajectory prediction model to determine a predicted driving trajectory of the obstacle.

6. The method according to claim 1, characterized in that According to the associated features of the obstacle in the time dimension and the space dimension respectively, the predicted driving trajectory of the obstacle is determined through the decoding layer of the trajectory prediction model, specifically including: According to the associated features of the obstacles in the time dimension and the space dimension, feature fusion is performed to determine the fusion features; the number of encoding times of the state matrix is ​​determined, and it is judged whether the number of encoding times reaches a preset value; If so, the fused features are input into the decoding layer of the trajectory prediction model to determine the predicted driving trajectory of the obstacle; if not, the fused features are re-used as a new state matrix, and the re-determined state matrix is ​​input into the time encoding layer, column convolution is performed on the state matrix, and the associated features of the obstacle in the time dimension are obtained, and the state matrix is ​​input into the space encoding layer, row convolution is performed on the state matrix, and the associated features of the obstacle in the space dimension are determined, and feature fusion is performed according to the associated features of the obstacle in the time dimension and space dimension, respectively, to determine the fused features, until the number of encoding times reaches a preset value.

7. The method according to claim 1, characterized in that Training trajectory prediction models, including: Acquire travel data of a plurality of obstacles within a first time period and a second time period in history, wherein the time corresponding to the first time period is earlier than the time corresponding to the second time period; For each obstacle, state data of the obstacle at several different times within the first time period is used as row data, and data of different data dimensions of the state data is used as column data to construct a state matrix as sample data of training samples, and the training samples are labeled according to the driving data of the obstacle within the second time period; For each training sample, the sample data of the training sample is input into the time coding layer of the trajectory prediction model to be trained, and the sample data is column-convolved to fuse the features of the same data dimension at different times, so as to obtain the associated features of the training sample in the time dimension; and the sample data of the training sample is input into the space coding layer of the trajectory prediction model, and the sample data is row-convolved to fuse the features of different data dimensions at the same time, so as to obtain the associated features of the training sample in the space dimension; According to the associated features of the training sample in the time dimension and the space dimension respectively, determining the predicted driving trajectory of the training sample within the second time length through the decoding layer of the trajectory prediction model; The model parameters in the trajectory prediction model are adjusted with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the label of each training sample. The trajectory prediction model is used to predict the driving trajectory of the obstacle.

8. The method according to claim 7, characterized in that With the goal of minimizing the difference between the predicted driving trajectory of each training sample and the annotation of each training sample, the model parameters in the trajectory prediction model are adjusted, specifically including: For each training sample, determine the key position points corresponding to each key moment in the predicted driving trajectory of the training sample; According to each key position point of the training sample, a fitting trajectory of the training sample is connected; The model parameters in the trajectory prediction model are adjusted with the goal of minimizing the difference between the predicted driving trajectory of each training sample and the annotation of each training sample, and the difference between the fitted trajectory of each training sample and the annotation of each training sample.

9. An obstacle trajectory prediction device, characterized in that: include: An acquisition module configured to acquire historical travel data of the obstacle within a first time period, wherein the travel data includes state data at several moments within the first time period; A construction module is configured to construct a state matrix using state data at different times as row data and data of different data dimensions of the state data as column data, wherein the state data includes data of several different data dimensions, including at least position coordinates, driving direction, speed, and acceleration; A time association module is configured to input the state matrix into a time encoding layer of a pre-trained trajectory prediction model, perform column convolution on the state matrix, fuse features of the same data dimension at different times, and obtain association features of the obstacle in the time dimension; A spatial association module is configured to input the state matrix into a spatial encoding layer of a pre-trained trajectory prediction model, perform row convolution on the state matrix, fuse features of different data dimensions at the same time, and obtain association features of the obstacle in the spatial dimension; The prediction module is configured to determine the predicted driving trajectory of the obstacle through the decoding layer of the trajectory prediction model according to the associated features of the obstacle in the time dimension and the space dimension respectively.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 8 is implemented.

Images