Obstacle trajectory prediction method, device, storage medium and processor

Abstract

The application discloses an obstacle trajectory prediction method and device, a storage medium and a processor. The method comprises the following steps: obtaining historical state information and environment information of a target obstacle, wherein the historical state information is used for representing the real behavior state of the target obstacle in a historical time period, and the environment information is used for representing the real environment condition of the target obstacle in the historical time period; predicting target state information of the target obstacle based on the historical state information and the environment information, wherein the target state information is used for representing the predicted behavior state of the target obstacle in a target time period; and generating a motion trajectory of the target obstacle by using the target state information. The application solves the technical problem of low prediction accuracy of a conventional vehicle technology field due to rule-based behavior and trajectory prediction for all sudden situation scenes.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle technology, and in particular to an obstacle trajectory prediction method, device, storage medium and processor. Background Art

[0002] In the field of intelligent vehicle technology, obstacle behavior and trajectory prediction mainly fall into two categories: rule-based and learning algorithm-based. Rule-based obstacle behavior and trajectory prediction involves establishing a behavioral rule library based on formal rules, traffic regulations, and driving common sense. Vehicle states are divided according to different scenarios, and obstacle behavior and trajectory are predicted by matching the vehicle states with the behavioral rule library. Learning algorithm-based obstacle behavior and trajectory prediction involves performing machine learning on different scenarios, establishing a data-driven behavioral rule library, and using machine learning algorithms to match different environmental information with the behavior of different traffic participants in different scenarios, ultimately outputting predicted behavior and trajectory.

[0003] Rule-based behavior and trajectory prediction can be used normally in most cases, but due to the vehicle state segmentation conditions, the vehicle behavior is inconsistent, and the trigger conditions of the behavior rule library are prone to overlap, causing system failure. Rule-based behavior and trajectory prediction also has the disadvantage of not being able to cover all emergency scenarios. Behavior and trajectory prediction based on deep learning has the advantage of deep scene traversal. For a specific segmented scenario, it is easier to cover all working conditions through a big data system. The network structure can simplify the scale of the decision-making algorithm and has self-learning performance. It can automatically extract environmental features and decision-making attributes, which facilitates system optimization and iteration. There is no need to traverse various working conditions. The model is improved through data training, and the model accuracy is improved as the data is complete.

[0004] In the field of traditional vehicle technology, due to the low prediction accuracy of rule-based behavior and trajectory prediction for all emergency scenarios, no effective solution has been proposed to the above problems. Summary of the Invention

[0005] Embodiments of the present invention provide an obstacle trajectory prediction method, device, storage medium, and processor to at least solve the technical problem in the field of traditional vehicle technology of low prediction accuracy for all emergency scenarios due to rule-based behavior and trajectory prediction.

[0006] According to an aspect of some embodiments of the present application, a method for predicting an obstacle trajectory is provided, comprising: obtaining historical state information of a target obstacle and environment information, wherein the historical state information is used to represent a real behavior state of the target obstacle in a historical time period, and the environment information is used to represent a real environment condition of the target obstacle in the historical time period; predicting target state information of the target obstacle based on the historical state information and the environment information, wherein the target state information is used to represent a predicted behavior state of the target obstacle in a target time period; and generating a motion trajectory of the target obstacle by using the target state information.

[0007] Optionally, predicting the target state information of the target obstacle based on the historical state information and the environment information comprises: analyzing the historical state information and the environment information by using a target network model to predict the target state information, wherein the target network model is trained by using a plurality of sets of data through machine learning, and each set of data in the plurality of sets of data comprises sample data and predicted state information.

[0008] Optionally, the target network model comprises a first model and a second model, and predicting the target state information by analyzing the historical state information and the environment information by using the target network model comprises: performing feature splicing processing on the historical state information and the environment information by using the first model to obtain an intermediate feature vector; and performing state prediction processing on the intermediate feature vector by using the second model to obtain the target state information.

[0009] Optionally, the method for predicting an obstacle trajectory further comprises: obtaining observation state information of the target obstacle, wherein the observation state information is used to represent a real behavior state of the target obstacle in the target time period; calculating a mean square error between the target state information and the observation state information; and adjusting a training parameter of the target network model according to the mean square error.

[0010] Optionally, the method for predicting an obstacle trajectory further comprises: the environment information comprises one or more of electronic map data, target vehicle data, target obstacle data, target scene data, and traffic light data.

[0011] Optionally, the method for predicting an obstacle trajectory further comprises: the sampling frequency of the historical state information and the environment information is the same.

[0012] According to an aspect of some embodiments of the present application, a method for predicting an obstacle trajectory is provided, comprising: obtaining historical state information of a target obstacle and environment information, wherein the historical state information is used to represent a real behavior state of the target obstacle in a historical time period, and the environment information is used to represent a real environment condition of the target obstacle in the historical time period; predicting target state information of the target obstacle based on the historical state information and the environment information, wherein the target state information is used to represent a predicted behavior state of the target obstacle in a target time period; and generating a motion trajectory of the target obstacle by using the target state information.

[0013] An acquisition module acquires historical state information and environmental information of a target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle within a historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle within a historical time period. A prediction module is used to predict the target state information of the target obstacle based on the historical state information and environmental information, wherein the target state information is used to represent the predicted behavior state of the target obstacle within a target time period. A generation module is used to generate the motion trajectory of the target obstacle using the target state information.

[0014] Optionally, the prediction module is also used to: analyze historical status information and environmental information through a target network model to predict the target status information, wherein the target network model is obtained through machine learning training using multiple sets of data, and each set of data in the multiple sets of data includes: sample data and predicted status information.

[0015] Optionally, the prediction module is further used to: use the first model to perform feature splicing processing on the historical state information and the environmental information to obtain an intermediate feature vector; and use the second model to perform state prediction processing on the intermediate feature vector to obtain target state information.

[0016] Optionally, the obstacle trajectory prediction device also includes: an adjustment module, used to obtain observation state information of the target obstacle, wherein the observation state information is used to represent the actual behavior state of the target obstacle within the target time period; calculate the mean square error between the target state information and the observation state information, and adjust the training parameters of the target network model according to the mean square error.

[0017] Optionally, the acquisition module is further used to acquire environmental information, which includes: one or more of electronic map data, target vehicle data, target obstacle data, target scene data and traffic light data.

[0018] Optionally, the acquisition module is further configured to acquire historical status information and environmental information, wherein the sampling frequencies of the historical status information and the environmental information are the same.

[0019] According to one embodiment of the present invention, a non-volatile storage medium is further provided, characterized in that a computer program is stored in the storage medium, wherein the computer program is configured to execute any of the above obstacle trajectory prediction methods when running.

[0020] According to one embodiment of the present invention, a processor is further provided, characterized in that the processor is used to run a program, wherein the program is configured to execute any of the above obstacle trajectory prediction methods during execution.

[0021] According to one embodiment of the present invention, an electronic device is further provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute any of the above methods for predicting obstacle trajectories.

[0022] In an embodiment of the present invention, historical state information and environmental information of a target obstacle are obtained, wherein the historical state information is used to represent the actual behavior state of the target obstacle within a historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle within the historical time period. Target state information of the target obstacle is then predicted based on the historical state information and environmental information, wherein the target state information represents the predicted behavior state of the target obstacle within a target time period. This achieves the purpose of generating the motion trajectory of the target obstacle using the target state information, thereby achieving the technical effect of improving the accuracy of target obstacle trajectory prediction. This solves the technical problem in the field of conventional vehicle technology of low prediction accuracy for all emergency scenarios due to rule-based behavior and trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a flowchart of an obstacle trajectory prediction method according to one embodiment of the present application;

[0025] Figure 2 Schematic diagram of the overall network structure of the obstacle behavior prediction and trajectory prediction algorithm according to one embodiment of the present application;

[0026] Figure 3 Schematic diagram of the overall network model of the obstacle behavior prediction and trajectory prediction algorithm according to one embodiment of the present application;

[0027] Figure 4 This is a schematic diagram of the overall network structure of a commonly used CNN network and SVM to implement an obstacle behavior prediction and trajectory prediction algorithm according to one embodiment of the present application;

[0028] Figure 5 Schematic diagram of the MSE error function in w space according to one embodiment of the present application;

[0029] Figure 6 It is a structural block diagram of an obstacle trajectory prediction device according to one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to an embodiment of the present invention, a method embodiment of an obstacle trajectory prediction method, apparatus, storage medium, and processor is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in an order different from that shown.

[0033] Figure 1 is a flow chart of an obstacle trajectory prediction method according to one embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0034] Step S11: Acquire historical state information and environmental information of the target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle in the historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle in the historical time period;

[0035] Step S12: predicting target state information of the target obstacle based on the historical state information and the environmental information, wherein the target state information is used to represent the predicted behavior state of the target obstacle within the target time period;

[0036] Step S13: Generate a motion trajectory of the target obstacle using the target state information.

[0037] In step S11, the historical state information of the target obstacle is a historical behavior data set of the target obstacle in the historical time period. For example, the historical behavior data set T pdst ={T1, T2, ... T t ,...,T n}, behavior states include turning, lane changing, cruising, emergency stop, etc., T t Represents the behavior data of the target obstacle at the tth time point in the historical time period.

[0038] Environmental information includes one or more of electronic map data, target vehicle data, target obstacle data, target scene data, and traffic light data. The environmental information of the target obstacle may include the coordinates of the vehicle detecting the target obstacle, the lane lines of the preset driving section, the obstacle conditions within a certain range around the target obstacle, and the traffic light conditions in front of the target obstacle. The image sampling of the environmental information within a certain range around the target obstacle and the obstacle conditions within a certain range around the target obstacle may be the sensor range of the vehicle or device detecting the target obstacle. Environmental Dataset X past ={X1, X2, ...X t ,...,X n}, X t ={A t , B t , C t , D t , E t} represents the environmental data of the target obstacle at the tth time point in the historical time period. t A t B is a high-precision data map within a certain range of the target obstacle at the tth time point. The range is set by the designer according to the actual situation; t is the speed, acceleration, heading and other information of the target obstacle at the tth time point; C t is the scene where the target obstacle is located at the tth time point, such as a roundabout, intersection, etc.; D t is the speed and distance of vehicles around the target obstacle at the tth time point; E t is the traffic light situation in front of the target obstacle at the tth time point.

[0039] The historical state information and environmental information of the target obstacle are collected at the same frequency. Both the historical state information and the environmental information are sampled at the same time interval. For example, both the historical state information and the environmental information use the information within the past 1s, and the sampling frequency is 0.2s. At time c, T c-past ={T c-1 , T c-2 , T c-3 , T c-4 , Tc-5}, T c-past is the historical behavior data set for the past 0.2s, 0.4s, 0.6s, 0.8s, and 1.0s at time c. Similarly, at time c, X c-past ={X c-1 , X c-2 , X c-3 , X c-4 , X c-5}, X c-past The environmental data of the target obstacle at 0.2s, 0.4s, 0.6s, 0.8s, and 1.0s past time c.

[0040] Furthermore, the historical state information and environmental information are normalized, that is, the data features are converted to the same scale, such as processing the data into feature vectors in the range of [0, 1], that is, mapping the data features to the interval of [0, 1]. There are many ways to normalize the historical state information and environmental information. For example, method one uses the minimum and maximum normalization method to normalize the value range of each feature of the sample to [0, 1] by scaling; method two uses the standardization method to adjust each dimension of the sample to a distribution with a mean of 0 and a variance of 1; method three uses the whitening method to reduce the redundancy between data features, that is, the correlation between features is reduced, and all features have the same variance. Whitening can also use principal component analysis (PCA) and zero-phase component analysis (ZCA). For the environmental data set X past , environmental data X at different times in the historical period t are all represented by vectors of the same length.

[0041] In step S12, the historical state information and environmental information from step S11 are predicted based on the target network model to obtain the target state information of the target obstacle. The historical state information and environmental information at the current moment are predicted, and the predicted state information for the next moment is used as the target state information for the target obstacle at the current moment. The target state information is used to represent the predicted behavior state of the target obstacle within the target time period, where the duration of the target time period is set by the designer based on actual conditions. For example, the target obstacle's behavior state information for the next moment is predicted based on the target obstacle's current historical state information and environmental information, and the predicted behavior state information for the next moment is used as the target obstacle's behavior state information for the next 8 seconds. As the target obstacle moves, the target state information will be different at different moments.

[0042] In step S13, the target state information obtained in step S12 is used to generate the motion trajectory of the target obstacle according to the motion model, wherein the motion model is designed by the designer to match the target state information to calculate the motion trajectory of the target obstacle in the target time period. For example, when the target state information is sudden stop, a deceleration motion model is used to generate the motion trajectory of the target obstacle at the current time. The target state information of the target obstacle at different times during the motion is different, and the motion trajectory of the target obstacle generated at different times is also different.

[0043] By steps S11 to S13, the historical state information of the target obstacle and the environmental information are obtained, wherein the historical state information is used to represent the real behavior state of the target obstacle in the historical time period, and the environmental information is used to represent the real environmental situation of the target obstacle in the historical time period; then the target state information of the target obstacle is predicted based on the historical state information and the environmental information, wherein the target state information is used to represent the predicted behavior state of the target obstacle in the target time period; the purpose of generating the motion trajectory of the target obstacle by using the target state information is achieved, thereby realizing the technical effect of improving the prediction accuracy of the target obstacle trajectory, and further solving the technical problem of low prediction accuracy of the traditional vehicle technology field due to the rule-based behavior and trajectory prediction for all sudden situation scenes.

[0044] Optionally, in step S12, the target state information of the target obstacle is predicted based on the historical state information and the environmental information, and the step includes the following execution steps:

[0045] Step S121, the historical state information and the environmental information are analyzed by a target network model to predict the target state information, wherein the target network model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes sample data and predicted state information.

[0046] The historical behavior data set T c-past and the environmental data set X c-past at each same time are one-to-one corresponding to each other, and the one-to-one corresponding behavior state information and environmental information are further one-to-one corresponding to the behavior state information at the next time to obtain a training sample. For example, the historical behavior data set T c-past and the historical environmental data set X c-1 corresponding to the target obstacle at time c are {T c-2 , T c-t , …T c-n , …, T c-past} and {X c-1 , X c-2 , …X c-t , …, Xc-n}, the behavioral state information at each same moment is matched with the environmental data one by one, and (X c-n ,T c-n )、(X c-n+1 ,T c-n+1 ),...,(X c-2 ,T c-2 )、(X c-1 ,T c-1 ). c-n ,T c-n )、(X c-n+1 ,T c-n+1 ),...,(X c-2 ,T c-2 )、(X c-1 ,T c-1 ) and then correspond one-to-one with their behavior status information at the next moment as training samples, and obtain ((X c-n ,T c-n ), T c-n+1 )、((X c-n+1 ,T c-n+1 ), T c-n+2 ),...,((X c-2 ,T c-2 ),T c-1 )、((X c-1 ,T c-1 ),T c ), where T c is the behavior state information of the current moment c, n≠0. Similarly, for some moments a and b past moment c, we can get ((X a-n ,T a-n ), T a-n+1 )、((X a-n+1 ,T a-n+1 ), T a-n+2 ),...,((X a-2 ,T a-2 ),T a-1 )、((X a-1 ,T a-1 ),T a )、((X b-n ,T b-n ), T b-n+1 )、((X b-n+1 ,T b-n+1 ), T b-n+2 ),...,((X b-2 ,T b-2 ),T b-1 )、((X b-1 ,T b-1 ),T b ), where Ta 、T b is the behavior status information at time a and b, X a 、X b is the environmental data at time a and time b, n≠0. A certain number of training samples are used as the training set to train the target network model to obtain the trained target network model, where the number of training samples is set by the designer.

[0047] For each training of each training sample in the training set, the target network model uses the error back propagation algorithm to adjust the training parameters in the target network model, that is, for each training sample ((X c-k ,T c-k ), T c-k+1 ), k≠0, execute the error back propagation algorithm to adjust the training parameters of the target network model.

[0048] According to the target network model trained with the training set, the c moment (X c ,T c ) as the input of the target network model to obtain the predicted state information for the next moment c+1 As output, we get ((X c ,T c ), ). It is also called the classification result for the next moment c+1.

[0049] According to the cross-validation method, the target network model is trained again multiple times by re-dividing the training set and the test set. For example, ((X c ,T c ), ) and a certain number of training samples at the past time c, together form the data set D, which is divided into k mutually exclusive subsets of similar size, that is, D = D1∪D2∪...∪D k , Each time, the union of k-1 subsets is used as the training set, and the remaining subset is used as the test set, so that the target network model can be trained and tested k times, and the target network model obtained by k training is tested again on (X c ,T c ) is substituted into the predicted state information of k c+1 moments. After comprehensive consideration, the classification result with the highest frequency among the predicted state information of k c+1 moments is returned as the predicted state information of c+1 moment for this fold cross validation. For the returned prediction status information The selection strategy is different depending on the design of the target network model classification algorithm. For example, the target network model predicts the behavior state at time c to obtain the predicted value to be judged, and then the prediction value interval is used to determine which behavior the prediction value belongs to. In this case, the prediction state information returned by the cross-validation method is The strategy can be, k times at c time (X c ,T c ) is substituted into the mean of the predicted state information of k c+1 moments. The value of k can be selected in many ways. When k=10, it is a 10-fold cross validation, which divides the data set D into 10 subsets, that is, ((X c ,T c ), ) and 9 training samples at different times in the past ((X c-j ,T c-j ), T c-j+1 ) form a data set D, j≠0. D is divided into D1, D2, D3, D4, D5, D6, D7, D8, D9, D 10 , each time 9 training samples are selected as the training set, and the remaining training sample is used as the test set. The target network model is trained 10 times to obtain 10 trained target network models. Each time, the c moment (X c ,T c ) is substituted into the prediction status information of 10 c+1 moments. According to the different classification designs in the target network model, different strategies are used to select and return the prediction status information of this 10-fold cross validation. For example, the classification result with the highest frequency among the 10 classification results is selected as the prediction status information at the c+1 moment of this 10-fold cross validation.

[0050] Dataset D, in which, except (( X c,T c ), ), 9 training samples at different times in the past ((X c-j ,T c-j ), T c-j+1 ) selection is not fixed, that is, the selection of data set D is not fixed. By changing the 9 training samples ((X c-j ,T c-j ), T c-j+1 ) changes the dataset D so that there are multiple ways to divide it into 10 subsets, and then perform multiple 10-fold cross validations, and take the classification results with the highest frequency in the multiple 10-fold cross validations as the final prediction status information of the multiple 10-fold cross validations The final prediction status information for multiple 10-fold cross-validation The selection strategy varies depending on the design of the target network model classification algorithm. The accuracy is the ratio of the number of test samples correctly classified in each 10-fold cross validation to the total number of test samples. For example, the accuracy of the target network model in a 10-fold cross validation is Where m is the total number of test samples, m = 10 in 10-fold cross validation; f(·) is the output function of the target network model; I A (·) is an indicator function, which takes values ​​of 1 and 0 when it is true and false respectively; f(x i ) is the corresponding target network model for the test set x i Output of y i For the corresponding target network model, test input x i During multiple 10-fold cross-validation training sessions, if the accuracy of the target network model no longer improves, stop using 10-fold cross-validation for training.

[0051] Through step S121, multiple sets of data are obtained for training the target network model through the historical state information and environmental information of the target obstacle at different times during the movement process.

[0052] Optionally, the target network model includes a first model and a second model. In step S121, historical state information and environmental information are analyzed based on the target network model to predict target state information. This step includes the following execution steps:

[0053] Step S1211, using the first model to perform feature splicing processing on the historical state information and the environmental information to obtain an intermediate feature vector;

[0054] Step S1212: Use the second model to perform state prediction processing on the intermediate feature vector to obtain target state information.

[0055] The first model extracts features from the historical state information and environmental information obtained in step S11 and then concatenates them to obtain an intermediate feature vector. The second model is used to perform state prediction on the intermediate feature vector obtained by the first model and then obtain a classification result of the target network model input as output, which serves as the target state information.

[0056] For example, the target network model predicts the behavior state information and environmental data at the current time c to obtain the predicted state information at time c+1 Right now is the target state information of the target obstacle in the target time period at time c. For example, the target time period of the target obstacle is 8s. = Turn, indicating that the target obstacle's behavior state at time c in the next 8 seconds is turning.

[0057] There are multiple ways to select the first network model and the second network model, as shown below:

[0058] Method 1: The first model uses a non-fully connected neural network convolutional neural network CNN (Convolution Neural Network, referred to as CNN), and the second model uses SVM (Support Vector Machine, referred to as SVM).

[0059] Method 2: The first model uses a non-convolutional fully connected neural network, and the second model uses softmax.

[0060] Method three: the first model uses a convolutional neural network model, and the second model uses a Bayesian classifier.

[0061] Method 4: The first model uses a non-convolutional fully connected neural network model, and the second model uses a Bayesian classifier.

[0062] Method 1 for achieving the target network model is as follows:

[0063] Convolutional neural networks consist of convolutional layers, activation layers, pooling layers, and fully connected layers.

[0064] A convolutional layer abstracts image feature information through convolution operations. It typically contains multiple convolution kernels, each corresponding to a feature map. Convolutional layer parameters include the kernel stride and kernel size. During convolution of a feature map, the kernel slides horizontally and vertically according to the stride. For example, with a stride of 1, the kernel slides by 1 distance at a time; with a stride of 2, the kernel slides by 2 distances at a time; and with a stride of k, the kernel slides by k distances at a time. In practical applications, convolution is often performed with a stride of 1. Each time the kernel slides across the input image or feature map, a region corresponding to the kernel is generated. The value at each position in this region is multiplied by the value at the kernel position. The results of these multiplications are then summed, and the resulting sum is used as the convolution result for sliding the kernel to that point. In practical applications, to maintain consistency in size between the input image or feature map and the output feature map, padding is typically performed on the periphery of the input image or feature map before convolution. To prevent the padding from affecting the input image or feature map during convolution, zero padding is used, i.e., zero padding is applied to the periphery of the input image or feature map. The first convolutional layer can generally only extract low-level features from the image, such as edges, lines, and corners. Networks with more layers can iteratively extract more complex features from the previous low-level features. For multidimensional images, i.e., multi-channel feature maps, convolution is still performed using a sliding window, from left to right and from top to bottom. When the number of channels is n, the n channels are multiplied and summed by the corresponding positions of the convolution kernel.

[0065] The activation layer uses the ReLU activation function, providing network nonlinear modeling capabilities and enabling neural networks to learn layered nonlinear mappings. The ReLU activation function outputs 0 when the input is less than or equal to 0, and equals the input when the input is greater than 0. The activation layer follows the convolutional layer, either with the activation layer first and then the pooling layer, or with the pooling layer first and then the activation layer. The order of the pooling and activation layers can be adjusted based on the pooling function selected in the actual situation.

[0066] The pooling layer uses the pooling function to compress the feature map obtained after the convolution layer according to the pooling size and step size, that is, the input feature map is sequentially from left to right and from top to bottom, and the new feature map after pooling is obtained according to the pooling function. The pooling function uses the overall statistical features of the adjacent outputs at a certain position to replace the output of the network at that position. Commonly used pooling functions include the maximum pooling function, which uses the maximum value in the adjacent rectangular area, the average pooling function, which uses the average value in the adjacent rectangular area, L 2 norm and a weighted average function based on the distance from the center pixel.

[0067] The fully connected layer connects all local features to obtain global features, and outputs the global features to the classifier.

[0068] Figure 2 FIG. 1 is a schematic diagram of the overall network structure of the obstacle behavior prediction and trajectory prediction algorithm according to one embodiment of the present application. Figure 2 As shown:

[0069] The first layer, the input layer, takes the historical state information of the target obstacle and the environmental information as the input of the target network model.

[0070] The second layer, convolution layer 1, extracts features from the information input into the input layer to obtain features Figure 1 The activation layer ReLU function is used after the convolution layer. The order of the activation layer and the pooling layer can be adjusted by the designer according to the actual situation.

[0071] The third layer, pooling layer 1, compresses features through the pooling function Figure 1 Get features Figure 2 .

[0072] The 4th layer, convolutional layer 2, is used to transform the input features Figure 2 Extract features again to get new features Figure 3 .

[0073] The fifth layer, pooling layer 2, uses the pooling function to transform the features Figure 3 Compression to obtain features Figure 4 .

[0074] The 6th layer, the fully connected layer, concatenates local features into global feature output.

[0075] The 7th layer, the SVM classification layer, classifies the global features output by the fully connected layer and uses the classification results as the target state information.

[0076] The 8th layer, the output layer, converts the target state information into the motion trajectory of the target obstacle within the target time period.

[0077] The first model uses the LeNet-5 network within the CNN framework. The target obstacle's current historical state and environmental information, as well as its current behavioral state and environmental data, are input into the target network model. The target network model then undergoes a first round of training using the backpropagation algorithm to adjust its parameters. The target obstacle's current behavioral state and environmental information are then input into the trained target network model. Based on the output, the target network model undergoes a second round of training, again combining the current historical state and environmental data with multiple 10-fold cross-validation steps. The target obstacle's current behavioral state and environmental data are then substituted into the trained target network model to obtain predicted state information for the next moment. This predicted state information for the target obstacle at the next moment serves as the target state information for the current moment. Based on the target state information for the current moment, the target obstacle's motion trajectory within the target time period is generated. CNN neural networks can also employ other commonly used convolutional networks, such as AlexNet (Alex Network), Inception networks, and residual networks.

[0078] The second model uses SVM, which is a binary classifier. To implement multi-classification using SVM, you can use the following methods:

[0079] (1) Direct method: Modify the objective function directly, merge the parameter solutions of multiple classification surfaces into one optimization problem, and achieve multi-class classification "at one time" by solving the optimization problem.

[0080] (2) Indirect method, which realizes the construction of multi-classifier by combining multiple binary classifiers. Common methods include one-versus-rest (OVR SVMs) and one-versus-one (OVO SVMs or pairwise).

[0081] 1. OVR SVMs Related Methods

[0082] Method 1, OVR SVMs: When training the target network model, samples of a certain category are sequentially classified into one category, and the remaining samples are classified into another category. In this way, k SVMs are constructed for samples of k categories. During classification, unknown samples are classified into the category with the largest classification function value.

[0083] For example, there are 4 categories that need to be divided, that is, 4 labels, namely A, B, C, and D.

[0084] When extracting the training set, extract:

[0085] (1) The eigenvector corresponding to A is taken as the positive set, and the eigenvectors corresponding to B, C, and D are taken as the negative set;

[0086] (2) The eigenvector corresponding to B is taken as the positive set, and the eigenvectors corresponding to A, C, and D are taken as the negative set;

[0087] (3) The eigenvector corresponding to C is taken as the positive set, and the eigenvectors corresponding to A, B, and D are taken as the negative set;

[0088] (4) The eigenvector corresponding to D is taken as the positive set, and the eigenvectors corresponding to A, B, and C are taken as the negative set.

[0089] The target network model is trained using these four training sets, resulting in four different target network models. During testing, the corresponding test feature vectors are tested using each of the four different target network models, resulting in four results for each test: f1(x), f2(x), f3(x), and f4(x). The label corresponding to the largest of the four values ​​is then used as the classification result.

[0090] Method 2: Classification based on decision tree derived from OVR SVMs

[0091] First, all categories are divided into two categories. Then, the subcategories are further divided into two subcategories. This cycle continues until all nodes contain only a single category. This node is a binary tree node. This classification breaks down the original classification problem into a series of two-category classification problems, in which the classification function between the two subcategories is SVM.

[0092] 2. OVO SVMs Related Methods

[0093] Method 1, OVO SVMs: Design an SVM between any two classes of samples. Therefore, for k classes of samples, k(k-1) / 2 SVMs are required. When classifying an unknown sample, the class with the most votes is the class of the unknown sample.

[0094] For example, there are 4 classes to be divided, i.e. 4 labels, A, B, C, D. In the training, the feature vectors corresponding to (A, B), (A, C), (A, D), (B, C), (B, D), (C, D) are selected as the training set, and then 6 different target network models are obtained as the training results. In the test, the corresponding test feature vectors are respectively tested on the 6 different target network models, and then a voting form is taken, and finally a group of results is obtained.

[0095] For example, the voting result is as follows:

[0096] A = B = C = D = 0;

[0097] (A, B) classification, if A, then A = A + 1; otherwise: B = B + 1.

[0098] (A, C) classification, if A, then A = A + 1; otherwise: C = C + 1. ...

[0100] (C, D) classification, if C, then C = C + 1; otherwise: D = D + 1.

[0101] The final classification result is Max (A, B, C, D).

[0102] Method two: classification method based on OVO SVMs derived from directed acyclic graph

[0103] An SVM is designed between any two classes of samples, so k classes of samples need to design k(k-1) / 2 SVMs. When classifying an unknown sample, the class with the most votes is the class of the unknown sample. The training process is similar to OVO SVMs, and the test process is different.

[0104] For example, there are 4 classes to be divided, i.e. 4 labels, A, B, C, D. The training process is similar to OVO SVMs.

[0105] (A, D) classification, if not D, go to (A, C) classification; if not A, go to (B, D) classification.

[0106] (A, C) classification, if not C, go to (A, B) classification; if not A, go to (B, C) classification.

[0107] (B, D) classification, if not D, go to (B, C) classification; if not B, go to (C, D) classification.

[0108] (A, B) classification, if not B, the classification result is A; if not A, the classification result is B.

[0109] (B, C) classification, if not C, the classification result is B; if not B, the classification result is C.

[0110] (C, D) classification, if it is not D, the classification result is C; if it is not C, the classification result is D.

[0111] Figure 3 FIG. 1 is a schematic diagram of the overall network model of the obstacle behavior prediction and trajectory prediction algorithm according to one embodiment of the present application. Figure 3 As shown:

[0112] The target network model inputs the target obstacle's historical behavior data and environmental data, as well as its current behavior state information and environmental data. Both environmental data and historical state information are collected at the same frequency, using the past 1 second of historical data every 0.2 seconds. The input environmental data is uniformly formatted as 300×300 images with 20 channels. The input behavioral state information includes five frames of historical state information and one frame of current behavior state information. The target network model has two convolutional layers, two pooling layers, and two fully connected layers. The convolutional kernel size is 5×5 with a stride of 1, and the pooling kernel size is 2×2. For example, the input 300×300×20 environmental data passes through convolutional layer 1 and pooling layer 1 to obtain 1280-dimensional features, while passing through convolutional layer 2 and pooling layer 2 to obtain 256-dimensional features. The convolutional and pooling layers compress the input feature maps through convolution and pooling functions, achieving dimensionality reduction. After the behavioral state input passes through a fully connected layer, the resulting 64-dimensional features are combined with the previously processed 256-dimensional environmental data through a fully connected layer for feature concatenation. This local feature is converted into a global feature and output to the SVM classifier. The SVM classifier classifies the input feature vector using an objective function, producing a one-dimensional classification result. This one-dimensional classification result is used as the target state information for the target obstacle within the target time period, and the target obstacle's motion trajectory is generated based on this target state information. Behavioral state information includes four types: turning, lane changing, cruising, and emergency stopping. The SVM classifier only outputs one type of behavioral state information as the classification result.

[0113] The target network model can be built based on Python language, combined with the openCV library, and using the Tensorflow framework.

[0114] Figure 4 This is a schematic diagram of the overall network structure of the obstacle behavior prediction and trajectory prediction algorithm implemented by the commonly used CNN network and SVM according to one embodiment of the present application.

[0115] like Figure 4As shown in the figure, a convolutional block consists of a consecutive convolutional and activation layers and b pooling layers (a is usually set to 2-5, and b is 0 or 1). A convolutional network can stack c consecutive convolutional blocks, followed by d fully connected layers (c can have a large range, such as 1-100 or even larger; d is usually 0-2). The activation layer uses the ReLU function.

[0116] Through steps S1211 to S1212, the target state information of the target obstacle within the target time period is obtained by analyzing and predicting the historical state information and the environmental information through the first model and the second model of the target network model.

[0117] Optionally, the above obstacle trajectory prediction method may include the following execution steps:

[0118] Step S14: obtaining observation state information of the target obstacle, wherein the observation state information is used to represent the actual behavior state of the target obstacle within the target time period;

[0119] Step S15, calculating the mean square error between the target state information and the observed state information, and adjusting the training parameters of the target network model according to the mean square error.

[0120] In step S14 and step S15, the training parameters involved in the first model in the target network model are adjusted using the error back propagation algorithm. The predicted state information of the target network model at time c for time c+1 is the target state information at time c. The observed state information at time c+1 is the actual behavior state information of the target obstacle at time c+1. ​​The error of the target network model uses the target state information of the target obstacle at time c and the observation state information T of the target obstacle at time c+1 c+1 The mean square error (MSE) between the two layers is used to represent the error of a neural network. The MSE is related to the weights w between the layers of the neural network and is a function of w, called the error function. The training process of the target network model can be viewed as a parameter optimization process, that is, finding the optimal set of parameters that minimizes the MSE.

[0121] The optimality includes two cases, local minimum and global minimum. Figure 5This is a schematic diagram of the MSE error function values ​​in w space according to one embodiment of the present application. The MSE local minimum solution is point A in the parameter space, and the error function values ​​of its neighboring points are not less than the function value of this point; the MSE global minimum solution refers to the error function values ​​of all points in the parameter space that are not less than the error function value of point B. The corresponding error function values ​​of the two are respectively called the local minimum and global minimum of the error function. Obviously, a point with a gradient of 0 in the parameter space is a local minimum point as long as its error function value is less than the error function value of the neighboring point; the global minimum is the local minimum, and the opposite is not true. The parameter optimization process is to find parameters so that the corresponding error function value is the global minimum. The parameter optimization process adopts the gradient descent method, and the negative gradient direction is the direction in which the function value decreases fastest. For example, in the process of iteratively searching for the optimal parameter value, each iteration first calculates the gradient of the error function at the current point, determines the negative gradient direction of the search, and searches for the optimal solution along the negative gradient direction.

[0122] (1) The error function has only one local minimum in the parameter space

[0123] If the gradient of the error function at the current point is 0, the point is a local minimum and a global minimum, and the parameter iteration is stopped, that is, the training is stopped.

[0124] (2) The error function has multiple local minima in the parameter space

[0125] If the gradient of the error function at the current point is 0, that point is a local minimum, and the search for the global minimum continues. If the error function has multiple local minima and it is unclear whether it is the global minimum, the following strategies can be used to help the parameter tuning process avoid local minima and approach the global minimum.

[0126] Stochastic gradient descent (SGD) is a method that uses randomness to calculate the gradient, unlike standard gradient descent, which accurately calculates the gradient. This means that even if the parameters are selected to achieve a local minimum in the error function, the calculated gradient may still be non-zero, allowing the search to continue beyond the local minimum.

[0127] Through steps S14 to S15, the mean square error of the target state information and the observation state information of the target obstacle is obtained, so that when the minimum value or the minimum value of the mean square error is found through parameter tuning, that is, the mean square error no longer decreases, the training of the target network model is stopped.

[0128] Optionally, the above obstacle trajectory prediction method may include: the environmental information includes: one or more of electronic map data, target vehicle data, target obstacle data, target scene data and traffic light data.

[0129] Optionally, the above obstacle trajectory prediction method may include: the sampling frequencies of the historical state information and the environmental information are the same.

[0130] Figure 6 is a structural block diagram of an obstacle trajectory prediction device according to one embodiment of the present application, such as Figure 6 As shown, the obstacle trajectory prediction device 600 includes:

[0131] The acquisition module 601 acquires historical state information and environmental information of the target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle in the historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle in the historical time period.

[0132] The prediction module 602 is configured to predict target state information of the target obstacle based on historical state information and environmental information, wherein the target state information is used to represent the predicted behavior state of the target obstacle within a target time period.

[0133] The generation module 603 is used to generate the motion trajectory of the target obstacle using the target state information.

[0134] Optionally, the prediction module 602 is also used to analyze historical status information and environmental information through a target network model to predict the target status information, wherein the target network model is obtained through machine learning training using multiple sets of data, and each set of data in the multiple sets of data includes: sample data and predicted status information.

[0135] Optionally, the prediction module 602 is further configured to perform feature concatenation processing on the historical state information and the environmental information using the first model to obtain an intermediate feature vector; and perform state prediction processing on the intermediate feature vector using the second model to obtain target state information.

[0136] Optionally, the obstacle trajectory prediction device 600 further includes: an adjustment module 604, configured to obtain observation state information of the target obstacle, wherein the observation state information is used to represent the actual behavior state of the target obstacle within a target time period; calculate the mean square error between the target state information and the observation state information, and adjust the training parameters of the target network model according to the mean square error.

[0137] Optionally, the acquisition module 601 is further configured to acquire environmental information, where the environmental information includes one or more of electronic map data, target vehicle data, target obstacle data, target scene data, and traffic light data.

[0138] Optionally, the acquisition module 601 is further configured to acquire historical status information and environmental information, wherein the historical status information and the environmental information have the same sampling frequency.

[0139] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0140] An embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when running.

[0141] Optionally, in this embodiment, the non-volatile storage medium may be configured to store a computer program for executing the following steps:

[0142] S1. Obtain historical state information and environmental information of the target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle in the historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle in the historical time period.

[0143] S2 is used to predict target state information of the target obstacle based on historical state information and environmental information, wherein the target state information is used to represent the predicted behavior state of the target obstacle within a target time period.

[0144] S3 is used to generate the motion trajectory of the target obstacle using the target state information.

[0145] Optionally, in this embodiment, the above-mentioned non-volatile storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0146] An embodiment of the present application further provides a processor for running a program, wherein the program is configured to execute the steps of any of the above method embodiments when running.

[0147] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0148] S1. Obtain historical state information and environmental information of the target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle in the historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle in the historical time period.

[0149] S2 is used to predict target state information of the target obstacle based on historical state information and environmental information, wherein the target state information is used to represent the predicted behavior state of the target obstacle within a target time period.

[0150] S3 is used to generate the motion trajectory of the target obstacle using the target state information.

[0151] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0152] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0153] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0156] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0158] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for predicting obstacle trajectories, characterized in that: include: Acquiring historical state information and environmental information of the target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle within a historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle within the historical time period; Predicting target state information of the target obstacle based on the historical state information and the environmental information, wherein the target state information is used to represent a predicted behavior state of the target obstacle within a target time period; generating a motion trajectory of the target obstacle using the target state information and a motion model, wherein the motion model is used to match the target state information to generate the motion trajectory of the target obstacle within a target time period; The method further includes: performing normalization processing on the historical state information and the environmental information to obtain a normalization processing result, wherein the normalization processing is used to convert data features of the historical state information and the environmental information into the same scale, and the historical state information and the environmental information have the same sampling frequency; The method further includes: taking the historical state information and the environmental information as inputs of the target network model; extracting features from the information inputted into the input layer to obtain a first feature map; compressing the first feature map through a first pooling function to obtain a second feature map; extracting features from the second feature map again to obtain a third feature map; compressing the third feature map through a second pooling function to obtain a fourth feature map; splicing the fourth feature map to obtain global features; and classifying the global features to obtain the target state information.

2. The obstacle trajectory prediction method according to claim 1, characterized in that: The predicting of the target state information of the target obstacle based on the historical state information and the environmental information includes: The historical state information and the environmental information are analyzed by the target network model to predict the target state information, wherein the target network model is obtained through machine learning training using multiple sets of data, and each set of data in the multiple sets of data includes: sample data and predicted state information.

3. The obstacle trajectory prediction method according to claim 2, characterized in that: The target network model includes: a first model and a second model, and the analyzing the historical state information and the environmental information by the target network model to predict the target state information includes: Using the first model to perform feature splicing processing on the historical state information and the environmental information to obtain an intermediate feature vector; The second model is used to perform state prediction processing on the intermediate feature vector to obtain the target state information.

4. The obstacle trajectory prediction method according to claim 3, characterized in that: The method further comprises: Acquiring observation state information of the target obstacle, wherein the observation state information is used to represent the actual behavior state of the target obstacle within the target time period; Calculate the mean square error between the target state information and the observed state information, and adjust the training parameters of the target network model according to the mean square error.

5. The obstacle trajectory prediction method according to claim 1, characterized in that: The environmental information includes: one or more of electronic map data, target vehicle data, target obstacle data, target scene data and traffic light data.

6. An obstacle trajectory prediction device, characterized in that: include: an acquisition module for acquiring historical state information and environmental information of a target obstacle, wherein the historical state information is used to represent the actual behavior state of the target obstacle within a historical time period, and the environmental information is used to represent the actual environmental conditions of the target obstacle within the historical time period; a prediction module, configured to predict target state information of the target obstacle based on the historical state information and the environmental information, wherein the target state information is used to represent a predicted behavior state of the target obstacle within a target time period; a generating module, configured to generate a motion trajectory of the target obstacle using the target state information and a motion model, wherein the motion model is configured to match the target state information to generate the motion trajectory of the target obstacle within a target time period; The device further includes: a processing module configured to perform normalization processing on the historical state information and the environmental information to obtain a normalization processing result, wherein the normalization processing is configured to convert data features of the historical state information and the environmental information into the same scale, and the historical state information and the environmental information have the same sampling frequency; Among them, the prediction module is also used to: use the historical state information and the environmental information as input of the target network model; extract features from the information input in the input layer to obtain a first feature map; compress the first feature map through a first pooling function to obtain a second feature map; extract features from the second feature map again to obtain a third feature map; compress the third feature map through a second pooling function to obtain a fourth feature map; splice the first feature map, the second feature map, the third feature map and the fourth feature map to obtain global features; classify the global features to obtain the target state information.

7. A non-volatile storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the obstacle trajectory prediction method according to any one of claims 1 to 5 when running.

8. A processor, characterized in that: The processor is used to run a program, wherein the program is configured to execute the obstacle trajectory prediction method described in any one of claims 1 to 5 when running.

9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the obstacle trajectory prediction method according to any one of claims 1 to 5.

Images