Trajectory prediction model generation and trajectory information processing method, device and medium

Abstract

Embodiments of the present application provide a trajectory prediction model generation and trajectory information processing method, device and medium, wherein the trajectory prediction model is a machine learning model or a neural network model. In the model training phase, the historical trajectory is used to predict the future trajectory, thereby completing the self-supervised trajectory prediction model training, greatly reducing the artificial labeling workload, having the characteristics of scene self-adaptation, no additional data labeling or rule setting, etc., effectively solving many problems faced in application practice. Further, the model is trained to enable the trajectory prediction model to predict the information of the trajectory points in multiple direction categories at each time point in the future time period, realize the trajectory prediction target of multiple time point and multiple trajectory distribution, and effectively improve the accuracy of trajectory prediction. In addition, predicting the trajectory in the future period of time contains more rich information than predicting a single trajectory point, and the rich information can help improve the accuracy of anomaly detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a trajectory prediction model generation method and a trajectory information processing method, device and medium. BACKGROUND

[0002] The accuracy of vehicle trajectory prediction directly affects road traffic safety. In an urban road traffic scenario, a road intersection or a road section can include multiple lanes and multiple vehicles traveling in different directions, presenting the characteristics of complex and variable road conditions. In addition, there are multiple possibilities for the movement of a vehicle at the next moment, for example, the vehicle can turn left, turn right, go straight, or stop to wait for a traffic light, etc. On a mixed road, a vehicle can even change lanes or even make a U-turn, presenting the characteristics of complex and variable vehicle movement. Due to the characteristics of complex and variable road conditions and complex and variable vehicle movement in an urban road traffic scenario, the accuracy of vehicle trajectory prediction becomes a difficulty. SUMMARY

[0003] Aspects of the present application provide a trajectory prediction model generation method and a trajectory information processing method, device and medium to improve the accuracy of trajectory prediction.

[0004] The embodiment of the present application provides a trajectory prediction model generation method, which comprises: obtaining a training sample, the training sample comprising first real trajectory data of a sample target object in a first time period and second real trajectory data in a second time period, the first time period being a time period preceding the second time period; inputting the first real trajectory data into an initial trajectory prediction model to obtain predicted trajectory data of the sample target object in the second time period; wherein the predicted trajectory data comprises predicted position coordinates and predicted probabilities of predicted trajectory points in multiple direction categories at each moment in the second time period; and adjusting model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data to obtain a trajectory prediction model for trajectory prediction.

[0005] The embodiment of the present application also provides a trajectory data processing method, which comprises: obtaining historical trajectory data of a target object in a historical time period; inputting the historical trajectory data into a trajectory prediction model to obtain future trajectory data of the target object in a future time period predicted by the trajectory prediction model; the future trajectory data comprising predicted position coordinates and predicted probabilities of predicted trajectory points in multiple direction categories at each moment in the future time period.

[0006] The embodiment of the present application also provides an electronic device, which comprises: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the trajectory prediction model generation method or the trajectory data processing method.

[0007] The embodiment of the present application further provides a computer storage medium storing a computer program, which enables a processor to implement the steps of the generation method of the trajectory prediction model or the trajectory data processing method when the computer program is executed by the processor.

[0008] In the embodiment of the present application, in the model training stage, the historical trajectory is used to predict the future trajectory, so as to complete the self-supervised trajectory prediction model training, greatly reducing the artificial labeling workload, having the characteristics of scene self-adaptation, no additional data labeling or rule setting, and effectively solving many problems faced in application practice, wherein the trajectory prediction model is a machine learning model or a neural network model. Further, the model training is performed with the goal of enabling the trajectory prediction model to predict the information of the trajectory points in multiple direction categories at each time point in a future time period, achieving the trajectory prediction goal of multiple time point and multiple trajectory distribution, and effectively improving the accuracy of trajectory prediction. In addition, predicting the trajectory in a future period of time contains more rich information than predicting a single trajectory point, and the rich information can help to improve the accuracy of anomaly detection. BRIEF DESCRIPTION OF DRAWINGS

[0009] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the exemplary embodiments of the present application and their descriptions serve to explain the present application and do not constitute improper limitations on the present application. In the drawings:

[0010] Figure 1 An application scenario provided for the embodiment of the present application;

[0011] Figure 2 A flowchart of the generation method of the trajectory prediction model provided for the embodiment of the present application;

[0012] Figure 3 A region division graph associated with the direction category provided for the embodiment of the present application;

[0013] Figure 4 A flowchart of the trajectory data processing method provided for the embodiment of the present application;

[0014] Figure 5 Another application scenario provided for the embodiment of the present application;

[0015] Figure 6 A structural schematic diagram of the generation device of the trajectory prediction model provided for the embodiment of the present application;

[0016] Figure 7 A structural schematic diagram of the trajectory data processing device provided for the embodiment of the present application;

[0017] Figure 8A structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0019] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" describes the access relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character " / " generally represents an "or" relationship between the associated objects. In addition, in the embodiments of the present application, "first", "second", "third", "fourth", "fifth", and "sixth" are only to distinguish the contents of different objects, and have no other special meanings.

[0020] Due to the characteristics of complex and variable road conditions and complex and variable vehicle motion situations in urban road traffic scenes, the accuracy of vehicle trajectory prediction becomes a difficulty. Therefore, the embodiments of the present application provide a trajectory prediction model generation and trajectory information processing method, device and medium. In the embodiments of the present application, in the model training stage, the historical trajectory is used to predict the future trajectory, thereby completing the self-supervised trajectory prediction model training, greatly reducing the manual annotation workload, having the characteristics of scene self-adaptation, no need for additional data annotation or rule setting, and effectively solving many problems faced in application practice, wherein the trajectory prediction model is a machine learning model or a neural network model. Further, the trajectory prediction model is trained to predict the information of the trajectory points in multiple direction categories at each time point in the future time period, realize the trajectory prediction target of multiple time point and multiple trajectory distribution, and effectively improve the accuracy of trajectory prediction. In addition, predicting the trajectory in a future period of time contains more rich information than predicting a single trajectory point, and the rich information can help improve the accuracy of anomaly detection.

[0021] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0022] Figure 1An application scenario provided for an embodiment of the present application. In a road traffic scenario, the road includes multiple lanes, some of which are ordinary lanes available for social vehicles to travel, and some of which are emergency lanes available for special vehicles such as police cars, fire engines, ambulances, and engineering rescue vehicles to travel for emergency tasks. In the ordinary lanes, there can be small passenger car lanes, truck lanes, and passenger truck lanes, of course, the lanes on the road can be flexibly divided according to actual application requirements. Referring to Figure 1 , a road intersection or section can include multiple lanes and multiple vehicles in different driving directions, presenting the characteristics of complex and variable road conditions. In addition, there are multiple possibilities for the movement of vehicles at the next moment, for example, the vehicle can turn left, turn right, go straight, or be stationary to wait for a traffic light, etc. On the mixed road, the vehicle can even change lanes or even make a U-turn, presenting the characteristics of complex and variable vehicle movement. Due to the characteristics of complex and variable road conditions and complex and variable vehicle movement of the urban road traffic scenario, the accuracy of vehicle trajectory prediction becomes a difficulty.

[0023] Figure 2 A flowchart of a trajectory prediction model generation method provided for an embodiment of the present application. Referring to Figure 2 , the method can include:

[0024] 201, obtaining a training sample, the training sample including first real trajectory data of a sample target object in a first time period and second real trajectory data in a second time period, the first time period being a previous time period of the second time period;

[0025] 202, inputting the first real trajectory data into an initial trajectory prediction model to obtain predicted trajectory data of the sample target object in the second time period; wherein the predicted trajectory data includes predicted position coordinates and predicted probabilities of predicted trajectory points in multiple direction categories at each time in the second time period.

[0026] 203, adjusting model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data to obtain a trajectory prediction model for trajectory prediction.

[0027] It is worth noting that steps 201 to 203 are a model training process. In actual application, steps 201 to 203 can be repeatedly executed multiple times using multiple training samples, that is, multiple model training is performed. After each model training, the trajectory prediction model with adjusted model parameters is used as the trajectory prediction model to be trained next time, and new training samples are used to perform the next model training until the end of training condition is met, and the trajectory prediction model obtained by the last training is used as the final trajectory prediction model. Wherein, the end of training condition can be that the number of model training reaches a specified number, or the trajectory prediction model converges, which is not specifically limited here.

[0028] In this embodiment, the trajectory prediction model can be a machine learning model or a neural network model, without limitation.

[0029] In this embodiment, the model structure of the trajectory prediction model includes, for example, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), and a long short-term memory (LSTM).

[0030] In this embodiment, before model training, a plurality of training samples are prepared first. When preparing the training samples, both the first real trajectory data of the sample target object in the first time period and the second real trajectory data of the sample target object in the second time period are prepared, the first time period being a time period prior to the second time period.

[0031] It is worth noting that the sample target object can be a vehicle, a pedestrian, a non-motor vehicle, an animal, and various movable objects, without further limitation.

[0032] It is worth noting that, for the convenience of distinguishing and understanding, the actual trajectory point is referred to as a real trajectory point, and the position coordinates of the real trajectory point are referred to as real position coordinates. Based on this, the first real trajectory data includes the real position coordinates of the real trajectory points of the sample target object at each time in the first time period, and similarly, the second real trajectory data includes the real position coordinates of the real trajectory points of the sample target object at each time in the second time period. For any training sample, the first time period is a historical time period of the second time period, and the second time period is a future time period of the first time period. Therefore, by taking the first real trajectory data of the sample target object in the first time period as the model input parameter and taking the second real trajectory data of the sample target object in the second time period as the expected model output parameter, model training is performed, and a trajectory prediction model for trajectory prediction can be obtained.

[0033] In this embodiment, the channel for obtaining the real trajectory data of the sample target object is not limited. For example, a video data including the sample target object can be collected by using an image collection device, and the video data is processed based on a trajectory tracking algorithm to obtain real position coordinates of real trajectory points of the sample target object at each time, so as to form the real trajectory data of the sample target object. For another example, real position coordinates of real trajectory points of the sample target object at each time are detected by using a laser radar, so as to form the real trajectory data of the sample target object. For another example, real position coordinates of real trajectory points of the sample target object at each time are positioned by using a GPS (Global Positioning System), so as to form the real trajectory data of the sample target object.

[0034] In this embodiment, the real trajectory data of a plurality of real trajectories is prepared in advance, and the real trajectory data of the plurality of real trajectories is denoted as X, X = [Tr 0 , Tr 1 ,..., Tr N ], N is a positive integer, N is the number of trajectories, and the length of each trajectory can be different. Taking a vehicle as an example, the real trajectory data of any trajectory of the vehicle from time 0 to time M is denoted as Tr = [tr M ] T , where T is a matrix transpose symbol, M is a positive integer, and M of different trajectories can be the same or different. The trajectory point at the tth time is denoted as tr t , where tr t = [x t , y t ] T , where x t is a position coordinate in the horizontal coordinate axis direction, and y t is a position coordinate in the vertical coordinate axis direction.

[0035] It is assumed that there are B trajectories, B is a positive integer, the real position coordinates of the first L s real trajectory points on each trajectory from the ith time are taken as the first real trajectory information, and the real position coordinates of the last L p real trajectory points are taken as the second real trajectory information, i, B, L s , L p are positive integers. Then, for the bth trajectory, b ∈ [0, B), b is a positive integer. The first real trajectory information collected from the bth trajectory is denoted as The second real trajectory information collected from the bth trajectory is denoted as Based on this, the first real trajectory information set obtained from the B trajectories is denoted as The second real trajectory information set obtained from the B trajectories is denoted as

[0036] In this embodiment, after preparing a plurality of training samples, a plurality of model training is performed using the plurality of training samples to obtain a trajectory prediction model capable of trajectory prediction. In order to facilitate understanding and distinction, the trajectory point predicted by the trajectory prediction model is referred to as a predicted trajectory point, and the position coordinates of the predicted trajectory point are referred to as predicted position coordinates.

[0037] In this embodiment, in the trajectory prediction process, the trajectory prediction model can predict the predicted position coordinates and prediction probabilities of the predicted trajectory points in each time period in the future time period.

[0038] In this embodiment, the direction category refers to the relative position relationship between the predicted trajectory point at the current time and the set predicted trajectory point at the previous time. The set predicted trajectory point at the previous time can be flexibly selected from any one of the predicted trajectory points in a plurality of direction categories at the previous time, for example, selecting the predicted trajectory point with the maximum prediction probability at the previous time as the set predicted trajectory point, for example, selecting any one of the predicted trajectory points at the previous time with a prediction probability greater than a set probability threshold as the set predicted trajectory point, for example, selecting any one of the predicted trajectory points with a larger prediction probability as the set predicted trajectory point, and the like, which is not limited.

[0039] Of course, when predicting the information of the predicted trajectory point at the first time in the second time period, the previous time thereof is the last time in the first time period, and at this time, the direction category refers to the relative position relationship between the predicted trajectory point at the first time and the real trajectory point at the previous time.

[0040] In this embodiment, the division of the direction category is not limited. For example, the direction category includes stationary, front, back. For example, the direction category includes stationary, left, right. For example, the direction category includes stationary, right front, left front, left back, right back. For example, the right front, left front, left back, right back, and the like are further divided into regions with smaller granularity, for example, the right front is divided into two halves, which are the first right front and the second right front; by analogy, the left front is divided into the first left front and the second left front; the left back is divided into the first left back and the second left back; the right back is divided into the first right back and the second right back. Thus, the direction category includes stationary, first right front, second right front, first left front, second left front, first left back, second left back, first right back, and second right back. Of course, the direction category can be flexibly divided according to actual application requirements, and is not limited to the above examples. It is worth noting that the more the direction categories, the higher the prediction accuracy. In actual applications, the required direction categories can be determined based on the prediction accuracy.

[0041] It is worth noting that when predicting the predicted trajectory point of each direction category at the current time, the specified predicted trajectory point of the last time of the current time is taken as the origin, and the current motion direction is taken as the forward direction as the precondition, and each direction category is defined. As an example, a two-dimensional coordinate axis is established by taking the specified predicted trajectory point of the last time of the current time as the origin, taking the forward direction as the longitudinal coordinate axis direction, and taking the vertical direction of the forward direction as the transverse coordinate axis direction, and the entire space region is divided into multiple intervals, and each interval corresponds to a position region of a direction category. Referring to Figure 3 , the four intervals corresponding to the two-dimensional coordinate axis are the position regions of the four direction categories of right front, left front, left rear, and right rear. The relative position relationship between the predicted trajectory point of the current time and the specified predicted trajectory point of the last time falling in these position regions is right front, left front, left rear, and right rear, respectively. Of course, the relative position relationship between the predicted trajectory point of the current time and the specified predicted trajectory point of the last time falling in the two-dimensional coordinate axis is static, that is, the predicted trajectory points of the five direction categories at each time are predicted. For another example, on the basis of Figure 3 , the entire space region is divided into eight intervals, that is, Figure 3 each interval is further divided into two small intervals Figure 3 (not shown in ), and the relative position relationship between the predicted trajectory point of the current time and the specified predicted trajectory point of the last time falling in these position regions is first right front, second right front, first left front, second left front, first left rear, second left rear, first right rear, and second right rear, respectively. Of course, the relative position relationship between the predicted trajectory point of the current time and the specified predicted trajectory point of the last time falling in the two-dimensional coordinate axis is static, that is, the predicted trajectory points of the nine direction categories at each time are predicted.

[0042] Specifically, for example, referring to Figure 3 , the prediction result of each time includes the predicted position coordinates of the predicted trajectory points of the five direction categories and their prediction probabilities. It is assumed that the prediction result of each time is denoted as Tr p = [tr p5 , p p5 ] T , the predicted position coordinates of the predicted trajectory points of the five direction categories at this time are denoted as , wherein , respectively, represent the predicted position coordinates of the predicted trajectory points of the five direction categories of static, right front, right rear, left rear, and left front, which are predicted under the precondition that the specified predicted trajectory point of the last time is taken as the origin and the current motion direction is taken as the forward direction. It is worth noting that the current motion direction can be determined according to the predicted position coordinates of the predicted trajectory points at a plurality of times before the current time. Referring to Figure 3, under the premise that the specified prediction trajectory point at the last time is determined as the origin and the current motion direction is determined as the forward direction, a two-dimensional coordinate axis is established with the specified prediction trajectory point at the last time as the origin, the forward direction as the longitudinal coordinate axis direction, and the vertical direction of the forward direction as the transverse coordinate axis direction, and the prediction position coordinates and their probabilities of the prediction trajectory point falling in the origin and four intervals of the two-dimensional coordinate axis are predicted. The prediction trajectory points falling in the four intervals correspond to the prediction trajectory points in four direction categories, i.e., the right front, the left front, the left rear, and the right rear, and the prediction trajectory point falling in the origin corresponds to a stationary prediction trajectory point.

[0043] The prediction probabilities of the prediction trajectory points in the five direction categories at this time are denoted as , respectively, represent the prediction probabilities of the prediction trajectory points in five direction categories, i.e., stationary, right front, right rear, left rear, and left front, wherein, That is, the sum of the prediction probabilities of the prediction trajectory points in the five direction categories is 1.

[0044] In this embodiment, the initial trajectory prediction model is used to predict the prediction trajectory data of the sample target object in the second time period, and then the loss value reflecting the difference between the predicted value and the true value of the training sample is determined according to the second true trajectory data of the sample target object in the second time period and the prediction trajectory data, and the model parameters of the initial trajectory prediction model are adjusted according to the loss value of the training sample. It is worth noting that the smaller the loss value is, the better the robustness of the model is. Therefore, in the iterative model training process, the model parameters of the trajectory prediction model are continuously adjusted to minimize the loss value until the loss value is minimized or the maximum number of iterative training is reached.

[0045] In this embodiment, any loss function can be used to determine the loss value of the training sample. For example, the loss function includes but is not limited to: mean square error loss function, L2 loss function (also known as Euclidean distance), L1 loss function (Manhattan distance), cross-entropy loss function, softmax loss function, and Focal loss loss function mainly used to solve the problem of uneven difficulty of samples, etc.

[0046] It is worth noting that when the loss function is used to determine the loss value of the training sample, the true position coordinates of the true trajectory point at each time in the second time period are input into the loss function as the true value, and the prediction position coordinates of the prediction trajectory point at each time are input into the loss function as the predicted value, and the loss function calculates the loss value of the training sample based on the input true value and predicted value. For more information about the loss function calculating the loss value, please refer to the related technology.

[0047] The technical scheme provided by the embodiments of the present application uses historical trajectories to predict future trajectories in the model training stage, thereby completing self-supervised trajectory prediction model training, greatly reducing the manual labeling workload, having the characteristics of scene self-adaptation, not requiring additional data labeling or rule setting, and effectively solving many problems faced in application practice, wherein the trajectory prediction model is a machine learning model or a neural network model. Further, the model is trained to enable the trajectory prediction model to predict the information of trajectory points in multiple direction categories at each time point in a future time period, achieve the trajectory prediction goal of multiple time point and multiple trajectory distribution, and effectively improve the accuracy of trajectory prediction. In addition, predicting trajectories in a future period of time contains richer information than predicting a single trajectory point, and the rich information can help improve the accuracy of anomaly detection.

[0048] In some optional embodiments, in order to further improve the trajectory prediction accuracy of multiple time points and multiple trajectory distribution, an optional implementation manner of adjusting the model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data is as follows: according to the second real trajectory data, the real probabilities of the real trajectory points at each time point in the second time period in each direction category are determined; according to the real probabilities of the real trajectory points at each time point in each direction category in the second time period and the predicted probabilities of the predicted trajectory points at each time point in each direction category, a first loss value of the training sample in the trajectory position probability dimension is calculated; and the model parameters of the initial trajectory prediction model are adjusted according to the first loss value.

[0049] In the present embodiment, first, the real probabilities of the real trajectory points at each time point in the second time period in each direction category are determined according to the second real trajectory data. Specifically, for any first time point in the second time period, the real position coordinates of the real trajectory point at the first time point are obtained from the second real trajectory data, and the real position coordinates of the real trajectory point at the second time point are obtained from the second real trajectory data or the first real trajectory data, the second time point being the previous time point of the first time point; the target relative position relationship between the real trajectory point at the first time point and the real trajectory point at the second time point is determined according to the real position coordinates of the real trajectory point at the first time point and the real position coordinates of the real trajectory point at the second time point; and the real probabilities of the real trajectory point at the first time point in each direction category are determined according to the target relative position relationship.

[0050] After determining the real probabilities of the real trajectory points at each time in the second time period in each direction category and the prediction probabilities of the prediction trajectory points at each time in each direction category, the real probabilities of the real trajectory points at each time in each direction category are input into the loss function as real values, the prediction probabilities of the prediction trajectory points at each time in each direction category are input into the loss function as prediction values, and the loss function calculates the first loss value of the training sample in the trajectory position probability dimension based on the input real values and prediction values.

[0051] The loss function used to calculate the first loss value of the training sample in the trajectory position probability dimension is not limited in the embodiment. Further, to further improve the model performance, the first loss value corresponding to each time in the second time period can be calculated respectively, and the first loss value corresponding to each time can be weighted summed, averaged, or operated by other operations such as arithmetic square to obtain the final first loss value of the training sample in the trajectory position probability dimension. In the calculation of the first loss value corresponding to each time in the second time period, the real probabilities of the real trajectory points at the time in each direction category can be input into the loss function as real values, the prediction probabilities of the prediction trajectory points at the time in each direction category can be input into the loss function as prediction values, and the loss function calculates the first loss value corresponding to the time based on the input real values and prediction values. Of course, the loss function used to calculate the first loss value corresponding to each time is not limited. Further, to further improve the model performance, the first loss value corresponding to each time can be calculated by formula (1).

[0052]

[0053] wherein, L Probability represents the first loss value corresponding to a single time, n is a positive integer, and n is the number of direction categories. is the real probability of the real trajectory point at the corresponding time in the i-th direction category; is the prediction probability of the prediction trajectory point at the corresponding time in the i-th direction category.

[0054] In some optional embodiments, to further improve the trajectory prediction accuracy of the multi-time point multi-trajectory distribution, in addition to adjusting the model parameters in combination with the first loss value of the training sample in the trajectory position probability dimension, the model parameters can also be adjusted in combination with at least one of the second loss value of the training sample in the trajectory direction dimension or the third loss value of the training sample in the trajectory distance dimension.

[0055] Based on this, according to the first loss value, an optional implementation manner of adjusting the model parameters of the initial trajectory prediction model is: determining a second loss value of the training sample in the trajectory direction dimension according to the real position coordinates of the real trajectory points at each time in the second time period and the predicted position coordinates of the predicted trajectory points with the maximum prediction probability; or determining a third loss value of the training sample in the trajectory distance dimension according to the real position coordinates of the real trajectory points at each time in the second time period and the predicted position coordinates of the predicted trajectory points with the maximum prediction probability; and adjusting the model parameters of the initial trajectory prediction model according to the first loss value and in combination with at least one of the second loss value and the third loss value.

[0056] In the embodiment, in the model training process, the loss information of the trajectory direction can help to accurately perform trajectory prediction. In particular, in a traffic scene, the vehicle movement direction is very important for vehicle trajectory prediction. Specifically, the second loss value of the training sample in the trajectory direction dimension reflects the difference information between the real trajectory direction and the predicted trajectory direction.

[0057] In an optional implementation manner, the specific process of determining the second loss value of the training sample in the trajectory direction dimension is: generating a first trajectory vector according to the real position coordinates of the real trajectory points at each time in the second time period; generating a second trajectory vector according to the predicted position coordinates of the predicted trajectory points with the maximum prediction probability at each time in the second time period; and determining the second loss value of the training sample in the trajectory direction dimension according to the cosine similarity between the first trajectory vector and the second trajectory vector.

[0058] Specifically, after the cosine similarity between the first trajectory vector and the second trajectory vector is determined, the cosine similarity can be directly taken as the second loss value of the training sample in the trajectory direction dimension, or the product of the cosine similarity and a constant can be taken as the second loss value of the training sample in the trajectory direction dimension, or the difference between the constant and the cosine similarity can be taken as the second loss value of the training sample in the trajectory direction dimension. As an example, the second loss value of the training sample in the trajectory direction dimension can be calculated by using formula (2).

[0059]

[0060] In formula (2), L Direction represents the second loss value of the training sample in the trajectory direction dimension, cossim() represents a cosine similarity function, and cossim(Tr y,(x,y) ,Tr p,(x,y) ) represents the cosine similarity between the first trajectory vector and the second trajectory vector. Tr y,(x,y) represents the first trajectory vector formed by the real position coordinates of the real trajectory points at each time in the second time period; and Tr p,(x,y)a second trajectory vector formed by predicted position coordinates of predicted trajectory points with maximum prediction probability at each time instant in the second time period.

[0061] In the embodiment, in the model training process, the loss information considering the trajectory distance can help to accurately perform trajectory prediction. The trajectory distance can be understood as the distance between the real trajectory point and the predicted trajectory point. Specifically, the third loss value of the training sample in the trajectory distance dimension reflects the distance information between the real trajectory point and the predicted trajectory point at each time instant. When determining the third loss value of the training sample in the trajectory distance dimension, first, the distance information between the real trajectory point and the predicted trajectory point with maximum prediction probability at each time instant in the second time period is determined, and then the distance information corresponding to each time instant in the second time period is weighted summed, averaged, or accumulated, to obtain the final third loss value of the training sample in the trajectory distance dimension.

[0062] As an example, the distance information between the real trajectory point and the predicted trajectory point can be represented by the absolute value and / or squared difference between the real position coordinates of the real trajectory point and the predicted position coordinates of the predicted trajectory point. Based on this, the implementation manner of determining the third loss value of the training sample in the trajectory distance dimension can be: determining the absolute value and / or squared difference between the real position coordinates of the real trajectory point and the predicted position coordinates of the predicted trajectory point with maximum prediction probability at each time instant in the second time period; and determining the third loss value of the training sample in the trajectory distance dimension according to the absolute value and / or squared difference at each time instant in the second time period.

[0063] For example, the third loss value between the real position coordinates and the predicted position coordinates of the predicted trajectory point with maximum prediction probability at each time instant in the second time period is calculated according to formula (3).

[0064]

[0065] wherein, L Distance denotes the third loss value corresponding to each time instant in the second time period, tr y denotes the real position coordinates of each time instant; tr p denotes the predicted position coordinates of the predicted trajectory point with maximum prediction probability at each time instant.

[0066] In some optional embodiments, the trajectory prediction model can undertake a prediction task of size information of the target object in addition to the trajectory prediction task. Based on this, the training sample further includes: first size information of the sample target object corresponding to the sample detection box at each time in the first time period, and second size information at each time in the second time period; the above method further includes: inputting the first size information of the sample detection box at each time in the first time period into the initial trajectory prediction model to obtain predicted size information of the sample detection box at each time in the second time period; and correspondingly, adjusting the model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data, including: adjusting the model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data, and according to the second size information and the predicted size information of the sample detection box at each time in the second time period.

[0067] In actual applications, the camera is usually erected at a certain height and angle, which causes the moving target in the video picture to present differently at different positions, and the size of the moving target usually presents large near and small far; under a fixed sampling frequency, the motion trajectory is unevenly distributed in space, presenting characteristics such as near comb and far dense. The common trajectory prediction method does not consider the size change of the moving target at different positions in the video picture, and thus cannot well reflect the actual motion of the moving target.

[0068] In the embodiment, the trajectory prediction model takes into account the prediction target of the size information of the sample target object, and can effectively solve the problems that the size of the moving target in the video picture presents large near and small far, and the motion trajectory is unevenly distributed in space, thereby improving the accuracy of trajectory prediction.

[0069] In the embodiment, the size information of the target object can be specifically represented by using the size information of the detection box surrounding the target object. It is assumed that the information of the detection box is denoted as (x, y, w, h). Wherein, x represents the horizontal coordinate of the center point of the detection box, y represents the vertical coordinate of the center point of the detection box, w represents the length of the detection box (i.e. the length of the upper boundary and the lower boundary in the detection box), and h represents the height of the detection box (i.e. the length of the left boundary and the right boundary in the detection box). The length and height of the detection box can be used to represent the size information of the detection box. Referring to Figure 3 , the detection box is a rectangular box, and of course, the shape of the detection box is not limited.

[0070] In actual applications, target detection can be performed on an image including a target object to obtain information of a detection box of the target object. For example, target detection can be performed in an artificial manner to obtain information of a detection box of the target object. For another example, target detection can be performed based on an SSD (Single Shot MultiBox Detector) detection framework or a RefineDet target detection framework to obtain information of a detection box of the target object. The SSD is a single-stage target detection algorithm, which performs feature extraction through a convolutional neural network, takes different feature layers for detection output, and is a multi-scale detection method. The RefineDet is based on a feedforward convolutional network, which generates a fixed number of bounding boxes and scores representing different categories of targets in the boxes, and produces the final result through Non-Maximum Suppression (NMS).

[0071] In this embodiment, for the convenience of distinguishing and understanding, the detection box surrounding the sample target object is referred to as a sample detection box. In the training phase, when each training sample is prepared, the first size information of the sample detection box corresponding to the sample target object at each time point in the first time period and the second size information at each time point in the second time period also need to be determined. The first size information of the sample detection box at each time point in the first time period is input into the initial trajectory prediction model to obtain the predicted size information of the sample detection box at each time point in the second time period. The fourth loss value of the training sample in the trajectory point distribution dimension can be determined according to the second size information and the predicted size information of the sample detection box at each time point in the second time period. Finally, the loss value determined according to the second real trajectory data and the predicted trajectory data and the fourth loss value are combined to adjust the model parameters of the initial trajectory prediction model. The loss value determined according to the second real trajectory data and the predicted trajectory data can be at least one of the first loss value, the second loss value, or the third loss value. In actual applications, at least two of the first loss value, the second loss value, the third loss value, and the fourth loss value can be weighted and summed, averaged, or accumulated, and the operation result is taken as the final loss value, and the model parameters of the initial trajectory prediction model are adjusted according to the final loss value.

[0072] Further optionally, in the case where the loss value determined according to the second real trajectory data and the predicted trajectory data is the first loss value, correspondingly, adjusting the model parameters of the initial trajectory prediction model according to the first loss value includes: determining the fourth loss value of the training sample in the trajectory point distribution dimension according to the second size information and the predicted size information of the sample detection box at each time point in the second time period; and adjusting the model parameters of the initial trajectory prediction model according to the first loss value and the fourth loss value.

[0073] It is worth noting that when adjusting the model parameters of the initial trajectory prediction model based on the first and fourth loss values, the first and fourth loss values ​​can be weighted, averaged, or summed, and the result can be used as the final loss value. The model parameters of the initial trajectory prediction model can then be adjusted based on the final loss value.

[0074] In this embodiment, when determining the fourth loss value of the training sample in the trajectory point distribution dimension, the second size information of the sample detection box at each time point in the second time period can be used as the true value and input into the loss function, and the second size information of the sample detection box at each time point in the second time period can be used as the predicted value and input into the loss function. The loss function determines the fourth loss value of the training sample in the trajectory point distribution dimension based on the input true value and predicted value.

[0075] Further, optionally, when the trajectory prediction model predicts not only trajectory information but also target size, the size information of the target object in the sample can be fused when determining the third loss value of the training sample in the trajectory distance dimension. Taking formula (3) as an example, tr p =[x p ,y p ,w p ,h p ], where tr p In addition to including the actual position coordinates of the actual trajectory points [x], the data also includes the actual position coordinates of the actual trajectory points. p ,y p It also incorporates the size information of the sample target object at the actual trajectory point. p ,h p Similarly, tr y =[x y ,y y ,w y ,h y ], tr y In addition to the predicted position coordinates of the predicted trajectory points [x], y ,y y It also incorporates the size information of the sample target object at the predicted trajectory points. y ,h y ].

[0076] In practical applications, the information of any predicted trajectory point can be represented as [x, y, w, h], which includes the position coordinates [x, y] and size information [w, h] of the predicted trajectory point. Furthermore, to improve trajectory prediction, a trajectory point distribution representation method is used to characterize the information of the predicted trajectory point. Specifically, the position coordinates [x, y] of the predicted trajectory point are set to follow a probability density function f(x, y; θ). x,y) ; the target size information [w, h] of the predicted trajectory point obeys a probability distribution with a probability density function f(w, h; θ w,h ) ; θ x,y and θ w,h are probability distribution parameters, so that the trajectory prediction model outputs are not specific position coordinates [x, y] and size information [w, h], but probability distribution parameters θ x,y and θ w,h .

[0077] Taking a two-dimensional normal distribution as an example, the predicted trajectory point information tr p at time p is represented as a two-dimensional normal distribution P(x, y) ~ N(μ x,y ,∑ x,y ), and the probability density function is: The probability distribution parameters θ x,y include the mean μ x,y and the covariance ∑ x,y , where

[0078] μ x is the mean of the position coordinate x, μ y is the mean of the position coordinate y, σ y , σ x , σ xy , etc. are standard deviations related to the position coordinates, and ρ is the correlation coefficient. By analogy, the related parameters of the two-dimensional normal distribution of the size information [w, h] of the predicted trajectory point can be calculated.

[0079] Continuing to refer to Figure 3 , the position coordinates of the predicted trajectory point information tr p at time p are represented as [μ x , μ y , σ x , σ y ], and the size information is represented as [μ w ± σ w , μ h ± σ h ], μ w is the mean of the width w in the size, μ h is the mean of the height h in the size, σ w is the standard deviation of the width w in the size; and σ h is the standard deviation of the height h in the size.

[0080] Further optionally, the implementation manner of determining the fourth loss value of the training sample in the trajectory point distribution dimension according to the second size information of the sample detection frame at each time in the second time period and the predicted size information is: in the case that the predicted position coordinates and the predicted size information at each time in the second time period are represented by the probability distribution parameter, the first probability density function corresponding to each time in the second time period is determined, which takes the position coordinates as the independent variable, and the second probability density function corresponding to each time in the second time period is determined, which takes the size information as the independent variable; the fourth loss value of the training sample in the trajectory point distribution dimension is determined according to the real position coordinates, the real size information, the first probability density function and the second probability density function of the real trajectory point at each time in the second time period.

[0081] Specifically, the loss value of each time in the trajectory point distribution dimension can be calculated based on the real position coordinates, the real size information, the first probability density function and the second probability density function of the real trajectory point at each time in the second time period, the loss values of each time in the trajectory point distribution dimension are summed, averaged or accumulated, and the operation result is taken as the fourth loss value of the training sample in the trajectory point distribution dimension.

[0082] For example, the loss value of each time in the trajectory point distribution dimension is calculated by formula (4).

[0083]

[0084] wherein, L Distribution represents the loss value of each time in the trajectory point distribution dimension, represents the loss value of each time in the position coordinates in the trajectory point distribution dimension, represents the loss value of each time in the size information in the trajectory point distribution dimension. (x,y),p represents the probability density value output by the first probability density function in the case that the independent variables x and y take the real position coordinates of the real trajectory point at this time, and the probability distribution parameter is θ (x,y),p which has the maximum prediction probability at this time; f(w, h; θ (w,h),p ) represents the probability density value output by the second probability density function in the case that the independent variables w and h take the real size information at this time, and the probability distribution parameter is θ (w,h),p which has the maximum prediction probability at this time.

[0085] After the trajectory prediction model is trained, the trajectory prediction model can be used for trajectory prediction, for example, vehicle trajectory, pedestrian trajectory and non-motor vehicle trajectory, but is not limited thereto. Therefore, the embodiment of the present application also provides a trajectory data processing method. Figure 4 The flow chart of the trajectory data processing method provided by the embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, the trajectory data processing method provided by the embodiment of the present application includes the following steps. Figure 3The method can comprise the following steps:

[0086] 401. Obtain historical trajectory data of the target object in a historical time period.

[0087] 402. Input the historical trajectory data into the trajectory prediction model to obtain future trajectory data of the target object in a future time period predicted by the trajectory prediction model; the future trajectory data comprises predicted position coordinates and predicted probabilities of predicted trajectory points in each time point in the future time period in multiple direction categories.

[0088] The technical scheme provided by the embodiments of the present application can predict the information of the trajectory points in multiple direction categories in each time point in the future time period, achieve the trajectory prediction target of multiple time point and multiple trajectory distribution, and effectively improve the accuracy of trajectory prediction. In addition, predicting the trajectory in a future period of time contains more abundant information than predicting a single trajectory point, and the abundant information can help improve the accuracy of anomaly detection.

[0089] In some optional embodiments, after predicting the future trajectory data of the target object in the future time period, it can be determined whether trajectory anomaly occurs in each time point in the future time period. Based on this, the above method can further comprise: for any target time point in the future time period, determining distance information between the real trajectory point of the target time point and the predicted trajectory points of each direction category according to the real position coordinates of the real trajectory point of the target time point and the predicted position coordinates of the predicted trajectory points in the multiple direction categories; and determining whether trajectory anomaly occurs in the target time point according to each distance information.

[0090] In order to facilitate understanding and distinction, any time point in the future time period is referred to as a target time point. For any target time point, the distance information between the real trajectory point of the target time point and the predicted trajectory point of any direction category can be the absolute value, the square difference or the L2 norm between the real position coordinates of the real trajectory point and the real position coordinates of the predicted trajectory point of the direction category, and this is not limited.

[0091] In this embodiment, the determination of whether a trajectory anomaly occurs at a target time is based on various distance information. For example, the various distance information is accumulated to obtain total distance information. This total distance information is compared with a flexibly set distance threshold. If the total distance information is greater than the distance threshold, it indicates that a trajectory anomaly has occurred at that target time, and the average probability of each predicted probability is taken as the trajectory anomaly probability at that time. If the total distance information is not greater than the distance threshold, it indicates that no trajectory anomaly has occurred at that target time. Alternatively, the various distance information is weighted and summed to obtain weighted distance information. This weighted distance information is compared with a flexibly set distance threshold. If the weighted distance information is greater than the distance threshold, it indicates that a trajectory anomaly has occurred at that target time, and the average probability of each predicted probability is taken as the trajectory anomaly probability at that time. If the weighted distance information is not greater than the distance threshold, it indicates that no trajectory anomaly has occurred at that target time. Optionally, to more accurately determine trajectory anomalies, the method for determining whether a trajectory anomaly occurred at a target time based on various distance information is as follows: First, determine if there is a target distance greater than a distance threshold among the various distance information. If no target distance information is found, then determine that a trajectory anomaly occurred at the target time, and determine the probability of a trajectory anomaly at the target time based on multiple predicted probabilities corresponding to each direction category at the target time. If target distance information is found, then determine that no trajectory anomaly occurred at the target time.

[0092] For ease of understanding, assume the information representation of the predicted trajectory point is [x,y,w,h], which includes the position coordinates [x,y] and size information [w,h] of the predicted trajectory point. The number of direction categories is denoted as J, where J is a positive integer, j∈[0,J-1]. Taking J as 5 as an example, j can be positive integers such as 0, 1, 2, 3, 4, etc. Formula (5) can be used to calculate the trajectory anomaly probability under the j-th direction category at the target time.

[0093]

[0094] In formula (5), This represents the predicted probability of the predicted trajectory point of the j-th direction category at the target time. This represents the probability of trajectory anomaly in the j-th direction category at time 1; Tr represents the position coordinates of the predicted trajectory point of the j-th direction category at the target time; y,p This represents the position coordinates of the actual trajectory point at the target time. This represents the distance information between the actual trajectory point at the target time and the predicted trajectory point of the j-th direction category; Δ is the distance threshold that can be flexibly set as needed.

[0095] Formula (5) means that if but The value is Otherwise, The value of the target distance information is 0.

[0096] It is worth noting that the maximum prediction probability or the minimum prediction probability can be selected from the multiple prediction probabilities corresponding to each direction category of the target time as the trajectory anomaly probability of the target time, or the weighted prediction probability obtained by weighting and summing the multiple prediction probabilities is taken as the trajectory anomaly probability of the target time, and no limitation is made to this.

[0097] It is worth noting that if one or more direction categories of the target time satisfy This condition, it is determined that no trajectory anomaly occurs at the target time, that is, the trajectory of the target time is normal.

[0098] Further optionally, in order to further improve the accuracy of trajectory anomaly judgment, if it is judged that there is no target distance information, before determining that the trajectory anomaly occurs at the target time, the intersection over union between the real detection frame and the predicted detection frame of the target object at the target time can also be determined based on the real size information of the real detection frame and the predicted size information of the predicted detection frame of the target object at the target time; and it is determined that the intersection over union is less than an intersection over union threshold.

[0099] Wherein, the intersection over union (Intersection over Union, IOU) is the overlap rate between the real detection frame and the predicted detection frame; the intersection over union threshold is flexibly set according to actual needs. If the intersection over union is greater than the intersection over union threshold, it means that the real detection frame and the predicted detection frame have a high degree of coincidence, and the probability of abnormal trajectory at the target time is small; if the intersection over union is less than the intersection over union threshold, it means that the real detection frame and the predicted detection frame have a low degree of coincidence, and the probability of abnormal trajectory at the target time is large.

[0100] In this embodiment, if it is judged that there is no target distance information, and the intersection over union is less than the intersection over union threshold, the probability of abnormal trajectory at the target time is large, at this time it can be determined that the trajectory anomaly occurs at the target time. If it is judged that there is no target distance information, and the intersection over union is greater than the intersection over union threshold, the probability of abnormal trajectory at the target time is small, at this time it can be determined that no trajectory anomaly occurs at the target time.

[0101] For ease of understanding, formula (6) is explained. Compared with formula (5), formula (6) constrains the intersection over union condition.

[0102]

[0103] Wherein, The predicted size information of the predicted detection frame of the target time, Tr y,p′ The real size information of the real detection frame of the target time; wherein, This represents the cross-union ratio (CUP) between the predicted detection boxes and the ground truth detection boxes, where β is the CUP threshold.

[0104] Formula (6) means that if as well as If these two conditions are met, then The value is Otherwise, if as well as If either of these two conditions is not met, then The value of is 0.

[0105] In some alternative embodiments, if the information of the predicted trajectory points is represented by probability distribution parameters, then the trajectory anomaly probability in the j-th direction category at the target time can be calculated using formula (7).

[0106]

[0107] in, This represents the probability density value output by the first probability density function, which is the probability density value under the target time and the j-th direction category θ. (x,y) The first probability density function is the parameter of the probability distribution. The function value output by the first probability density function is when the independent variables x and y of the first probability density function take the true position coordinates of the true trajectory point at the target time.

[0108] Where, α x,y The threshold value for probability density is set flexibly.

[0109] Formula (7) means that if but The value is Otherwise, The value of is 0.

[0110] In some optional embodiments, if the information of the predicted trajectory points is represented by probability distribution parameters, then the trajectory anomaly probability in the j-th direction category at the target time can be calculated using formula (8).

[0111]

[0112] in, This represents the probability density value output by the second probability density function, which is the probability density value under the target time and the j-th direction category θ. w,h The first probability density function is the probability distribution parameter. The second probability density function outputs the function value when the independent variables w and h of the second probability density function take the values ​​of the true size information at the target time.

[0113] Where, α w,ha probability density value threshold is flexibly set.

[0114] Formula (8) indicates that if and both conditions are met, the value of is Otherwise, the value of is 0.

[0115] In order to better understand the technical solutions provided by the embodiments of the present application, in combination with Figure 5 a specific scene embodiment is introduced.

[0116] In the urban road scene, the field of view range of a camera on a road intersection or a road section usually contains multiple lanes and driving information of vehicles in different directions. Efficiently extracting and utilizing road condition information is of great significance for optimizing vehicle trajectory prediction and trajectory anomaly detection in complex scenarios. Conventional trajectory prediction methods cannot well solve the problems encountered in actual scenarios.

[0117] Firstly, the camera is usually erected at a certain height and angle, which causes the moving target on the video picture to present differently at different positions, and the vehicle target size usually presents large near and small far; under a fixed sampling frequency, the trajectory is unevenly distributed in space, presenting characteristics such as near comb and far dense. Conventional trajectory prediction methods do not consider the size change of the target at different positions on the video picture, and thus cannot well reflect the actual movement of the moving target. Some other methods try to restore the real vehicle target size by using affine transformation, but this calculation needs to know the camera erection parameters in advance, which is difficult to implement in complex urban traffic scenarios.

[0118] Secondly, in the real scene, the road has a certain width, so the movement of the vehicle target has a large range of variation, which is different on different roads at the same intersection, or even at different distances on the same road. Conventional trajectory prediction methods cannot accurately describe the width and other information of the road; in addition, in the conventional trajectory anomaly detection method, the manual setting of the threshold value cannot adapt to the complex and variable road conditions. In trajectory prediction and trajectory anomaly detection, more attention is usually paid to whether the vehicle target is driving in the correct direction, and the speed of driving may have a relatively small influence, and the conventional trajectory prediction method does not pay attention to this problem.

[0119] Thirdly, in the urban road scene, especially at the intersection, the movement of the vehicle at the next moment has multiple possibilities, which can be stationary (waiting for the red light), forward, and even on the mixed road, it can change lanes, turn left, turn right, or even U-turn. Conventional trajectory prediction methods usually add more prior information for such complex trajectory prediction, and cannot flexibly adapt to different roads or even different positions.

[0120] Finally, in practical applications, due to the complexity of the application scene and the complexity of the road situation, trajectory data or road data labeling is a very time-consuming and labor-intensive task, and how to quickly use the trajectory prediction model in new scenes and new roads is a great challenge.

[0121] To this end, a self-supervised trajectory prediction method is proposed. Referring to Figure 5 In the model training stage, first, video data is collected by a camera. Specifically, the camera can be set on a city road, can be a drone, or can be set on a vehicle, and no limitation is made to this. Second, trajectory extraction is performed on the video data. The extracted vehicle trajectory data can be used as training data for model training to obtain a trajectory prediction model, i.e., a trajectory model.

[0122] Referring to Figure 5 In the prediction stage, video data is collected by a camera. Trajectory extraction is performed on the video data. The extracted vehicle trajectory data is used as prediction data, the prediction data is input into the trajectory model for trajectory prediction, and abnormality judgment is performed based on the trajectory prediction result.

[0123] It is worth noting that in the model training stage, for any training sample, the loss value of the training sample can be determined from the trajectory position probability dimension, the trajectory direction dimension, the trajectory distance dimension, and the trajectory point distribution dimension, and the model parameters are adjusted based on the loss values, thereby obtaining a trajectory prediction model with better model performance.

[0124] In the model training stage, in addition to predicting the information of the trajectory point at each time, the trajectory prediction model also predicts the size information of the vehicle at each time.

[0125] The self-supervised trajectory prediction method has the following advantages:

[0126] First, to address the problem that the size of the vehicle target on the video screen presents near large and far small, and the vehicle trajectory is not uniformly distributed in space, the technical solution provided by the embodiments of the present application directly uses the size of the vehicle target as the target to be predicted, and learns the scene-related information in a self-supervised manner during the model training process, without the need for camera calibration and the like.

[0127] Secondly, in view of the fact that the conventional trajectory prediction method cannot accurately describe the width of the road and the like in a real scene, and needs to use an artificial rule setting method to cope with the complex and changeable road, the trajectory prediction result of the technical solution provided in the embodiment of the application is represented by the trajectory point position distribution and the trajectory point target box size distribution. In this way, the problem of predicting the trajectory range can be effectively solved. Moreover, the distribution is automatically learned according to the road conditions, without the need for complex rule setting. Thirdly, in view of the problem of the changeable driving direction of the vehicle and the multiple lanes, the technical solution provided in the embodiment of the application simultaneously predicts the possible trajectory point situation in multiple directions at each time; in addition, the technical solution provided in the embodiment of the application predicts the trajectory in a future period of time, which contains more rich information than predicting a single trajectory point, such as the direction information of the trajectory. The trajectory information representation proposed in the technical solution provided in the embodiment of the application more effectively reflects the real trajectory situation, and solves the complex target motion situation in the complex road scene.

[0128] Finally, in view of the above considerations, in order to more effectively realize the iteration of the prediction model, the technical solution provided in the embodiment of the application proposes a model training loss calculation method, including a trajectory position probability loss, a trajectory distribution loss, a trajectory distance loss and a trajectory direction loss, which can more effectively guide the iteration optimization of the model. In the technical solution provided in the embodiment of the application, in the training stage, the trajectory in the future period of time is predicted using the trajectory in the historical period of time, so as to complete the self-supervised trajectory model training without the need for manual annotation; this method can quickly adapt to the trajectory prediction and trajectory anomaly detection in different scenes and under different road conditions.

[0129] Figure 6 A structure schematic diagram of a trajectory prediction model generation device provided in the embodiment of the application is shown in FIG. 1. Figure 6 As shown in the figure, the device can include:

[0130] The acquisition module 61 is configured to acquire a training sample, the training sample including first real trajectory data of a sample target object in a first time period and second real trajectory data in a second time period, the first time period being a time period preceding the second time period.

[0131] The prediction module 62 is configured to input the first real trajectory data into an initial trajectory prediction model to obtain predicted trajectory data of the sample target object in the second time period; wherein the predicted trajectory data includes predicted position coordinates and predicted probabilities of the predicted trajectory points in multiple direction categories at each time in the second time period.

[0132] The adjusting module 63 is configured to adjust model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data, so as to obtain a trajectory prediction model used for trajectory prediction.

[0133] Further, when adjusting the model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data, the adjusting module 63 is specifically configured to: determine real probabilities of real trajectory points in each direction category at each time in the second time period according to the second real trajectory data; calculate a first loss value of the training sample in the trajectory position probability dimension according to the real probabilities of the real trajectory points in each direction category at each time in the second time period and the predicted probabilities of the predicted trajectory points in each direction category at each time; and adjust the model parameters of the initial trajectory prediction model according to the first loss value.

[0134] Further, when determining the real probabilities of the real trajectory points in each direction category at each time in the second time period according to the second real trajectory data, the adjusting module 63 is specifically configured to: for any first time in the second time period, obtain real position coordinates of the real trajectory point at the first time from the second real trajectory data, and obtain real position coordinates of the real trajectory point at the second time from the second real trajectory data or the first real trajectory data, the second time being the previous time of the first time; determine a target relative position relationship between the real trajectory point at the first time and the real trajectory point at the second time according to the real position coordinates of the real trajectory point at the first time and the real position coordinates of the real trajectory point at the second time; and determine the real probabilities of the real trajectory point at the first time in each direction category according to the target relative position relationship.

[0135] Further, when adjusting the model parameters of the initial trajectory prediction model according to the first loss value, the adjusting module 63 is specifically configured to: determine a second loss value of the training sample in the trajectory direction dimension according to the real position coordinates of the real trajectory point at each time in the second time period and the predicted position coordinates of the predicted trajectory point with the maximum predicted probability; or determine a third loss value of the training sample in the trajectory distance dimension according to the real position coordinates of the real trajectory point at each time in the second time period and the predicted position coordinates of the predicted trajectory point with the maximum predicted probability; and adjust the model parameters of the initial trajectory prediction model according to the first loss value and in combination with at least one of the second loss value and the third loss value.

[0136] Further, when the adjustment module 63 determines the second loss value of the training sample in the trajectory direction dimension according to the real position coordinates of the real trajectory points at each time in the second time period and the predicted position coordinates of the predicted trajectory points with the maximum prediction probability at each time in the second time period, the adjustment module 63 is specifically configured to: generate a first trajectory vector according to the real position coordinates of the real trajectory points at each time in the second time period; generate a second trajectory vector according to the predicted position coordinates of the predicted trajectory points with the maximum prediction probability at each time in the second time period; and determine the second loss value of the training sample in the trajectory direction dimension according to the cosine similarity between the first trajectory vector and the second trajectory vector.

[0137] Further, when the adjustment module 63 determines the third loss value of the training sample in the trajectory distance dimension, the adjustment module 63 is specifically configured to: determine the absolute value and / or squared difference between the real position coordinates of the real trajectory points at each time in the second time period and the predicted position coordinates of the predicted trajectory points with the maximum prediction probability at each time in the second time period; and determine the third loss value of the training sample in the trajectory distance dimension according to the absolute value and / or squared difference at each time in the second time period.

[0138] Further, the training sample further includes: first size information of the sample detection frame corresponding to the sample target object at each time in the first time period and second size information of the sample detection frame at each time in the second time period; and the prediction module 62 is further configured to: input the first size information of the sample detection frame at each time in the first time period into the initial trajectory prediction model to obtain predicted size information of the sample detection frame at each time in the second time period; and correspondingly, when the adjustment module 63 adjusts the model parameters of the initial trajectory prediction model according to the first loss value, the adjustment module 63 is specifically configured to: determine a fourth loss value of the training sample in the trajectory point distribution dimension according to the second size information and the predicted size information of the sample detection frame at each time in the second time period; and adjust the model parameters of the initial trajectory prediction model according to the first loss value and the fourth loss value.

[0139] Further, when the adjustment module 63 determines the fourth loss value of the training sample in the trajectory point distribution dimension according to the second size information and the predicted size information of the sample detection frame at each time in the second time period, the adjustment module 63 is specifically configured to: in the case that the predicted position coordinates and the predicted size information at each time in the second time period are represented by probability distribution parameters, determine a first probability density function with the position coordinates as the independent variable and a second probability density function with the size information as the independent variable corresponding to each time in the second time period; and determine the fourth loss value of the training sample in the trajectory point distribution dimension according to the real position coordinates, the real size information, the first probability density function and the second probability density function of the real trajectory points at each time in the second time period.

[0140] Figure 6 The trajectory prediction model generation apparatus shown in the embodiments can perform Figure 2The implementation principle and technical effects of the generation method of the trajectory prediction model of the illustrated embodiment will not be described again. The specific manner in which each module, unit in the generation device of the trajectory prediction model in the above-described embodiment performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0141] Figure 7 A structural schematic diagram of a trajectory data processing device provided by an embodiment of the present application is shown. As shown in the figure, the device can include: Figure 7

[0142] The acquisition module 71 is configured to acquire historical trajectory data of the target object in a historical time period.

[0143] The prediction module 72 is configured to input the historical trajectory data into the trajectory prediction model to obtain future trajectory data of the target object in a future time period predicted by the trajectory prediction model; the future trajectory data includes predicted position coordinates and predicted probabilities of predicted trajectory points in each direction category at each time in the future time period.

[0144] Further optionally, the device further includes an anomaly judgment module 73 configured to: for any target time in the future time period, determine distance information between a real trajectory point of the target time and predicted trajectory points of each direction category according to real position coordinates of the real trajectory point of the target time and the predicted position coordinates of the predicted trajectory points in the multiple direction categories; and determine whether a trajectory anomaly occurs at the target time according to each distance information.

[0145] Further optionally, when the anomaly judgment module 73 determines whether a trajectory anomaly occurs at the target time according to each distance information, the anomaly judgment module 73 is specifically configured to: determine whether there is target distance information greater than a distance threshold in each distance information; if it is determined that there is no target distance information, it is determined that a trajectory anomaly occurs at the target time, and a trajectory anomaly probability of the target time is determined according to the multiple predicted probabilities corresponding to each direction category of the target time.

[0146] Further optionally, before determining that a trajectory anomaly occurs at the target time, the anomaly judgment module 73 is further configured to: determine an intersection over union between a real bounding box of the target object at the target time and a predicted bounding box based on real size information of the real bounding box and predicted size information of the predicted bounding box; and determine that the intersection over union is less than an intersection over union threshold.

[0147] Figure 7 The trajectory data processing device shown can perform Figure 4 ​The trajectory data processing method of the illustrated embodiment, its implementation principle and technical effect will not be repeated. For the specific manner in which each module, unit in the trajectory data processing device in the above embodiment performs the operation has been described in detail in the embodiment related to the method, which will not be described in detail here.

[0148] It should be noted that the execution subject of each step of the method provided by the above embodiment can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 201 to 203 can be device A; for example, the execution subject of steps 201 and 202 can be device A, and the execution subject of step 203 can be device B; and the like.

[0149] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clear that these operations can be executed or executed in parallel without the order in which they appear in this text. The serial numbers of the operations such as 201, 202, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second" and the like in this text are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. Also, "first" and "second" are not different types.

[0150] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. As shown in the figure, Figure 8 The electronic device includes a memory 81 and a processor 82;

[0151] The memory 81 is used to store computer programs and can be configured to store other various data to support operations on the computing platform. Examples of these data include instructions for any application or method operating on the computing platform, contact data, phonebook data, messages, pictures, videos, etc.

[0152] The memory 81 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0153] The processor 82 is coupled with the memory 81 and used for executing a computer program in the memory 81, so as to: acquire a training sample, the training sample including first real trajectory data of a sample target object in a first time period and second real trajectory data in a second time period, the first time period being a time period before the second time period; input the first real trajectory data into an initial trajectory prediction model to obtain predicted trajectory data of the sample target object in the second time period; wherein the predicted trajectory data includes predicted position coordinates and predicted probabilities of predicted trajectory points in multiple direction categories at each time in the second time period; and adjust model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data, to obtain a trajectory prediction model for trajectory prediction; or,

[0154] The processor 82 is coupled with the memory 81 and used for executing a computer program in the memory 81, so as to: acquire a training sample, the training sample including first real trajectory data of a sample target object in a first time period and second real trajectory data in a second time period, the first time period being a time period before the second time period; input the first real trajectory data into an initial trajectory prediction model to obtain predicted trajectory data of the sample target object in the second time period; wherein the predicted trajectory data includes predicted position coordinates and predicted probabilities of predicted trajectory points in multiple direction categories at each time in the second time period; and adjust model parameters of the initial trajectory prediction model according to the second real trajectory data and the predicted trajectory data, to obtain a trajectory prediction model for trajectory prediction; or,

[0155] Further, as shown in Figure 8 , the electronic device further includes a communication component 83, a display 84, a power supply component 85, an audio component 86 and other components. Figure 8 Some components are only schematically shown in the electronic device, and it does not mean that the electronic device only includes Figure 8 the components shown. In addition, Figure 8 the components in the dashed box are optional components, not mandatory components, and the specific implementation depends on the product form of the electronic device. The electronic device of the embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone or an IOT device, or a server device such as a conventional server, a cloud server or a server array. If the electronic device of the embodiment is implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone or an IOT device, it can include Figure 8 the components in the dashed box; if the electronic device of the embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it can not include Figure 8 the components in the dashed box.

[0156] The detailed implementation process of the processor performing each action can be referred to the related description in the foregoing method embodiments or device embodiments, which will not be described here.

[0157] Correspondingly, the embodiment of the application also provides a computer readable storage medium storing a computer program, which can implement each step that can be executed by the electronic device in the foregoing method embodiments when the computer program is executed.

[0158] Accordingly, the embodiments of the present application also provide a computer program product, including computer programs / instructions, which, when executed by a processor, enable the processor to implement each step of the above-mentioned method embodiments that can be performed by an electronic device.

[0159] The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, a 2G, 3G, 4G / LTE, 5G, or the like mobile communication network, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0160] The display includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect a duration and a pressure related to a touch or a slide operation.

[0161] The power supply component provides power to various components of the device where the power supply component is located. The power supply component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0162] The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device where the audio component is located is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in a memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0163] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0164] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for functionally implementing the one or more functions specified in the flowchart block or blocks.

[0165] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for functionally implementing the one or more functions specified in the flowchart block or blocks.

[0166] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 Figure 1 means for functionally implementing the one or more functions specified in the flowchart block or blocks.

[0167] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0168] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0169] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0170] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0171] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for generating a trajectory prediction model, characterized in that, include: Acquire training samples, which include first real trajectory data of the target object in a first time period and second real trajectory data in a second time period. The first time period is the time period preceding the second time period. The first real trajectory data includes the real position coordinates of the target object's real trajectory points at each moment in the first time period, and the second real trajectory data includes the real position coordinates of the target object's real trajectory points at each moment in the second time period. The training samples also include: the first size information of the sample detection box corresponding to the sample target object at each moment in the first time period, and the second size information at each moment in the second time period; The first real trajectory data is input into the initial trajectory prediction model to obtain the predicted trajectory data of the sample target object in the second time period; wherein, the predicted trajectory data includes the predicted position coordinates and predicted probabilities of the predicted trajectory points under multiple directional categories at each moment in the second time period; Based on the true probability of the true trajectory point at each moment in the second time period determined by the second true trajectory data under each directional category and the predicted probability of the predicted trajectory point at each moment in the second time period under each directional category in the predicted trajectory data, the first loss value of the training sample in the trajectory position probability dimension is calculated. The first size information of the sample detection box at each time point within the first time period is input into the initial trajectory prediction model to obtain the predicted size information of the sample detection box at each time point within the second time period; based on the second size information and the predicted size information of the sample detection box at each time point within the second time period, the fourth loss value of the training sample in the trajectory point distribution dimension is determined. Based on the first loss value and the fourth loss value, the model parameters of the initial trajectory prediction model are adjusted to obtain a trajectory prediction model for trajectory prediction. The trajectory prediction model is used to predict the predicted position coordinates and predicted probabilities of predicted trajectory points under multiple direction categories at each time in the future time period. The direction category reflects the relative positional relationship between the predicted trajectory point at the current time and the set predicted trajectory point at the previous time.

2. The method according to claim 1, characterized in that, Based on the second true trajectory data, determine the true probability of the true trajectory point at each moment within the second time period under each directional category, including: For any first moment within the second time period, the true position coordinates of the true trajectory point at the first moment are obtained from the second true trajectory data, and the true position coordinates of the true trajectory point at the second moment are obtained from the second true trajectory data or the first true trajectory data, wherein the second moment is the moment preceding the first moment; Based on the true position coordinates of the true trajectory points at the first time and the true position coordinates of the true trajectory points at the second time, determine the target relative position relationship between the true trajectory points at the first time and the true trajectory points at the second time. Based on the relative positional relationship of the targets, determine the true probability of the true trajectory point at the first moment in each directional category.

3. The method according to claim 1, characterized in that, Based on the first loss value and the fourth loss value, the model parameters of the initial trajectory prediction model are adjusted, including: Based on the actual position coordinates of the real trajectory points at each moment within the second time period and the predicted position coordinates of the predicted trajectory point with the highest prediction probability, determine the second loss value of the training sample in the trajectory direction dimension; or... Based on the real position coordinates of the real trajectory points at each time point in the second time period and the predicted position coordinates of the predicted trajectory point with the highest prediction probability, the third loss value of the training sample in the trajectory distance dimension is determined. The model parameters of the initial trajectory prediction model are adjusted based on the first loss value and the fourth loss value, and in combination with at least one of the second loss value and the third loss value.

4. The method according to claim 3, characterized in that, Based on the actual position coordinates of the real trajectory points at each time point within the second time period and the predicted position coordinates of the predicted trajectory point with the highest prediction probability, the second loss value of the training sample in the trajectory direction dimension is determined, including: The first trajectory vector is generated based on the actual position coordinates of the actual trajectory points at each time point within the second time period; A second trajectory vector is generated based on the predicted position coordinates of the predicted trajectory point with the highest prediction probability at each moment within the second time period; The second loss value of the training sample in the trajectory direction dimension is determined based on the cosine similarity between the first trajectory vector and the second trajectory vector.

5. The method according to claim 3, characterized in that, Determining the third loss value of the training sample in the trajectory distance dimension includes: Determine the absolute value and / or squared difference between the true position coordinates of the true trajectory points at each time point in the second time period and the predicted position coordinates of the predicted trajectory point with the highest prediction probability; Based on the absolute value and / or the squared difference at each moment within the second time period, the third loss value of the training sample in the trajectory distance dimension is determined.

6. The method according to claim 1, characterized in that, Based on the second size information and predicted size information of the sample detection box at each time point within the second time period, the fourth loss value of the training sample in the trajectory point distribution dimension is determined, including: When the predicted position coordinates and predicted size information at each time point in the second time period are represented by probability distribution parameters, the first probability density function with position coordinates as independent variables and the second probability density function with size information as independent variables are determined for each time point in the second time period. Based on the true position coordinates and true size information of the true trajectory points at each time point within the second time period, as well as the first probability density function and the second probability density function, the fourth loss value of the training sample in the trajectory point distribution dimension is determined.

7. The method according to claim 1, characterized in that, The first real trajectory data or the second real trajectory data is obtained by processing video data, including the sample target object, based on a trajectory tracking algorithm; Alternatively, the trajectory prediction model may be a machine learning model or a neural network model.

8. A trajectory data processing method, characterized in that, include: Obtain historical trajectory data of the target object within a historical time period; The historical trajectory data is input into the trajectory prediction model to obtain the future trajectory data of the target object in a future time period predicted by the trajectory prediction model; the future trajectory data includes the predicted position coordinates and predicted probabilities of the predicted trajectory points under multiple directional categories at each moment in the future time period; The trajectory prediction model is trained using the method described in any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, Also includes: For any target time within the future time period, based on the actual position coordinates of the target object's actual trajectory points at the target time and the predicted position coordinates of the predicted trajectory points under multiple directional categories, the distance information between the actual trajectory points at the target time and the predicted trajectory points under each directional category is determined. Based on the distance information, determine whether the target has experienced trajectory anomalies at any given time.

10. The method according to claim 9, characterized in that, Determine whether the target trajectory anomaly occurs at any given time based on various distance information, including: Determine whether any of the distance information contains a target distance greater than a distance threshold. If it is determined that the target distance information does not exist, then it is determined that the target time has an abnormal trajectory, and the trajectory abnormality probability of the target time is determined according to multiple predicted probabilities corresponding to each direction category of the target time.

11. The method according to claim 10, characterized in that, Before determining that the trajectory anomaly occurred at the target time, the process also includes: Based on the actual size information of the actual detection box of the target object at the target time and the predicted size information of the predicted detection box, the intersection-over-union ratio between the actual detection box and the predicted detection box of the target object at the target time is determined; Determine that the cross-union ratio is less than the cross-union ratio threshold.

12. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is coupled to the memory for executing the computer program to perform the steps of the method according to any one of claims 1-11.

13. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-11.

Images