Model training and trajectory prediction method and device, medium and equipment

By model training on historical video data and using perturbation parameters to optimize the prediction model, the problem of insufficient prediction accuracy in the fields of autonomous driving and other fields is solved, and accurate motion trajectory prediction in the case of disturbance is achieved.

CN119007088AInactive Publication Date: 2024-11-22ZHEJIANG LAB
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411478721.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-22
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing trajectory prediction technologies, such as Kalman filters or particle filters, have poor prediction results and are difficult to achieve accurate motion trajectory prediction in fields such as autonomous driving.

Method used

Through the model training method, historical video data is obtained, the initial coordinate sequence of the target object is determined, and virtual adversarial training is carried out through perturbation parameters to optimize the anti-perturbation ability of the prediction model and improve prediction accuracy.

Benefits of technology

During the training of the prediction model, by adding perturbations to the initial coordinate sequence of the target object, the anti-interference ability of the model can be enhanced, so that the movement trajectory of the target object can be accurately predicted under the disturbance changes, and the performance of the prediction model is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007088A_ABST
    Figure CN119007088A_ABST
Patent Text Reader

Abstract

The invention discloses a model training and trajectory prediction method and device, a medium and equipment. Comprising the following steps: acquiring historical video data and determining a trajectory prediction target object; and determining an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame of image in the historical video data. And determining an initial coordinate sequence after disturbance according to preset disturbance parameters and the initial coordinate sequence. And for each round of training, inputting the disturbed initial coordinate sequence into a to-be-trained prediction model to obtain a prediction coordinate sequence output by the prediction model, determining a first loss value according to a first difference between the prediction coordinate sequence and an actual coordinate sequence of the target object, and determining a second loss value according to the initial coordinate sequence and the disturbed initial coordinate sequence. A second loss value is determined. And determining a comprehensive loss value corresponding to the round of training according to the first loss value and the second loss value, so as to train the to-be-trained prediction model according to the comprehensive loss value corresponding to the round of training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, medium and equipment for model training and trajectory prediction. Background Art

[0002] At present, in video analysis systems, target detection and trajectory prediction are very important research directions, and have important application value in the fields of intelligent monitoring, autonomous driving, and robot navigation. Among them, target detection is used to identify the target object in the video stream and determine the position of the target object. Trajectory prediction is used to predict the position and motion trajectory of the target object in the next period of time based on the position of the target object for a period of time.

[0003] Taking autonomous driving as an example, vehicles can use target detection technology to determine the position and movement trajectory of surrounding vehicles and pedestrians, and then use trajectory prediction technology to predict the possible movement trajectory and arrival position of surrounding vehicles and pedestrians, thereby achieving safe vehicle navigation.

[0004] Since target detection technology detects target objects based on events that have already occurred, there are relatively mature methods for the research of target detection technology, which can accurately detect target objects. However, trajectory prediction technology predicts events related to target objects that have not yet occurred. Existing trajectory prediction technologies, such as trajectory prediction based on Kalman filters or particle filters, do not produce ideal prediction results.

[0005] To this end, this specification provides a model training and trajectory prediction method, device, medium and equipment. Summary of the invention

[0006] This specification provides a model training and trajectory prediction method, device, medium and equipment to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions: This specification provides a model training method, including: Acquire historical video data of a preset historical period, and determine a target object for trajectory prediction in the historical video data; For each frame of the historical video data, determine the coordinates of the target object in the frame of the image as the corresponding coordinates of the target object in the frame of the image; Determining an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data; Determining the initial coordinate sequence after disturbance according to the preset disturbance parameters and the initial coordinate sequence; For each round of training, the disturbed initial coordinate sequence is input into the prediction model to be trained, and a predicted coordinate sequence of the target object in a set period after the preset historical period is obtained, which is output by the prediction model, and a first loss value is determined according to a first difference between the predicted coordinate sequence and an actual coordinate sequence of the target object in the set period, wherein the first difference is positively correlated with the first loss value, and a second loss value is determined according to the initial coordinate sequence and the disturbed initial coordinate sequence; According to the first loss value and the second loss value, the comprehensive loss value corresponding to this round of training is determined, so as to train the prediction model to be trained according to the comprehensive loss value corresponding to this round of training, and adjust the perturbation parameters used in the next round of training according to the second loss value until the preset training end conditions are met.

[0008] Optionally, determining a second loss value according to the initial coordinate sequence and the disturbed initial coordinate sequence specifically includes: Determining a first distance value according to the predicted coordinate sequence and the initial coordinate sequence; Determining a second distance value according to the predicted coordinate sequence and the initial coordinate sequence after disturbance; A second loss value is determined according to a second difference between the first distance value and the second distance value, and the second loss value is positively correlated with the second difference.

[0009] Optionally, adjusting the perturbation parameters used in the next round of training according to the second loss value specifically includes: According to a preset disturbance parameter limit value and the second loss value, the disturbance parameter to be used in the next round of training is determined, wherein the direction of the derivative of the second loss value is the same as the direction of the disturbance parameter to be used in the next round.

[0010] Optionally, determining the coordinates of the target object in the frame image specifically includes: According to a preset target detection algorithm, the bounding box coordinates of the target object in the frame image are determined as the coordinates of the target object in the frame image.

[0011] Optionally, determining the initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data specifically includes: Determining the center coordinates of the target object in each frame of the historical video data according to the bounding box coordinates of the target object in each frame of the historical video data; According to the time sequence of each frame image in the historical video data and based on each center coordinate of the target object, an initial coordinate sequence of the target object is constructed.

[0012] This specification provides a trajectory prediction method, including: Acquire time-series video data of a preset time period, and determine a target object for trajectory prediction in the time-series video data; For each frame of image in the time-series video data, determine the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; Determining an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the time-series video data; Inputting the initial coordinate sequence into a trained prediction model to obtain a predicted coordinate sequence of the target object in a set time period after the preset time period output by the prediction model, wherein the prediction model is trained by the above-mentioned model training method; The motion trajectory of the target object in the set time period is determined according to the predicted coordinate sequence.

[0013] This specification provides a model training device, including: A training acquisition module is used to acquire historical video data of a preset historical period and determine a target object for trajectory prediction in the historical video data; A first determination module is used to determine, for each frame of image in the historical video data, the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; A second determination module is used to determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data; A disturbance module, used for determining a disturbed initial coordinate sequence according to preset disturbance parameters and the initial coordinate sequence; a third determination module, for inputting the disturbed initial coordinate sequence into the prediction model to be trained for each round of training, obtaining a predicted coordinate sequence of the target object in a set period after the preset historical period output by the prediction model, and determining a first loss value according to a first difference between the predicted coordinate sequence and an actual coordinate sequence of the target object in the set period, wherein the first difference is positively correlated with the first loss value, and determining a second loss value according to the initial coordinate sequence and the disturbed initial coordinate sequence; A training module is used to determine the comprehensive loss value corresponding to the round of training according to the first loss value and the second loss value, to train the prediction model to be trained according to the comprehensive loss value corresponding to the round of training, and to adjust the perturbation parameters used in the next round of training according to the second loss value until the preset training end condition is reached.

[0014] This specification provides a trajectory prediction device, including: An acquisition module, used to acquire time-series video data of a preset time period, and determine a target object for trajectory prediction in the time-series video data; A coordinate determination module, used for determining, for each frame of image in the time-series video data, the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; A sequence determination module, used to determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the time-series video data; A prediction module, used for inputting the initial coordinate sequence into a trained prediction model to obtain a predicted coordinate sequence of the target object in a set time period after the preset time period output by the prediction model, wherein the prediction model is trained by the above-mentioned model training method; The trajectory determination module is used to determine the motion trajectory of the target object in the set time period according to the predicted coordinate sequence.

[0015] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned model training and trajectory prediction method is implemented.

[0016] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a model training and trajectory prediction method when executing the program.

[0017] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: The model training method provided in this specification obtains historical video data of a preset historical period, and determines the target object for trajectory prediction in the historical video data. For each frame of the historical video data, the coordinates of the target object in the frame are determined as the corresponding coordinates of the target object in the frame. According to the coordinates corresponding to each frame of the target object in the historical video data, the initial coordinate sequence of the target object is determined. According to the preset perturbation parameters and the initial coordinate sequence, the initial coordinate sequence after perturbation is determined. For each round of training, the initial coordinate sequence after perturbation is input into the prediction model to be trained, and the predicted coordinate sequence of the target object output by the prediction model in the set period after the preset historical period is obtained, and the first loss value is determined according to the first difference between the predicted coordinate sequence and the actual coordinate sequence of the target object in the set period, and the first difference is positively correlated with the first loss value, and the second loss value is determined according to the initial coordinate sequence and the initial coordinate sequence after perturbation. According to the first loss value and the second loss value, the comprehensive loss value corresponding to the round of training is determined, so that the prediction model to be trained is trained according to the comprehensive loss value corresponding to the round of training, and the perturbation parameters used in the next round of training are adjusted according to the second loss value until the preset training end condition is reached.

[0018] In the process of training the prediction model, additional disturbances are added to the initial coordinate sequence of the target object, and virtual adversarial training is performed on the prediction model, so that the prediction model can accurately predict the motion trajectory of the target object even if it is disturbed during the trajectory prediction process. And by adjusting the disturbance parameters added in each round of model training, the disturbance caused by the disturbance parameters is enhanced, and the trained prediction model is further made more capable of accurate prediction under the condition of disturbance changes, thereby improving the performance of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 A flowchart of a model training method provided in an embodiment of this specification; Figure 2 A flow chart of a trajectory prediction method provided in this specification; Figure 3 A schematic diagram of a model training device provided in this specification; Figure 4 A schematic diagram of a trajectory prediction device provided in this specification; Figure 5 A method corresponding to the Figure 1 and Figure 2Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0022] Figure 1 A flow chart of a model training method provided in an embodiment of this specification includes the following steps: S100: Acquire historical video data of a preset historical period, and determine a target object for trajectory prediction in the historical video data.

[0023] In this specification, the process of model training and trajectory prediction involves the processing of video data. In the embodiments of this specification, the model training and target trajectory prediction process can be performed by a server. Of course, this specification does not limit the device used to perform the model training and target trajectory prediction process. For example, personal computers and mobile terminals can also be used to perform model training and target trajectory prediction. For ease of description, the following description is based on the server as the execution subject.

[0024] In one or more embodiments of the present specification, the server obtains historical video data of a preset historical period for model training, and determines a target object for trajectory prediction in the historical video data.

[0025] It is worth noting that in this specification, historical video data can be video data of the positions and motion trajectories of surrounding vehicles and pedestrians within a period of time collected by the autonomous driving car in the autonomous driving scene. In order to avoid collisions with pedestrians and ensure the driving safety of the autonomous driving vehicle, the pedestrians around the autonomous driving car are the targets that require trajectory prediction. Historical video data can also be video data of the motion trajectories of surrounding moving objects collected by the robot within a period of time in the robot automatic navigation scene. In order to avoid accidents such as collisions between the robot and the moving objects, the moving objects are taken as targets that require trajectory prediction, and the possible motion trajectories and arrival positions of the moving objects are predicted, so as to perform safe automatic navigation for the robot. Therefore, in this specification, there is no restriction on the scenarios of the historical video data obtained by the server, and the target object can be determined according to the actual shooting conditions of the historical video data.

[0026] In this specification, considering the training efficiency of the prediction model trained based on historical video data, a preset historical period that is too long may increase the model training time and computing resource consumption, and a preset historical period that is too short may result in insufficient performance of the trained model. Therefore, for each historical video data selected for training the prediction model, historical video data of a preset historical period that is suitable for balancing the model training overhead and model performance can be selected. Of course, this specification does not limit the preset historical period, and it can be selected according to actual conditions.

[0027] For example, the server selects 10 frames of video data as the historical video data of a preset historical period, then the target to be predicted can be the video data of the set period after the video data of these 10 frames of images. If it is desired to predict the motion trajectory of the target object in the video data of the 11th to 15th frames of images based on the video data of these 10 frames of images, then the video data of the set period of time can be the video data of the 11th to 15th frames of images.

[0028] S102: For each frame of image in the historical video data, determine the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image.

[0029] In one or more embodiments of the present specification, the server determines the coordinates of the target object in each frame of the historical video data as the corresponding coordinates of the target object in the frame of the image.

[0030] Specifically, the server may determine the bounding box coordinates of the target object in the frame image according to a preset target detection algorithm. The bounding box coordinates of the target object may be expressed in the form of the center coordinates / upper left corner coordinates of the bounding box and the length and width of the bounding box. Of course, the bounding box coordinates may also be expressed in the form of the upper left corner coordinates and the lower right corner coordinates of the bounding box.

[0031] The target detection algorithm may be a You Only Look Once (YOLO) series algorithm, such as a YOLO version 8 (YOLO v8) algorithm. Of course, the target detection algorithm is not limited in this specification and may be set according to actual conditions.

[0032] Of course, in this specification, the method for determining the coordinates of the target object can also be to use an edge detection method to determine the coordinates of the target object. Alternatively, the coordinates of the target object can be determined by using an anchor frame-based method. In this specification, there is no limitation on the method for determining the coordinates of the target object, which can be set according to actual conditions.

[0033] S104: Determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data.

[0034] In one or more embodiments of the present specification, after the server determines the corresponding coordinates of the target object in each frame of the historical video data, it can determine the coordinate sequence of the target object in the historical video data based on these coordinates as the initial coordinate sequence of the target object.

[0035] Specifically, the server can determine the center coordinates of the target object corresponding to each frame image in the historical video data according to the coordinates of each bounding box of the target object in the historical video data determined by the target detection algorithm. Then, these center coordinates are arranged in sequence according to the time sequence of each frame image in the historical video data to construct the initial coordinate sequence of the target object.

[0036] S106: Determine the disturbed initial coordinate sequence according to the preset disturbance parameters and the initial coordinate sequence.

[0037] S108: For each round of training, the initial coordinate sequence after the disturbance is input into the prediction model to be trained, and the predicted coordinate sequence of the target object in the set time period after the preset historical time period is obtained by the prediction model output, and a first loss value is determined according to a first difference between the predicted coordinate sequence and the actual coordinate sequence of the target object in the set time period, and the first difference is positively correlated with the first loss value, and a second loss value is determined according to the initial coordinate sequence and the initial coordinate sequence after the disturbance.

[0038] In one or more embodiments of the present specification, in order to enable the prediction model to learn a more robust representation of coordinates and predict a more accurate motion trajectory during the training process, the server introduces a small disturbance to the initial coordinate sequence, that is, according to a preset disturbance parameter, the same disturbance parameter is added to each coordinate in the initial coordinate sequence to determine the initial coordinate sequence after the disturbance.

[0039] In one or more embodiments of the present specification, after the server determines the initial coordinate sequence after the disturbance for each round of training of the prediction model, it can input it into the prediction model to be trained to obtain the predicted coordinate sequence of the target object output by the prediction model for a set period after the preset historical period. In the present specification, the prediction model can be a long short-term memory network (Long Short-Term Memory, LSTM) model. Of course, the type of prediction model is not limited in this specification, and it can be set according to the actual situation.

[0040] In one or more embodiments of the present specification, after the server obtains the predicted coordinate sequence of the target object in a set time period after a preset historical time period output by the prediction model, it can determine the first loss value based on the first difference between the predicted coordinate sequence and the actual coordinate sequence of the target object in the set time period, and the first difference is positively correlated with the first loss value.

[0041] At the same time, since the initial coordinate sequence input to the prediction model is a coordinate sequence with disturbance parameters added, its purpose is to optimize the anti-disturbance ability of the prediction model. Therefore, it is also necessary to determine the second loss value based on the initial coordinate sequence and the initial coordinate sequence after disturbance to improve the anti-disturbance ability of the prediction model.

[0042] The server may determine a first distance value based on the predicted coordinate sequence and the initial coordinate sequence. A second distance value may be determined based on the predicted coordinate sequence and the initial coordinate sequence after disturbance. A second loss value may be determined based on a second difference between the first distance value and the second distance value, and the second loss value is positively correlated with the second difference. The specific calculation formula is as follows:

[0043] in, is the second loss value. is the first distance value, is the second distance value, p represents the distance value, y is the coordinate in the predicted coordinate sequence, x is the coordinate in the initial coordinate sequence, r represents the disturbance parameter, x+r represents the coordinate after disturbance, θ is the preset hyperparameter, It represents the derivative of the second loss value, εd represents a directional unit vector, ε represents the limit value of the perturbation parameter, and g represents the result of taking the value at r=εd after the derivative of the second loss value.

[0044] S110: Determine the comprehensive loss value corresponding to this round of training according to the first loss value and the second loss value, train the prediction model to be trained according to the comprehensive loss value corresponding to this round of training, and adjust the perturbation parameters used in the next round of training according to the second loss value until the preset training end conditions are met.

[0045] In one or more embodiments of the present specification, the server determines the comprehensive loss value corresponding to the round (i.e., the current training round) according to the determined first loss value and the second loss value, so as to train the prediction model to be trained according to the comprehensive loss value. In addition, the server may also adjust the perturbation parameters used in the next round of training according to the second loss value, and then perturb the initial coordinate sequence determined by the historical video data of the next historical period obtained by the server until the preset training end condition of the prediction model is reached.

[0046] It is worth noting that the preset training end condition of the prediction model can be a preset training round, and the training of the prediction model is stopped when the preset training round is reached. The preset training end condition of the prediction model can also be that the training time of the prediction model reaches a preset time, then the training is stopped. Of course, in this specification, there is no limitation on the preset training end condition of the prediction model, and it can be set according to actual conditions.

[0047] In one or more embodiments of the present specification, the method for the server to adjust the perturbation parameters used in the next round of training according to the second loss value may be that the server determines the perturbation parameters to be used in the next round of training according to a preset perturbation parameter limit value and the second loss value. The calculation formula is as follows:

[0048] in, The result is a unit vector with direction, and ε represents the limit value of the perturbation parameter.

[0049] Since g represents the result of the value taken at r=εd after the derivative of the second loss value, the direction of the derivative of the second loss value is the same as the direction of the perturbation parameter to be used in the next round (the perturbation parameter is a vector in vector form, so it has a direction). This allows the initial coordinate sequence determined by the historical video data of the next round of historical time period to be disturbed by the enhanced disturbance parameter, further making the trained prediction model more capable of accurate prediction under changing disturbance conditions.

[0050] In summary, when training the prediction model, the server determines the first loss value based on the predicted coordinate sequence and the actual coordinate sequence, so that the trained prediction model can predict the coordinates more accurately. On the other hand, the server introduces the second loss value based on the initial coordinate sequence and the initial coordinate sequence after disturbance, so that the trained prediction model can predict more accurately and have anti-interference ability. The prediction model is trained by combining the first loss value and the second loss value, so that the trained prediction model can eliminate possible interference in the prediction process according to the initial coordinate sequence of the target object and output an accurate prediction coordinate sequence in actual application.

[0051] In this specification, the prediction model is trained by the above-mentioned model training method to obtain a trained prediction model. After that, the trained prediction model can be deployed to perform trajectory prediction tasks. For example, in an autonomous driving scenario, the trained prediction model can be deployed to an autonomous driving car, and the autonomous driving car can input the collected video data into the trained prediction model to perform the trajectory prediction task of the target object. A trajectory prediction method provided in this specification is described in detail below.

[0052] Figure 2A flow chart of a trajectory prediction method provided in this specification specifically includes the following steps: S200: Acquire time-series video data of a preset time period, and determine a target object for trajectory prediction in the time-series video data.

[0053] In one or more embodiments of the present specification, the server obtains time-series video data of a preset time period requiring trajectory prediction, and determines a target object requiring trajectory prediction in the time-series video data.

[0054] S202: For each frame image in the time-series video data, determine the coordinates of the target object in the frame image as the corresponding coordinates of the target object in the frame image.

[0055] In one or more embodiments of the present specification, the server determines the coordinates of the target object in each frame of the time-series video data as the corresponding coordinates of the target object in the frame. For the method of determining the coordinates of the target object, reference may be made to the content of the above-mentioned model training method, which will not be repeated here.

[0056] S204: Determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the time-series video data.

[0057] S206: Inputting the initial coordinate sequence into a trained prediction model to obtain a predicted coordinate sequence of the target object in a set time period after the preset time period output by the prediction model, wherein the prediction model is trained by the above-mentioned model training method.

[0058] In one or more embodiments of the present specification, the server determines an initial coordinate sequence of the target object based on the coordinates corresponding to each frame image of the target object in the time-series video data, and inputs the initial coordinate sequence into a prediction model trained by the above-mentioned model training method to obtain a predicted coordinate sequence of the motion trajectory of the target object in a set time period after a preset time period output by the prediction model.

[0059] S208: Determine the movement trajectory of the target object in the set time period according to the predicted coordinate sequence.

[0060] In one or more embodiments of the present specification, after determining the predicted coordinate sequence of the next motion trajectory of the target object, the server can determine the motion trajectory of the target object in a set time period.

[0061] In one or more embodiments of the present specification, after determining the initial coordinate sequence of the target object, the server may also predict the coordinates corresponding to the motion trajectory of the target object in a set time period based on non-model prediction methods such as linear interpolation, polynomial fitting, and probability distribution prediction to determine a predicted coordinate sequence.

[0062] In one or more embodiments of the present specification, in order to make the coordinate prediction corresponding to the motion trajectory of the target object more accurate, the weight ratio between the prediction model and the non-model prediction method is set according to historical experience. Then, the server can obtain the predicted coordinate sequence output by the prediction model and the predicted coordinate sequence determined by the non-model prediction method according to the initial coordinate sequence through the prediction model and the non-model prediction method. Then, according to the weight ratio, the predicted coordinate sequence output by the prediction model and the predicted coordinate sequence determined by the non-model prediction method are weighted to obtain the final predicted coordinate sequence, and the motion trajectory of the target object is determined according to the final predicted coordinate sequence.

[0063] The above is a model training method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding model training device, such as Figure 3 shown.

[0064] Figure 3 A schematic diagram of a model training device provided in this specification, specifically including: The training acquisition module 300 is used to acquire historical video data of a preset historical period and determine the target object for trajectory prediction in the historical video data; A first determination module 302 is used to determine, for each frame of image in the historical video data, the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; A second determination module 304 is used to determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data; A disturbance module 306, configured to determine a disturbed initial coordinate sequence according to preset disturbance parameters and the initial coordinate sequence; A third determination module 308 is used for inputting the disturbed initial coordinate sequence into the prediction model to be trained for each round of training, obtaining a predicted coordinate sequence of the target object in a set period after the preset historical period output by the prediction model, and determining a first loss value according to a first difference between the predicted coordinate sequence and an actual coordinate sequence of the target object in the set period, wherein the first difference is positively correlated with the first loss value, and determining a second loss value according to the initial coordinate sequence and the disturbed initial coordinate sequence; The training module 310 is used to determine the comprehensive loss value corresponding to the round of training according to the first loss value and the second loss value, to train the prediction model to be trained according to the comprehensive loss value corresponding to the round of training, and to adjust the perturbation parameters used in the next round of training according to the second loss value until the preset training end condition is reached.

[0065] Optionally, the third determination module 308 is used to determine a first distance value based on the predicted coordinate sequence and the initial coordinate sequence, determine a second distance value based on the predicted coordinate sequence and the disturbed initial coordinate sequence, and determine a second loss value based on a second difference between the first distance value and the second distance value, wherein the second loss value is positively correlated with the second difference.

[0066] Optionally, the training module 310 is used to determine the disturbance parameter to be used in the next round of training based on a preset disturbance parameter limit value and the second loss value, wherein the direction of the derivative of the second loss value is the same as the direction of the disturbance parameter to be used in the next round.

[0067] Optionally, the first determination module 302 is used to determine the bounding box coordinates of the target object in the frame image according to a preset target detection algorithm as the coordinates of the target object in the frame image.

[0068] Optionally, the first determination module 302 is used to determine the center coordinates of the target object corresponding to each frame image in the historical video data according to the bounding box coordinates of the target object corresponding to each frame image in the historical video data, and construct an initial coordinate sequence of the target object according to the center coordinates of the target object in the time sequence of each frame image in the historical video data.

[0069] This specification also provides a trajectory prediction device, such as Figure 4 shown.

[0070] Figure 4 A schematic diagram of a trajectory prediction device provided in this specification specifically includes: An acquisition module 400 is used to acquire time-series video data of a preset period of time, and determine a target object for trajectory prediction in the time-series video data; A coordinate determination module 402, configured to determine, for each frame of image in the time-series video data, the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; A sequence determination module 404, configured to determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the time-series video data; A prediction module 406 is used to input the initial coordinate sequence into a trained prediction model to obtain a predicted coordinate sequence of the target object in a set time period after the preset time period output by the prediction model, wherein the prediction model is trained by the above-mentioned model training method; The trajectory determination module 408 is used to determine the motion trajectory of the target object in the set time period according to the predicted coordinate sequence.

[0071] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A model training and trajectory prediction method is provided.

[0072] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training and trajectory prediction method described.

[0073] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0074] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0075] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0076] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0077] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0078] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0083] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0084] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0086] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0088] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0089] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A model training method, characterized in that: include: Acquire historical video data of a preset historical period, and determine a target object for trajectory prediction in the historical video data; For each frame of the historical video data, determine the coordinates of the target object in the frame of the image as the corresponding coordinates of the target object in the frame of the image; Determining an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data; Determining the initial coordinate sequence after disturbance according to the preset disturbance parameters and the initial coordinate sequence; For each round of training, the disturbed initial coordinate sequence is input into the prediction model to be trained, and a predicted coordinate sequence of the target object in a set period after the preset historical period is obtained, which is output by the prediction model, and a first loss value is determined according to a first difference between the predicted coordinate sequence and an actual coordinate sequence of the target object in the set period, wherein the first difference is positively correlated with the first loss value, and a second loss value is determined according to the initial coordinate sequence and the disturbed initial coordinate sequence; According to the first loss value and the second loss value, the comprehensive loss value corresponding to this round of training is determined, so as to train the prediction model to be trained according to the comprehensive loss value corresponding to this round of training, and adjust the perturbation parameters used in the next round of training according to the second loss value until the preset training end conditions are met.

2. The method according to claim 1, characterized in that Determining a second loss value according to the initial coordinate sequence and the disturbed initial coordinate sequence specifically includes: Determining a first distance value according to the predicted coordinate sequence and the initial coordinate sequence; Determining a second distance value according to the predicted coordinate sequence and the initial coordinate sequence after disturbance; A second loss value is determined according to a second difference between the first distance value and the second distance value, and the second loss value is positively correlated with the second difference.

3. The method according to claim 1, characterized in that The perturbation parameters used in the next round of training are adjusted according to the second loss value, specifically including: According to a preset disturbance parameter limit value and the second loss value, the disturbance parameter to be used in the next round of training is determined, wherein the direction of the derivative of the second loss value is the same as the direction of the disturbance parameter to be used in the next round.

4. The method according to claim 1, characterized in that Determining the coordinates of the target object in the frame image specifically includes: According to a preset target detection algorithm, the bounding box coordinates of the target object in the frame image are determined as the coordinates of the target object in the frame image.

5. The method according to claim 4, characterized in that Determining an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data specifically includes: Determining the center coordinates of the target object in each frame of the historical video data according to the bounding box coordinates of the target object in each frame of the historical video data; According to the time sequence of each frame image in the historical video data and based on each center coordinate of the target object, an initial coordinate sequence of the target object is constructed.

6. A trajectory prediction method, characterized in that: include: Acquire time-series video data of a preset time period, and determine a target object for trajectory prediction in the time-series video data; For each frame of image in the time-series video data, determine the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; Determining an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the time-series video data; Inputting the initial coordinate sequence into a trained prediction model to obtain a predicted coordinate sequence of the target object in a set time period after the preset time period output by the prediction model, wherein the prediction model is trained by the method according to any one of claims 1 to 5; The motion trajectory of the target object in the set time period is determined according to the predicted coordinate sequence.

7. A model training device, characterized in that: include: A training acquisition module, used to acquire historical video data of a preset historical period, and determine a target object for trajectory prediction in the historical video data; A first determination module is used to determine, for each frame of image in the historical video data, the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; A second determination module is used to determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the historical video data; A disturbance module, used for determining a disturbed initial coordinate sequence according to preset disturbance parameters and the initial coordinate sequence; a third determination module, for inputting the disturbed initial coordinate sequence into the prediction model to be trained for each round of training, obtaining a predicted coordinate sequence of the target object in a set period after the preset historical period output by the prediction model, and determining a first loss value according to a first difference between the predicted coordinate sequence and an actual coordinate sequence of the target object in the set period, wherein the first difference is positively correlated with the first loss value, and determining a second loss value according to the initial coordinate sequence and the disturbed initial coordinate sequence; A training module is used to determine the comprehensive loss value corresponding to the round of training according to the first loss value and the second loss value, to train the prediction model to be trained according to the comprehensive loss value corresponding to the round of training, and to adjust the perturbation parameters used in the next round of training according to the second loss value until the preset training end condition is reached.

8. A trajectory prediction device, characterized in that: include: An acquisition module, used to acquire time-series video data of a preset period of time, and determine a target object for trajectory prediction in the time-series video data; A coordinate determination module, used for determining, for each frame of image in the time-series video data, the coordinates of the target object in the frame of image as the corresponding coordinates of the target object in the frame of image; A sequence determination module, used to determine an initial coordinate sequence of the target object according to the coordinates of the target object corresponding to each frame image in the time-series video data; A prediction module, used for inputting the initial coordinate sequence into a trained prediction model to obtain a predicted coordinate sequence of the target object in a set time period after the preset time period output by the prediction model, wherein the prediction model is trained by the method according to any one of claims 1 to 5; The trajectory determination module is used to determine the motion trajectory of the target object in the set time period according to the predicted coordinate sequence.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Position prediction model training method, position prediction method and track prediction method for moving objects in video images

    CN107492113A

  • Trajectory prediction model generation method and device, readable storage medium and electronic equipment

    CN111626219A

  • Automatic driving control method and system and storage medium

    CN115709727A

  • Training method of trajectory prediction model, trajectory prediction method, device and medium

    CN116342652A

  • Training method and device of trajectory planning model, terminal equipment and storage medium

    CN117808113A