Motion trajectory prediction method and device, computer device and storage medium

Abstract

The application relates to a motion trajectory prediction method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining environment data samples obtained by collecting a target environment, and down-sampling environment data samples obtained by down-sampling the environment data samples; performing motion sensing processing on the environment data samples and the down-sampling environment data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment; training a pre-trained machine learning model based on the first motion data samples and the second motion data samples to obtain a prediction model; and predicting motion data obtained by down-sampling the environment data by using the prediction model to obtain the motion trajectory of the object. The method can improve the accuracy of the predicted motion trajectory.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a motion trajectory prediction method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] With the development of computer technology, in the process of automatic driving, it is necessary to calculate the motion trajectory of the target object in a future period of time through the machine learning model of the online platform in real time. In order to ensure the real-time of calculation, it is usually necessary to reduce the calculation amount of the machine learning model, so the machine learning model of the online platform carries out trajectory prediction according to the motion data generated by the down-sampled environment data. When training the machine learning model, the motion data as the training sample should have the same data distribution as the motion data used by the machine learning model in actual prediction. Therefore, in the traditional method, the machine learning model is trained using the motion data obtained according to the down-sampled environment data. Since the down-sampled environment data has large noise jitter, the prediction accuracy of the prediction model obtained by training is low, which leads to low accuracy of the motion trajectory predicted by the prediction model. SUMMARY

[0003] Therefore, it is necessary to provide a motion trajectory prediction method, device, computer equipment, computer readable storage medium and computer program product capable of improving accuracy in view of the above technical problems.

[0004] In a first aspect, the present application provides a motion trajectory prediction method. The method comprises:

[0005] obtaining an environment data sample obtained by collecting a target environment, and a down-sampled environment data sample obtained by down-sampling the environment data sample;

[0006] respectively performing motion perception processing on the environment data sample and the down-sampled environment data sample to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment;

[0007] training a pre-trained machine learning model based on the first motion data samples and the second motion data samples to obtain a prediction model;

[0008] predicting the motion data obtained by the down-sampled environment data through the prediction model to obtain the motion trajectory of the object.

[0009] In a second aspect, the present application further provides a motion trajectory prediction device. The device comprises:

[0010] an acquisition module, configured to acquire an environment data sample obtained by collecting a target environment, and a down-sampled environment data sample obtained by down-sampling the environment data sample;

[0011] a processing module, configured to perform motion-aware processing on the environment data sample and the down-sampled environment data sample respectively, to obtain a first motion data sample and a second motion data sample corresponding to each target object in the target environment;

[0012] a training module, configured to train a pre-trained machine learning model based on the first motion data sample and the second motion data sample, to obtain a prediction model;

[0013] a prediction module, configured to predict motion data obtained by down-sampling the environment data by using the prediction model, to obtain a motion trajectory of an object.

[0014] In an embodiment, the training module is further configured to:

[0015] extract a first target data sample of a target object in the target environment in a first time period from the second motion data sample;

[0016] extract a second target data sample of the target object in a second time period from the first motion data sample; the second time period is a next time period adjacent to the first time period;

[0017] train a pre-trained machine learning model based on a sample set composed of the first target data sample and the second target data sample, to obtain a prediction model.

[0018] In an embodiment, the training module is further configured to:

[0019] extract a first tracking identifier corresponding to the target object from the first motion data sample;

[0020] find a second tracking identifier matching the first tracking identifier from the second motion data;

[0021] extract a second target data sample of the target object in the second time period from the second motion data sample according to the second tracking identifier.

[0022] In an embodiment, the apparatus further comprises:

[0023] A determination module is used to determine the motion duration of each object in the target environment based on the first motion data; among the objects in the target environment, if the corresponding motion duration of the object is greater than a preset duration, the object is regarded as a target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

[0024] In one embodiment, the training module is further configured to:

[0025] Performing prediction based on the second motion data sample using the pre-trained machine learning model to obtain predicted trajectory data;

[0026] Performing calculations based on the first motion data sample to obtain trajectory label data;

[0027] Calculating a loss value based on the predicted trajectory data and the trajectory label data;

[0028] The parameters of the pre-trained machine learning model are adjusted according to the loss value to obtain a prediction model.

[0029] In one embodiment, the apparatus further comprises:

[0030] The acquisition module is further used to acquire environmental data obtained by collecting the driving environment during the autonomous driving process;

[0031] A downsampling module, configured to downsample the environmental data to obtain downsampled environmental data;

[0032] The prediction module is further used to:

[0033] Performing motion sensing processing on the downsampled environmental data to obtain motion data;

[0034] The motion data is predicted using the prediction model to obtain the motion trajectory of each object in the driving environment.

[0035] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0036] Acquire an environmental data sample obtained by collecting a target environment, and a downsampled environmental data sample obtained by downsampling the environmental data sample;

[0037] Performing motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment;

[0038] Training a pre-trained machine learning model based on the first motion data sample and the second motion data sample to obtain a prediction model;

[0039] The motion data obtained from the downsampled environmental data is predicted by the prediction model to obtain the motion trajectory of the object.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0041] Acquire an environmental data sample obtained by collecting a target environment, and a downsampled environmental data sample obtained by downsampling the environmental data sample;

[0042] Performing motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment;

[0043] Training a pre-trained machine learning model based on the first motion data sample and the second motion data sample to obtain a prediction model;

[0044] The motion data obtained from the downsampled environmental data is predicted by the prediction model to obtain the motion trajectory of the object.

[0045] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0046] Acquire an environmental data sample obtained by collecting a target environment, and a downsampled environmental data sample obtained by downsampling the environmental data sample;

[0047] Performing motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment;

[0048] Training a pre-trained machine learning model based on the first motion data sample and the second motion data sample to obtain a prediction model;

[0049] The motion data obtained from the downsampled environmental data is predicted by the prediction model to obtain the motion trajectory of the object.

[0050] The aforementioned motion trajectory prediction method, apparatus, computer device, storage medium, and computer program product obtain environmental data samples collected from a target environment and downsampled environmental data samples obtained by downsampling the environmental data samples. Motion sensing processing is then performed on the environmental data samples and the downsampled environmental data samples to obtain first and second motion data samples corresponding to each target object in the target environment. A pre-trained machine learning model is trained based on the first and second motion data samples to obtain a prediction model. Because the first motion data samples, obtained from un-downsampled environmental data samples, have low noise and jitter and can accurately describe the motion of objects in the target environment, using the first motion data samples during model training can improve the prediction accuracy of the trained prediction model. Furthermore, the motion trajectory predicted by the prediction model is more accurate. Furthermore, because the second motion data samples are obtained from the downsampled environmental data samples, the second motion data samples and the data used by the online platform's prediction model during actual prediction have the same data distribution. Therefore, the trained prediction model can be applied to the online platform to predict lightweight motion data obtained from the downsampled environmental data, thereby obtaining accurate motion trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A diagram showing an application environment of a motion trajectory prediction method in one embodiment;

[0052] Figure 2 1 is a flow chart of a motion trajectory prediction method according to an embodiment;

[0053] Figure 3 is a schematic diagram of a motion trajectory in one embodiment;

[0054] Figure 4 Schematic diagram of a flow chart of a prediction model training method in one embodiment;

[0055] Figure 5 1 is a flow chart of a method for extracting a second target data sample in one embodiment;

[0056] Figure 6 A schematic diagram of marking a tracking mark in one embodiment;

[0057] Figure 7 Schematic diagram of a flow chart of a motion trajectory prediction method according to another embodiment;

[0058] Figure 8 is a structural block diagram of a motion trajectory prediction device in one embodiment;

[0059] Figure 9is a structural block diagram of a motion trajectory prediction device in another embodiment;

[0060] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] The motion trajectory prediction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the environmental data samples obtained by collecting the target environment and the downsampled environmental data samples obtained by downsampling the environmental data samples from the terminal 102; performs motion perception processing on the environmental data samples and the downsampled environmental data samples respectively to obtain the first motion data samples and the second motion data samples corresponding to each target object in the target environment; based on the first motion data samples and the second motion data samples, the pre-trained machine learning model is trained to obtain a prediction model; the motion data obtained from the downsampled environmental data is predicted by the prediction model to obtain the motion trajectory of the object. Among them, the terminal 102 can be, but is not limited to, a smart car-mounted device or a portable wearable device. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0063] In one embodiment, Figure 2 As shown, a motion trajectory prediction method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0064] S202: Acquire an environmental data sample obtained by collecting the target environment and a downsampled environmental data sample obtained by downsampling the environmental data sample.

[0065] The target environment may be an environment including various moving objects. For example, the target environment may be a driving environment of a vehicle. The environmental data samples are data samples collected from the target environment, and may be point cloud data samples obtained by scanning the target environment through a laser radar, or image data samples, video data samples, etc. collected from the target environment through a camera. The image data samples may be color image data samples, or black and white image data samples; the video data samples may be color video data samples, or black and white video data samples. Downsampling the environmental data samples may be spatial downsampling of the environmental data samples, or temporal downsampling of the environmental data samples. For example, the server may spatially downsample M×N image data samples by a factor of s to obtain (M / s)×(N / s) image data samples. For example, the server may temporally downsample video data samples containing N video frames, and extract n (1≤n <N)帧作为下采样环境数据样本。

[0066] S204 , performing motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment.

[0067] Motion sensing is the process of capturing the motion of moving objects, and includes both target detection and motion tracking. Target detection is the process of detecting various objects in the target environment, while motion tracking is the process of tracking the detected objects.

[0068] The motion data samples are data samples that describe the motion of an object, including the object's position coordinates (x, y, z), motion velocity (v_x, v_y, v_z), acceleration (a_x, a_y, a_z), heading angle, and dimensions (length, width, height) in the world coordinate system. The first motion data samples are motion data samples obtained by performing motion sensing processing on the environmental data samples, and the second motion data samples are motion data samples obtained by performing motion sensing processing on the downsampled environmental data samples. Since the downsampled environmental data samples lose some data through downsampling and have lower data accuracy, the accuracy of the second motion data samples is lower than that of the first motion data samples.

[0069] In one embodiment, the server performs motion perception processing on the environmental data samples through an offline perception model to obtain first motion data samples corresponding to each target object in the target environment; and performs motion perception processing on the downsampled environmental data samples through an online perception model to obtain second motion data samples corresponding to each target object in the target environment.

[0070] Among them, the offline perception model is a perception model running on an offline platform, including a detection model and a tracking model. The detection model is used to detect objects in the target environment. The tracking model is used to track the motion of the detected objects. Because the offline perception model runs on an offline platform, the real-time requirements for data processing are lower. It can have a complex model structure and use large amounts of environmental data samples for motion perception processing. For example, the detection model in the offline perception model can be a 3D auto labeling model, an Auto4D model, etc.

[0071] Among them, the online perception model is a perception model running on the online platform, including a detection model and a tracking model. Since the online perception model runs on the online platform, it has high requirements for the real-time performance of data processing. Compared with the offline perception model, the model structure is simpler and it is a lightweight model that uses downsampled environmental data samples with a smaller data volume for motion perception processing. For example, the detection model in the offline perception model can be a PointPillar (point cloud detection) model or a PointVoxel-RCNN (Point Voxel-Region Convolutional Neural Network, three-dimensional voxel region convolutional neural network) model.

[0072] S206: Based on the first motion data sample and the second motion data sample, the pre-trained machine learning model is trained to obtain a prediction model.

[0073] The machine learning model is an artificial intelligence model constructed based on machine learning technology, and can be a neural network model, a convolutional neural network model, a deep learning model, etc. For example, the machine learning model can be a Scene Transformer model, an mm Transformer model, a VectorNet model, or a MultiPath++ model.

[0074] In one embodiment, the server concatenates the first motion data sample and the second motion data sample to obtain a training data sample, and uses the training data sample to train a pre-trained machine learning model. The second motion data is then tested using the trained prediction model to obtain a test result. When the test result meets a training stop condition, the machine learning model training is determined to be complete, and the trained machine learning model is used as the prediction model for predicting the object's motion trajectory. The training stop condition is a condition for determining whether the machine learning model has been trained. For example, the training stop condition may be that the accuracy of the test result exceeds a preset value.

[0075] S208, predicting the motion data obtained by the down-sampled environment data by the prediction model to obtain the motion trajectory of the object.

[0076] The motion trajectory is a route that the object moves from a start position to an end position, and can be represented by position coordinates of each point passed by the object during the movement. For example, as shown in Figure 3 , the motor vehicle moves from point A to point B, and the motion trajectory is a curve from point A to point B.

[0077] In an embodiment, S208 further includes, during the automatic driving, obtaining environment data collected by collecting a driving environment; down-sampling the environment data to obtain down-sampled environment data; S208 specifically includes: performing motion perception processing according to the down-sampled environment data to obtain motion data; and predicting the motion data by the prediction model to obtain the motion trajectory of each object in the driving environment.

[0078] The environment data is data obtained by collecting the driving environment, which can be point cloud data obtained by scanning the driving environment by a laser radar, or image data, video data, etc. collected by a camera. The image data can be color image data, or can be black and white image data; the video data can be color video data, or can be black and white video data. The down-sampling of the environment data can be spatial down-sampling of the environment data, or can be temporal down-sampling of the environment data.

[0079] After the server trains the prediction model, the prediction model is deployed to an online platform for use. The server performs motion perception processing on the down-sampled environment data by the online perception model to obtain motion data, and then predicts the motion data by the prediction model to obtain the motion trajectory of each object in the driving environment.

[0080] In the above embodiments, the environment data sample obtained by collecting the target environment and the down-sampled environment data sample obtained by down-sampling the environment data sample are obtained. Then, the motion perception processing is performed on the environment data sample and the down-sampled environment data sample respectively, to obtain the first motion data sample and the second motion data sample corresponding to each target object in the target environment. Based on the first motion data sample and the second motion data sample, the pre-trained machine learning model is trained to obtain the prediction model. Since the first motion data sample obtained based on the environment data sample without down-sampling processing has less noise jitter, it can accurately describe the motion of the object in the target environment. Therefore, by using the first motion data sample in model training, the prediction accuracy of the prediction model obtained by training can be improved, and further, the motion trajectory obtained by prediction through the prediction model is more accurate. Moreover, since the second motion data sample is obtained by down-sampling the environment data sample, the second motion data sample and the data used by the prediction model of the online platform in actual prediction have the same data distribution. Therefore, the prediction model obtained by training can be applied to the online platform to predict the lightweight motion data obtained by down-sampling the environment data, and obtain accurate object motion trajectory.

[0081] In one embodiment, as shown in FIG. 6, S206 specifically includes the following steps: Figure 4

[0082] S402, in the second motion data sample, the first target data sample of the target object in the target environment in the first time period is extracted.

[0083] The target object can be any object in the target environment, including motor vehicles, pedestrians, animals, etc. The first time period is any time period in which the target object appears in the target environment, for example, the first time period can be the time period from the first second to the third second in which the target object appears in the target environment. The first target data sample is the motion data of the target object in the first time period. For example, the first target data sample includes the position coordinates, speed, acceleration, orientation angle or size of the target object in the first to third second.

[0084] In one embodiment, the second motion data sample is [T1_onboard, T2_onboard, T3_onboard], where T1_onboard, T2_onboard, T3_onboard are the second motion data samples of the target object in the time periods [0, T1], (T1, T2], (T2, T3] respectively, and the first target data sample extracted by the server can be T1_onboard.

[0085] ​S404 , extracting a second target data sample of the target object in a second time period from the first motion data sample; the second time period is a next time period adjacent to the first time period.

[0086] The second time period is the next time period adjacent to the first time period. For example, the first time period is a time period from 0 to 3 seconds, and the second time period is a time period from 3 to 8 seconds. The second target data sample is a motion data sample of the target object during the second time period. For example, the second target data sample includes the position coordinates, velocity, acceleration, heading angle, or size of the target object during the period from 3 to 8 seconds.

[0087] In one embodiment, the first motion data sample is [T1_offboard, T2_offboard, T3_offboard], and the second motion data sample is [T1_onboard, T2_onboard, T3_onboard]. T1_offboard, T2_offboard, and T3_offboard are the first motion data samples of the target object in the time periods of [0, T1], (T1, T2], and (T2, T3], respectively; T1_onboard, T2_onboard, and T3_onboard are the second motion data samples of the target object in the time periods of [0, T1], (T1, T2], and (T2, T3], respectively. When the first target data sample extracted by the server is T1_onboard, the second target data sample extracted by the server is T2_offboard.

[0088] S406: Train the pre-trained machine learning model according to the sample set consisting of the first target data sample and the second target data sample to obtain a prediction model.

[0089] The server trains the pre-trained machine learning model based on a sample set consisting of the first target data sample and the second target data sample. For example, the server trains the pre-trained machine learning model based on [T1_onboard, T2_offboard] to obtain a prediction model.

[0090] In one embodiment, S406 specifically includes: using a pre-trained machine learning model, making predictions based on the first target data sample to obtain predicted trajectory data; performing calculations based on the second target data sample to obtain trajectory label data; calculating a loss value based on the predicted trajectory data and the trajectory label data; and adjusting the parameters of the pre-trained machine learning model according to the loss value to obtain a prediction model.

[0091] The trajectory label data is actual motion trajectory data calculated according to the second target data sample. The loss value is used to represent the difference between the predicted trajectory data and the trajectory label data. In an embodiment, the server can calculate the loss value by using a cross-entropy loss function, an exponential loss function, a logarithmic loss function, an absolute value loss function, and the like.

[0092] In the above embodiment, the server extracts the first target data sample of the target object in the target environment within the first time period from the second motion data sample. Then, the server extracts the second target data sample of the target object within the second time period from the first motion data sample. The pre-trained machine learning model is trained according to the sample set composed of the first target data sample and the second target data sample, to obtain the prediction model. Since the data used by the second motion data sample and the prediction model of the online platform when actually predicting has the same data distribution, the prediction model trained by the first target data sample extracted from the second motion data sample can be applied to the online platform. Since the first motion data sample has less noise jitter and can accurately describe the motion of the object in the target environment, using the first motion data sample as test data during model training can improve the prediction accuracy of the trained prediction model.

[0093] In an embodiment, S402 further includes: determining the motion duration of each object in the target environment based on the first motion data; among the objects in the target environment, if the motion duration of an object is greater than a preset duration, regarding the object as a target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

[0094] The motion duration is the time length during which the object appears in the target environment, starting from the time when the object enters the target environment to the time when the object leaves the target environment. The preset duration is a pre-set time length, which is greater than the sum of the length of the first time period and the length of the second time period. For example, the length of the first time period is 3 seconds, and the length of the second time period is 5 seconds, and the preset duration can be set to 10 seconds.

[0095] The server selects each object in the target environment, removes objects with a motion duration less than a preset duration, and regards objects with a motion duration greater than the preset duration as target objects. Since the preset duration is greater than the sum of the length of the first time period and the length of the second time period, the target object has complete first target data samples and second target data samples within the first time period and the second time period, which ensures the integrity of the data samples used for training.

[0096] In the above embodiment, the server determines the motion duration of each object in the target environment and identifies objects whose motion duration exceeds a preset duration as target objects. It then extracts first and second target data samples corresponding to the target objects. The server then trains a pre-trained machine learning model based on the sample set consisting of the first and second target data samples to generate a prediction model. This ensures the integrity of the training data samples and improves the accuracy of the trained prediction model.

[0097] In one embodiment, Figure 5 As shown, S404 specifically includes the following steps:

[0098] S502: Extract a first tracking identifier corresponding to a target object from a first motion data sample.

[0099] The first tracking identifier (Tracking Id) is the identifier of the target object in the first motion data sample. Figure 6 As shown, the tracking mark of the motor vehicle is 1, the tracking mark of the tree is 2, and the tracking mark of the child is 3.

[0100] S504: Search the second motion data for a second tracking identifier that matches the first tracking identifier.

[0101] Since the tracking identifiers of the same target object may be different in the first motion data sample and the second motion data, the server searches for the second tracking identifier that matches the first tracking identifier in the second motion data to find the target object in the second motion data, so that the extracted first target data sample and the second target data sample are data samples of the same target object.

[0102] In one embodiment, the server may search for a second tracking identifier that matches the first tracking identifier by using a Hungarian Algorithm or a KM (Kuhn-Munkres Algorithm) algorithm.

[0103] S506 : Extracting, from the second motion data samples, second target data samples of the target object within a second time period according to the second tracking identifier.

[0104] The server extracts, from the second motion data sample, a second target data sample of the target object corresponding to the second tracking identifier within the second time period. For example, if the first tracking identifier of target object A in the first motion data sample is 1, the server finds that the second tracking identifier corresponding to target object A in the second motion data sample is 2, and extracts, from the second motion data sample, the second target data sample of the target object with tracking identifier 2 within the second time period.

[0105] In the above embodiment, the server extracts a first tracking identifier corresponding to the target object from the first motion data sample, searches for a second tracking identifier that matches the first tracking identifier in the second motion data, and extracts a second target data sample of the target object within the second time period from the second motion data sample based on the second tracking identifier. This allows the tracking identifier of the target object in the first motion data sample to be associated with the tracking identifier in the second motion data sample, thereby matching the first motion data sample with the second motion data sample to the same target object. The first and second target data samples corresponding to the same target object can then be extracted and used to train a machine learning model, thereby improving the prediction accuracy of the machine learning model and, further, the accuracy of the predicted motion trajectory.

[0106] In one embodiment, S206 specifically includes: using a pre-trained machine learning model, making predictions based on the second motion data sample to obtain predicted trajectory data; performing calculations based on the first motion data sample to obtain trajectory label data; calculating a loss value based on the predicted trajectory data and the trajectory label data; and adjusting the parameters of the pre-trained machine learning model according to the loss value to obtain a prediction model.

[0107] The predicted trajectory data is data used to record the motion trajectory predicted by the machine learning model. The trajectory label data is data used to record the actual motion trajectory of the object. The server calculates a loss value based on the predicted trajectory data and the trajectory label data, and adjusts the parameters of the pre-trained machine learning model based on the loss value to minimize the loss value between the predicted trajectory data and the trajectory label data, thereby optimizing the pre-trained machine learning model. In one embodiment, the server can calculate the loss value using a cross-entropy loss function, an exponential loss function, a logarithmic loss function, an absolute value loss function, or the like.

[0108] In the above embodiment, the server uses a pre-trained machine learning model to perform predictions based on the second motion data sample to obtain predicted trajectory data. Calculations are performed based on the first motion data sample to obtain trajectory label data. A loss value is calculated based on the predicted trajectory data and the trajectory label data. The parameters of the pre-trained machine learning model are adjusted based on the loss value to obtain a prediction model. This allows for iterative optimization of the pre-trained machine learning model based on the loss value, thereby improving the accuracy of the trained prediction model.

[0109] In one embodiment, during the driving process, the autonomous vehicle scans the target environment through a laser radar to obtain point cloud data, or shoots the target environment through a camera to obtain image data. The server uses the point cloud data or image data as an environmental data sample, and downsamples the environmental data sample to obtain a downsampled environmental data sample. The server performs motion perception processing on the environmental data sample through an offline perception model to obtain a first motion data sample; and performs motion perception processing on the downsampled environmental data sample through an online perception model to obtain a second motion data sample. The server determines the motion duration of each object in the first motion data sample and the second motion data sample, deletes the motion data samples of objects whose motion duration is less than a preset duration (for example, 10 seconds), and only retains the first motion data sample and the second motion data sample of the target object whose motion duration is greater than or equal to the preset duration. Assume that the first motion data sample remaining after deletion is [T1_offboard, T2_offboard, T3_offboard], where T1_offboard, T2_offboard and T3_offboard are the first motion data samples of the target object in the time periods [0, T1], (T1, T2] and (T2, T3] respectively. Assume that the second motion data sample remaining after deletion is [T1_onboard, T2_onboard, T3_onboard], where T1_onboard, T2_onboard and T3_onboard are the second motion data samples of the target object in the time periods [0, T1], (T1, T2] and (T2, T3] respectively. The server extracts the data of T1_onboard from the second motion data sample, extracts the data of T2_offboard from the first motion data sample, and [T1_onboard, T2_offboard] form a sample set. The server uses T1_onboard as training data to train the pre-trained machine learning model, and uses T2_offboard as test data to test the trained machine learning model. The parameters of the machine learning model are adjusted based on the test results. When the test result of the trained machine learning model meets the training stop condition, the machine learning model that meets the training stop condition is used as the prediction model for predicting the object's motion trajectory. Because T2_offboard is a motion data sample obtained by the offline perception model through motion perception processing of environmental data samples, it has less noise jitter than the motion data sample obtained by the online perception model and can more accurately describe the object's motion. The prediction model trained using T2_offboard as test data has stronger noise resistance and high prediction accuracy.

[0110] While driving, an autonomous vehicle collects environmental data from the driving environment and uploads it to a server. The server then downsamples the environmental data to generate downsampled environmental data. The server then uses an online perception model to perform motion sensing on the downsampled environmental data, generating motion data. The server then uses a prediction model to predict the motion data and determine the motion trajectories of objects in the driving environment.

[0111] In one embodiment, Figure 7 As shown, the motion trajectory prediction method includes the following steps:

[0112] S702: Acquire an environmental data sample obtained by collecting the target environment and a downsampled environmental data sample obtained by downsampling the environmental data sample.

[0113] S704 , performing motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment.

[0114] S706 : Extract, from the second motion data samples, a first target data sample of the target object in the target environment within the first time period.

[0115] S708 : Extract a first tracking identifier corresponding to the target object from the first motion data sample, and search for a second tracking identifier matching the first tracking identifier in the second motion data.

[0116] S710 , extracting a second target data sample of the target object in a second time period from the second motion data sample according to the second tracking identifier; the second time period is a next time period adjacent to the first time period.

[0117] S712: Train the pre-trained machine learning model according to the sample set consisting of the first target data sample and the second target data sample to obtain a prediction model.

[0118] S714 , predicting the motion data obtained from the downsampled environmental data using a prediction model to obtain a motion trajectory of the object.

[0119] S716, during the autonomous driving process, obtaining environmental data obtained by collecting the driving environment, and downsampling the environmental data to obtain downsampled environmental data.

[0120] S718: Perform motion sensing processing based on the downsampled environmental data to obtain motion data.

[0121] S720: Predict the motion data using a prediction model to obtain motion trajectories of objects in the driving environment.

[0122] For the specific contents of S702 to S720 above, please refer to the specific implementation process above.

[0123] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0124] Based on the same inventive concept, the present application also provides a motion trajectory prediction device for implementing the motion trajectory prediction method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more of the following embodiments of the motion trajectory prediction device can be found in the above-mentioned limitations of the motion trajectory prediction method and will not be repeated here.

[0125] In one embodiment, Figure 8 As shown, a motion trajectory prediction device is provided, including: an acquisition module 802, a processing module 804, a training module 806 and a prediction module 808, wherein:

[0126] An acquisition module 802 is configured to acquire an environmental data sample obtained by collecting the target environment and a downsampled environmental data sample obtained by downsampling the environmental data sample;

[0127] A processing module 804 is configured to perform motion sensing processing on the environmental data samples and the downsampled environmental data samples, respectively, to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment;

[0128] A training module 806 is configured to train a pre-trained machine learning model based on the first motion data sample and the second motion data sample to obtain a prediction model;

[0129] The prediction module 808 is configured to predict the motion data obtained from the downsampled environmental data using a prediction model to obtain a motion trajectory of the object.

[0130] In the above embodiment, environmental data samples obtained by collecting the target environment and downsampled environmental data samples obtained by downsampling the environmental data samples are obtained. Motion sensing processing is then performed on the environmental data samples and the downsampled environmental data samples to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment. Based on the first motion data samples and the second motion data samples, a pre-trained machine learning model is trained to obtain a prediction model. Because the first motion data samples obtained based on the environmental data samples that have not been downsampled have less noise and jitter and can accurately describe the motion of objects in the target environment, using the first motion data samples during model training can improve the prediction accuracy of the trained prediction model. Furthermore, the motion trajectory predicted by the prediction model is more accurate. Moreover, because the second motion data samples are obtained by downsampling the environmental data samples, the second motion data samples and the data used by the prediction model of the online platform during actual prediction have the same data distribution. Therefore, the trained prediction model can be applied to the online platform to predict the lightweight motion data obtained from the downsampled environmental data to obtain an accurate object motion trajectory.

[0131] In one embodiment, the training module 806 is further configured to:

[0132] Extracting, from the second motion data sample, a first target data sample of the target object in the target environment within the first time period;

[0133] Extracting a second target data sample of the target object in a second time period from the first motion data sample; the second time period is a next time period adjacent to the first time period;

[0134] The pre-trained machine learning model is trained according to a sample set consisting of the first target data sample and the second target data sample to obtain a prediction model.

[0135] In one embodiment, the training module 806 is further configured to:

[0136] Extracting a first tracking identifier corresponding to the target object from the first motion data sample;

[0137] searching, in the second motion data, for a second tracking identifier that matches the first tracking identifier;

[0138] According to the second tracking identifier, a second target data sample of the target object within the second time period is extracted from the second motion data sample.

[0139] In one embodiment, Figure 9 As shown, the device also includes:

[0140] Determination module 810 is used to determine the motion duration of each object in the target environment based on the first motion data; among the objects in the target environment, if the corresponding motion duration of the object is greater than the preset duration, the object is regarded as the target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

[0141] In one embodiment, the training module 806 is further configured to:

[0142] Using a pre-trained machine learning model, prediction is performed based on the second motion data sample to obtain predicted trajectory data;

[0143] Calculating based on the first motion data sample to obtain trajectory label data;

[0144] Calculate the loss value based on the predicted trajectory data and trajectory label data;

[0145] The parameters of the pre-trained machine learning model are adjusted according to the loss value to obtain a prediction model.

[0146] In one embodiment, the apparatus further comprises:

[0147] The acquisition module 802 is further used to acquire environmental data obtained by collecting driving environment during the autonomous driving process;

[0148] A downsampling module 812 is configured to downsample the environmental data to obtain downsampled environmental data;

[0149] The prediction module 808 is further configured to:

[0150] Perform motion perception processing based on the downsampled environmental data to obtain motion data;

[0151] The motion data is predicted through the prediction model to obtain the motion trajectory of each object in the driving environment.

[0152] Each module in the aforementioned motion trajectory prediction device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0153] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store motion trajectory prediction data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a motion trajectory prediction method is implemented.

[0154] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0155] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: obtaining environmental data samples obtained by collecting a target environment, and downsampled environmental data samples obtained by downsampling the environmental data samples; performing motion perception processing on the environmental data samples and the downsampled environmental data samples, respectively, to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment; training a pre-trained machine learning model based on the first motion data samples and the second motion data samples to obtain a prediction model; and predicting the motion data obtained from the downsampled environmental data by using the prediction model to obtain the motion trajectory of the object.

[0156] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in the second motion data sample, a first target data sample of the target object in the target environment within the first time period is extracted; in the first motion data sample, a second target data sample of the target object within the second time period is extracted; the second time period is the next time period adjacent to the first time period; based on the sample set consisting of the first target data sample and the second target data sample, a pre-trained machine learning model is trained to obtain a prediction model.

[0157] In one embodiment, when the processor executes the computer program, it further implements the following steps: extracting a first tracking identifier corresponding to the target object from the first motion data sample; searching for a second tracking identifier matching the first tracking identifier in the second motion data; and extracting a second target data sample of the target object within a second time period from the second motion data sample based on the second tracking identifier.

[0158] In one embodiment, when the processor executes the computer program, it also implements the following steps: based on the first motion data, determining the motion duration of each object in the target environment; among the objects in the target environment, if the corresponding motion duration of the object is greater than a preset duration, the object is regarded as a target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

[0159] In one embodiment, when the processor executes the computer program, it also implements the following steps: using a pre-trained machine learning model, prediction is performed based on the second motion data sample to obtain predicted trajectory data; calculation is performed based on the first motion data sample to obtain trajectory label data; a loss value is calculated based on the predicted trajectory data and the trajectory label data; and the parameters of the pre-trained machine learning model are adjusted according to the loss value to obtain a prediction model.

[0160] In one embodiment, when the processor executes the computer program, it also implements the following steps: during the autonomous driving process, obtaining environmental data collected from the driving environment; downsampling the environmental data to obtain downsampled environmental data; performing motion perception processing based on the downsampled environmental data to obtain motion data; and predicting the motion data through a prediction model to obtain the motion trajectory of each object in the driving environment.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining environmental data samples obtained by collecting a target environment, and downsampled environmental data samples obtained by downsampling the environmental data samples; performing motion perception processing on the environmental data samples and the downsampled environmental data samples, respectively, to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment; training a pre-trained machine learning model based on the first motion data samples and the second motion data samples to obtain a prediction model; and predicting the motion data obtained from the downsampled environmental data by using the prediction model to obtain the motion trajectory of the object.

[0162] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in the second motion data sample, a first target data sample of the target object in the target environment within the first time period is extracted; in the first motion data sample, a second target data sample of the target object within the second time period is extracted; the second time period is the next time period adjacent to the first time period; based on the sample set consisting of the first target data sample and the second target data sample, a pre-trained machine learning model is trained to obtain a prediction model.

[0163] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: extracting a first tracking identifier corresponding to the target object in the first motion data sample; searching for a second tracking identifier matching the first tracking identifier in the second motion data; and extracting a second target data sample of the target object within a second time period in the second motion data sample based on the second tracking identifier.

[0164] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the first motion data, the motion duration of each object in the target environment is determined; among the objects in the target environment, if the corresponding motion duration of the object is greater than a preset duration, the object is regarded as a target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

[0165] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: prediction is performed based on the second motion data sample through a pre-trained machine learning model to obtain predicted trajectory data; calculation is performed based on the first motion data sample to obtain trajectory label data; a loss value is calculated based on the predicted trajectory data and the trajectory label data; and the parameters of the pre-trained machine learning model are adjusted according to the loss value to obtain a prediction model.

[0166] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: during the autonomous driving process, environmental data collected from the driving environment is acquired; down-sampled the environmental data to obtain down-sampled environmental data; motion perception processing is performed based on the down-sampled environmental data to obtain motion data; and the motion data is predicted through a prediction model to obtain the motion trajectory of each object in the driving environment.

[0167] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: obtaining environmental data samples obtained by collecting a target environment, and downsampled environmental data samples obtained by downsampling the environmental data samples; performing motion perception processing on the environmental data samples and the downsampled environmental data samples, respectively, to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment; training a pre-trained machine learning model based on the first motion data samples and the second motion data samples to obtain a prediction model; and predicting the motion data obtained from the downsampled environmental data by using the prediction model to obtain a motion trajectory of the object.

[0168] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in the second motion data sample, a first target data sample of the target object in the target environment within the first time period is extracted; in the first motion data sample, a second target data sample of the target object within the second time period is extracted; the second time period is the next time period adjacent to the first time period; based on the sample set consisting of the first target data sample and the second target data sample, a pre-trained machine learning model is trained to obtain a prediction model.

[0169] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: extracting a first tracking identifier corresponding to the target object in the first motion data sample; searching for a second tracking identifier matching the first tracking identifier in the second motion data; and extracting a second target data sample of the target object within a second time period in the second motion data sample based on the second tracking identifier.

[0170] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the first motion data, the motion duration of each object in the target environment is determined; among the objects in the target environment, if the corresponding motion duration of the object is greater than a preset duration, the object is regarded as a target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

[0171] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: prediction is performed based on the second motion data sample through a pre-trained machine learning model to obtain predicted trajectory data; calculation is performed based on the first motion data sample to obtain trajectory label data; a loss value is calculated based on the predicted trajectory data and the trajectory label data; and the parameters of the pre-trained machine learning model are adjusted according to the loss value to obtain a prediction model.

[0172] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: during the autonomous driving process, environmental data collected from the driving environment is acquired; down-sampled the environmental data to obtain down-sampled environmental data; motion perception processing is performed based on the down-sampled environmental data to obtain motion data; and the motion data is predicted through a prediction model to obtain the motion trajectory of each object in the driving environment.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0175] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A motion trajectory prediction method, characterized in that: The method comprises: Acquire an environmental data sample obtained by collecting a target environment, and a downsampled environmental data sample obtained by downsampling the environmental data sample; Performing motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment; extracting, from the second motion data samples, a first target data sample of the target object in the target environment within a first time period; Extracting a second target data sample of the target object in a second time period from the first motion data sample; the second time period is a next time period adjacent to the first time period; Training a pre-trained machine learning model based on a sample set consisting of the first target data sample and the second target data sample to obtain a prediction model; The motion data obtained from the downsampled environmental data is predicted by the prediction model to obtain the motion trajectory of the object.

2. The method according to claim 1, characterized in that Extracting, from the first motion data sample, a second target data sample of the target object within the second time period includes: Extracting a first tracking identifier corresponding to the target object from the first motion data sample; searching, in the second motion data, for a second tracking identifier that matches the first tracking identifier; According to the second tracking identifier, a second target data sample of the target object within the second time period is extracted from the second motion data sample.

3. The method according to claim 1, characterized in that The method further comprises: determining a motion duration of each object in the target environment based on the first motion data; Among the objects in the target environment, if the corresponding motion duration of the object is greater than a preset duration, the object is regarded as a target object; the preset duration is greater than the sum of the length of the first time period and the length of the second time period.

4. The method according to claim 1, wherein The step of training a pre-trained machine learning model based on a sample set consisting of the first target data sample and the second target data sample to obtain a prediction model includes: Performing prediction based on the second motion data sample using the pre-trained machine learning model to obtain predicted trajectory data; Performing calculations based on the first motion data sample to obtain trajectory label data; Calculating a loss value based on the predicted trajectory data and the trajectory label data; The parameters of the pre-trained machine learning model are adjusted according to the loss value to obtain a prediction model.

5. The method according to claim 1, wherein The method further comprises: During the autonomous driving process, environmental data collected from the driving environment is obtained; Downsampling the environmental data to obtain downsampled environmental data; The predicting the motion data obtained from the downsampled environmental data using the prediction model to obtain the motion trajectory of the object includes: Performing motion sensing processing on the downsampled environmental data to obtain motion data; The motion data is predicted using the prediction model to obtain the motion trajectory of each object in the driving environment.

6. The method according to any one of claims 1 to 5, characterized in that The environmental data samples are point cloud data samples obtained by scanning the target environment with a laser radar, or image data samples obtained by capturing the target environment with a camera, or video data samples obtained by capturing the target environment with a camera.

7. A motion trajectory prediction device, characterized in that: The device comprises: An acquisition module is used to acquire an environmental data sample obtained by collecting the target environment, and a downsampled environmental data sample obtained by downsampling the environmental data sample; a processing module, configured to perform motion sensing processing on the environmental data samples and the downsampled environmental data samples respectively, to obtain first motion data samples and second motion data samples corresponding to each target object in the target environment; a training module configured to extract, from the second motion data samples, a first target data sample of a target object in the target environment within a first time period; and, from the first motion data samples, extract a second target data sample of the target object within a second time period, wherein the second time period is a time period next to the first time period; and train a pre-trained machine learning model based on a sample set consisting of the first target data sample and the second target data sample to obtain a prediction model; The prediction module is used to predict the motion data obtained from the downsampled environmental data using the prediction model to obtain the motion trajectory of the object.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images