Automated iterative method for trajectory prediction models, electronic device and storage medium

Abstract

The embodiment of the present application provides an automatic iteration method of a trajectory prediction model. The method comprises the following steps: receiving prediction scene data determined by a trajectory prediction model of a mobile device; storing the prediction scene data into a cloud scene library, and performing data shunting on all scene data in the cloud scene library to distinguish normal scene data and abnormal scene data; respectively performing perception labeling on the normal scene data and the abnormal scene data to obtain trajectory prediction training samples with labels; training the trajectory prediction model in the cloud based on the trajectory prediction training samples with labels; and downloading parameters of the trained trajectory prediction model to the mobile device to update and iterate the trajectory prediction model of the mobile device. The embodiment of the present application adopts a mode of cloud training based on vehicle-side reasoning, automatically collects valuable data in a targeted manner, and automatically completes data labeling, thereby improving the efficiency of iteration of the trajectory prediction model of the vehicle-side of the autonomous vehicle.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and more particularly to an automated iterative method for trajectory prediction models, an electronic device, and a storage medium. Background Technology

[0002] With the continuous development of artificial intelligence and machine learning technologies, deep learning technology has been increasingly used in the field of autonomous driving, especially in vehicle autonomous driving, environmental perception, and trajectory prediction. Deep learning-based trajectory prediction is a data-driven technology, belonging to supervised learning, which requires the pre-establishment of an offline model. Its performance depends on the diversity and accuracy of the data. To ensure the accuracy and robustness of the trajectory prediction model (a deep learning model) under actual driving conditions, the training data needs to cover as many autonomous driving scenarios as possible, and the model needs to be iteratively validated within these scenarios.

[0003] Autonomous driving scenarios can be categorized by driving risk into ordinary scenarios and abnormal scenarios. Abnormal scenarios, also known as the "long tail," refer to sudden, low-frequency, and unpredictable situations, such as intersections with malfunctioning traffic lights, roads under construction, or objects falling to the ground. These scenarios can all lead to traffic accidents. How to handle this complex and rare long-tail problem is a challenge for the autonomous driving industry and has gradually become a key factor restricting its development. To solve these problems, autonomous driving systems need to accumulate a large amount of data and continuously iterate and validate deep learning models such as trajectory prediction.

[0004] Data shows that there are over a billion cars worldwide, but an accident occurs on average every 30 seconds, indicating that traffic accidents are low-frequency events. To make self-driving cars a reality and ensure their safety, these low-frequency issues must be addressed, at least achieving the safe driving level of current human drivers, and perhaps even surpassing it in every aspect.

[0005] As can be seen from the above data, to comprehensively test the safety, comfort and other performance of a car, at least billions of kilometers of road testing are required. That is to say, tens of thousands or even hundreds of thousands of cars would be running 24 hours a day for hundreds of days. However, generating effective (valuable) problem data during the testing process is an extremely difficult task.

[0006] To address the aforementioned issues, traditional deep learning model iteration verification methods, such as trajectory prediction, employ a functional test-driven model iteration approach. This involves data collection driven by requirements and problems, manual analysis of labeled data, and design of optimization schemes. At the testing end, scenarios are manually built or random real-vehicle tests are conducted, ultimately forming a serial iterative process of labeling, development, and testing. This method is effective for software function development, allowing limited manpower to solve limited problems and achieve a specific range of functions.

[0007] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0008] Traditional methods of iterative verification of deep learning models, such as trajectory prediction, are insufficient for the true implementation of autonomous driving and for ensuring the safe operation of the entire autonomous driving system across all time periods and conditions. Their shortcomings are specifically manifested in the following three aspects:

[0009] (1) The traditional problem-driven approach mainly relies on the serial development mode to optimize the model, which has a long development and testing cycle and cannot be carried out in parallel.

[0010] (2) Manual data annotation is time-consuming and inefficient, and cannot automatically trigger the filtering of valuable problem data;

[0011] (3) Most tests are conducted by manually building typical scenarios or random testing to verify the model, but the coverage is low in actual operating scenarios.

[0012] The above three points illustrate that the existing methods are no longer able to meet the needs of a large number of problems in real-world driving scenarios, cannot automate the solution of most problems, and cannot effectively achieve the goal of implementing autonomous driving. Summary of the Invention

[0013] To address the limitations of existing technologies in automatically acquiring real-world scenario data, and the resulting difficulty in automating model iteration with the acquired data, this invention provides, in a first aspect, an automated iteration method for trajectory prediction models, applied in the cloud, comprising:

[0014] Receive the predicted scenario data determined by the trajectory prediction model of the mobile device;

[0015] The predicted scene data is stored in a cloud scene library, and all scene data in the cloud scene library is split to distinguish between normal scene data and abnormal scene data.

[0016] The normal scene data and the abnormal scene data are respectively labeled to obtain labeled trajectory prediction training samples;

[0017] The trajectory prediction model is trained in the cloud based on the labeled trajectory prediction training samples.

[0018] The parameters of the trained trajectory prediction model are sent to the mobile device to update and iterate the trajectory prediction model of the mobile device.

[0019] Secondly, embodiments of the present invention provide an automated iterative method for trajectory prediction models, applied to mobile devices, comprising:

[0020] When the preset abnormal triggering conditions are met, the training objective required for the trajectory prediction model of the mobile device is determined.

[0021] According to the training objective, sensor data is collected and input into the trajectory prediction model to obtain trajectory prediction results;

[0022] Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, prediction scene data reflecting the mobile device's location is constructed.

[0023] The predicted scenario data is sent to the cloud, and the trajectory prediction model is updated and iterated after receiving the parameters of the trajectory prediction model from the cloud.

[0024] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of an automated iterative method for a trajectory prediction model according to any embodiment of the present invention.

[0025] Fourthly, embodiments of the present invention provide a mobile device, including a main body and an electronic device according to any embodiment of the present invention mounted on the main body.

[0026] Fifthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of an automated iterative method for a trajectory prediction model according to any embodiment of the present invention.

[0027] Sixthly, embodiments of the present invention also provide a computer program product that, when the computer program product is run on a computer, causes the computer to execute the automated iterative method for trajectory prediction model as described in any one of the embodiments of the present invention.

[0028] The beneficial effects of this invention are as follows: The training iteration between the mobile device and the cloud leverages the cloud's powerful computing and data storage capabilities to automatically and in real-time complete a series of operations, including generating labeled datasets, model training, and model iteration. This allows for accurate extraction of data valuable for performance improvement from the anomaly-prone data collected by the mobile device. This mobile device and cloud training mode fully utilizes the cloud's resource advantages, improving the efficiency of iterative training for the autonomous vehicle trajectory prediction model. The mobile device deploys a multi-task, lightweight trajectory prediction model and uses scene markers to initially label the trajectory prediction results. This expands the scope of anomaly scene collection, enabling the mobile device to automatically and selectively collect data valuable for improving the vehicle trajectory prediction model's performance based on training objectives. However, this only indicates anomaly potential, not necessarily anomaly. The cloud can further extract valuable data for performance improvement from this anomaly-prone data. This mobile device and cloud training mode fully utilizes the cloud's resource advantages, improving the efficiency of iterative training for the autonomous vehicle trajectory prediction model. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of an automated iterative method for a trajectory prediction model provided in an embodiment of the present invention;

[0031] Figure 2 This is a system architecture diagram of an automated iterative method for trajectory prediction models provided in an embodiment of the present invention.

[0032] Figure 3 This is a flowchart of an automated iterative method for a trajectory prediction model provided in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the trajectory prediction results output by a map-based trajectory prediction model, which is part of an automated iterative method for trajectory prediction models provided in an embodiment of the present invention.

[0034] Figure 5 This is a flowchart of an automated iterative method for a trajectory prediction model provided in an embodiment of the present invention;

[0035] Figure 6This is a flowchart of an automated iteration method for a trajectory prediction model provided in an embodiment of the present invention.

[0036] Figure 7 This is an example diagram illustrating how a mobile device data acquisition module collects data that is valuable for improving the performance of a trajectory prediction model, according to an embodiment of the present invention, in an automated iterative method for a trajectory prediction model.

[0037] Figure 8 This is an example diagram showing the types of elements included in a scenario of an automated iterative method for a trajectory prediction model provided in an embodiment of the present invention;

[0038] Figure 9 This is an example diagram illustrating how a mobile device data acquisition module collects data that is valuable for improving the performance of a trajectory prediction model, according to another embodiment of the present invention, in an automated iterative method for a trajectory prediction model.

[0039] Figure 10 This is a schematic diagram of the inference results and expected values ​​of a trajectory prediction model based on an automated iterative method for trajectory prediction models provided in an embodiment of the present invention.

[0040] Figure 11 This is a schematic diagram of a vehicle computing system provided in an embodiment of the present invention;

[0041] Figure 12 This is an example of an autonomous vehicle and an on-board execution device provided in an embodiment of the present invention;

[0042] Figure 13 This is a schematic diagram of a cloud execution device structure provided in an embodiment of the present invention;

[0043] Figure 14 This is a schematic diagram of the structure of a cloud execution device according to an embodiment of the present invention;

[0044] Figure 15 This is a schematic diagram of the structure of a mobile device-side execution device according to an embodiment of the present invention;

[0045] Figure 16 This is a schematic diagram of an embodiment of an electronic device for automated iteration of a trajectory prediction model, provided as an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Those skilled in the art will recognize that embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0048] For ease of understanding, the technical terms used in this application are explained below:

[0049] The term "mobile device" as used in this application can refer to any device capable of mobility, including but not limited to automobiles, ships, submarines, airplanes, aircraft, etc. Among them, automobiles include vehicles with six levels of autonomous driving technology, L0-L5, as defined by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Classification of Driving Automation for Automobiles", hereinafter referred to as autonomous driving vehicles (ADV).

[0050] The term "Autonomous Vehicle (ADV)" as used in this application can refer to vehicle equipment or robotic equipment with the following various functions:

[0051] (1) Passenger transport function, such as family cars, buses, etc.;

[0052] (2) Cargo carrying function, such as ordinary trucks, box trucks, trailers, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks, etc.

[0053] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, hoists, excavators, bulldozers, loaders, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc.

[0054] (4) Entertainment functions, such as recreational vehicles, amusement park automatic driving devices, balance bikes, etc.;

[0055] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, and engineering emergency rescue vehicles.

[0056] like Figure 1 The diagram shows a flowchart of an automated iterative method for a trajectory prediction model according to an embodiment of the present invention, comprising the following steps:

[0057] S11: Receive the predicted scene data determined by the trajectory prediction model of the mobile device;

[0058] S12: Store the predicted scene data in the cloud scene library, and perform data splitting on all scene data in the cloud scene library to distinguish between normal scene data and abnormal scene data;

[0059] S13: Perform perceptual annotation on the normal scene data and the abnormal scene data respectively to obtain labeled trajectory prediction training samples;

[0060] S14: Train the trajectory prediction model in the cloud based on the labeled trajectory prediction training samples;

[0061] S15: Send the parameters of the trained trajectory prediction model to the mobile device to update and iterate the trajectory prediction model of the mobile device.

[0062] In this embodiment, the trajectory prediction model is automatically iterated from two aspects (cloud and mobile device) to effectively solve the practical problems of long model iteration cycle and low verification efficiency.

[0063] For the cloud-based execution device, the device includes a cloud data acquisition module, a trajectory sample automatic annotation module, a trajectory prediction model training module, and a trajectory prediction model iterative model, with the specific structure as follows: Figure 2 (The upper part, the cloud area), is shown. The cloud-based execution device can be implemented by a cloud server. Data transmission between the cloud-based execution device and the mobile device is achieved through a communication interface. This communication interface can employ vehicle-to-everything (V2X) wireless communication technology, vehicular Ethernet, 3G / 4G / 5G mobile communication technology, etc., and is not limited here.

[0064] For step S11, during the mobile device's operation, it continuously collects environmental information through sensors to construct the predicted scene data of the mobile device's current location.

[0065] In one implementation, the predicted scene data is determined by the trajectory prediction result with scene markers output by the trajectory prediction model. In this implementation, the mobile device sends the predicted scene data with scene markers to the cloud (wherein the scene markers are marked by the anomaly triggering module in the mobile device). The cloud data acquisition module in the cloud collects this data sent by the mobile device, which may be valuable for training the trajectory prediction model (that is, the mobile device expands the collection range of abnormal scenes, but the mobile device's computing power is limited, and it can only collect data with abnormal tendencies during driving; further operations require processing by the cloud with powerful computing capabilities).

[0066] Specifically, the cloud-based data acquisition module has data acquisition capabilities and sends the acquired data to the cloud database. Through continuous data acquisition, the cloud database becomes richer, and in subsequent steps, the host computer analyzes and processes the data. There is a data transmission relationship between the cloud-based data acquisition module and the mobile device's data acquisition module; the cloud-based data acquisition module obtains potentially valuable data from the vehicle-side acquisition module as needed.

[0067] In the cloud data acquisition module, data can be acquired and sent via buses such as USB, PXI, PCI, PCI Express, FireWire (1394), PCMCIA, ISA, Compact Flash, 485, 232, Ethernet, and various wireless networks.

[0068] For step S12, the data in the continuously expanding cloud database is split into data from abnormal scenarios and data from normal scenarios.

[0069] As one implementation method, the data diversion of all scene data in the cloud scene library includes:

[0070] The scene flags are used to split all scene data in the cloud scene library, distinguishing between normal scene data and abnormal scene data.

[0071] In this embodiment, scene markers can be used to easily distinguish between normal and abnormal scene data, further simplifying the calculation process. For abnormal data, multiple data-driven model corrections are applied in subsequent steps, along with a more stringent trajectory screening scheme, and data augmentation strategies with different training objectives are employed in the subsequent trajectory prediction model training module.

[0072] For step S13, multiple data-driven models are used to perform perceptual annotation correction on the normal scene data and abnormal scene data in the previous step. This step can be called a data-driven sub-module in the cloud.

[0073] As one implementation method, the perception annotation method includes: fusing perception and / or multi-target tracking and / or detection and tracking integration and / or pre-trained model tuning.

[0074] In this embodiment, taking fusion perception as an example, a data-driven model is used to perform fusion perception on the scene data output by the data splitting submodule (to generate trajectory information of obstacles throughout their entire lifecycle). Specifically, in scene data identified as having anomalies, the powerful computing capabilities of the cloud are used to determine the true trajectory prediction data and mark it as abnormal. Compared to the limitations of mobile device computing resources, which require rapid inference to obtain detection results with limited computing power, cloud computing resources not only have powerful computing capabilities but also lower real-time requirements. Therefore, for the same target, fusion perception using a data-driven model in the cloud can yield more accurate results. This result can then be used as labeled data to train the vehicle-side trajectory prediction model, achieving the goal of training the model and improving its inference capabilities (making the inferred trajectory prediction results more accurate).

[0075] Specifically, the fusion perception results include the number of obstacles, timestamps, and frame number of each obstacle target (e.g., obstacles include surrounding landscapes and traffic facilities in a static environment, with each obstacle marked with a different number), for example, the tracking numbers of surrounding landscapes #2 and traffic facilities #3, categories (e.g., categories of static environments include obstacles, surrounding landscapes, traffic facilities, and roads; categories of traffic participants include (e.g., vehicles, pedestrians, and animals), and categories can be further refined, such as motor vehicles: passenger cars, buses, trucks, etc., and vulnerable road users: pedestrians, cyclists, etc.), confidence (a number between [0,1]), horizontal, vertical, and three-dimensional coordinates (unit: meters), length, width, and height (unit: meters), and heading angle (unit: radians), forming the trajectory information of obstacles throughout their entire lifecycle.

[0076] Taking multi-object tracking and / or integrated detection and tracking and / or pre-trained model tuning as an example, the data-driven model is a deep learning model. It can employ any deep learning method, such as YOLO+Deep Sort (a detection-based multi-object tracking method), CenterTrack (a keypoint-based multi-object tracking method that estimates the target motion in the current frame by performing detection on an image pair and combining the target detection results from previous frames), integrated detection and tracking methods, or SimTrack (which integrates target association, dead object removal, and new-born object detection, reducing the complexity of the tracking system). These methods will not be elaborated upon further here.

[0077] For abnormal scenario data in the data diversion submodule, a more stringent perception standard is adopted to perform consistency checks on the fusion perception results based on different algorithm logics. All perception results must be consistent (such as the tracking sequence remaining unchanged for a certain period of time, and the trajectory motion pattern formed after fusion). Only when the trajectory deviation is less than a certain threshold (e.g., 1 meter) can the result be sent to the next data optimization submodule for processing.

[0078] For step S14, the trajectory information generated by the perception annotation in the previous step is optimized and filtered. This step can be called the data optimization submodule.

[0079] As one implementation, optimizing the trajectory prediction data of the anomaly markers to output smooth trajectory prediction data for use as multiple training samples for the trajectory prediction model includes:

[0080] The trajectory prediction data of the anomaly markers are optimized based on curve fitting filtering and / or velocity filtering and / or behavior filtering and / or higher-order derivative filtering to output smooth trajectory prediction data.

[0081] In this embodiment, optimized filtering can filter the trajectories of the same obstacle, discarding fusion perception errors or inaccurate and abrupt results, and outputting smooth, high-quality trajectory data. Specific filtering methods include: curve fitting-based filtering and / or velocity-based filtering and / or behavior-based filtering and / or higher-order derivative-based filtering.

[0082] The filtering method based on curve fitting first uses polynomial curve fitting (quadratic polynomial curve, second-order or third-order Bezier curve, which can be a single or multiple combinations) to fit all the waypoints on each obstacle trajectory to generate a smooth curve. Then, the average position offset error between the real waypoints and the corresponding waypoints on the fitted curve is calculated. Finally, a high-quality trajectory sample is selected using an error threshold.

[0083] The speed-based filtering method: Based on curve fitting filtering, it further randomly selects obstacle targets with different speeds according to their corresponding probabilities. Since the obstacle targets are slow and have a high probability of being discarded, a dictionary type is first used to implement the key-value pair correspondence between speed obj_speed and discard probability obj_p: {obj_speed:obj_p}. The relationship between multiple speeds and discard probabilities is {obj_speed1:obj_p1, obj_speed2:obj_p2,...,obj_speedi:obj_pi}. The simulated probability generated by 0-1 random numbers is compared with the discard probability to finally decide whether to discard the sample. Specifically, if the speed of an obstacle target is lower than a certain threshold obj_speedi, and the simulated probability random_p generated by 0-1 random numbers is greater than the discard speed obstacle probability obj_i_p, then the trajectory sample of that obstacle target is discarded.

[0084] The behavior-based filtering method, building upon speed filtering, randomly filters obstacle targets during lane keeping based on corresponding probabilities. This is achieved by judging driving behavior through the consistency between the starting and ending lanes of the search trajectory, and then using 0-1 random numbers to generate simulated probabilities to determine whether to discard trajectory samples. Specifically, the lane number of the trajectory starting point is first determined by the lateral and longitudinal positions, heading angle, and lane boundaries marked by map file elements. Similarly, the lane number of the trajectory ending point is determined. Then, it is checked whether the lane numbers of the starting and ending points are consistent. If they are inconsistent, the corresponding trajectory sample is retained; otherwise, the simulated probability generated by 0-1 random numbers is compared with the lane keeping discard probability to decide whether to discard the trajectory sample. If the simulated probability is less than the lane keeping discard probability, the trajectory sample is retained; otherwise, it is discarded.

[0085] Higher-order derivative filtering: Building upon behavior-based filtering, this method filters out trajectories that do not conform to the true physical meaning by using the higher-order derivatives of trajectory points (horizontal and longitudinal positions), thus ensuring the quality of trajectory data. Specifically, firstly, trajectory samples from abnormal scene data are evaluated for quality based on the higher-order derivatives of future trajectories. The path points of the differentiated trajectory samples are then analyzed to determine if there are any anomalies in the acceleration (second derivative of trajectory position point), acceleration change rate (third derivative of trajectory position point), and angle change rate (tangent values ​​of longitudinal and lateral positions) of the trajectory's starting and ending points. If the statistical value of any abnormal physical quantity exceeds a certain threshold, the corresponding trajectory sample is discarded for further screening. Then, based on the obstacle target's historical motion state, the lateral angle change rate (first derivative of lateral angle) and physically meaningful acceleration values ​​are discarded, completing the cascaded judgment of obstacle trajectory samples.

[0086] In step S15, by selectively collecting data valuable for improving the performance of the mobile device trajectory prediction model, and through the progressive optimization processes of data diversion, perception labeling, and optimization described above, valuable data for the training objective is gradually selected, thereby quickly and effectively achieving the training goal. The parameters of the trajectory prediction model trained using this valuable data are then distributed to the mobile device for model updates and iterations.

[0087] As one implementation method, the step of sending the parameters of the trained trajectory prediction model to the mobile device for updating and iterating the trajectory prediction model includes:

[0088] The trained trajectory prediction model is tested. If the test results meet the iterative requirements of the training objective, the parameters of the trained trajectory prediction model are sent to the mobile device for iteration.

[0089] In this implementation, the trained trajectory prediction model is tested. Only when the test results meet the iteration requirements (indicating that the inference ability of the trained vehicle trajectory prediction model is significantly better than the currently used vehicle trajectory prediction model) are the model parameters sent to the vehicle-side computing module to complete the iterative operation of the vehicle trajectory prediction model. If the test results do not meet the iteration requirements, sending the model parameters to the mobile device may have the opposite effect. This further ensures the accuracy of the iteration.

[0090] Meanwhile, since the data-driven model has been trained, it can be further tested. When the test results meet the iteration requirements (indicating that the reasoning ability of the trained data-driven model is significantly better than that of the data-driven model currently in use), the model parameters will be sent to the automatic labeling module to complete the data-driven model iteration operation.

[0091] This implementation demonstrates that the training iteration between mobile devices and the cloud leverages the cloud's powerful computing and data storage capabilities to automatically and in real-time complete a series of operations, including generating labeled datasets, training models, and iterating models. This allows for the accurate extraction of data that is valuable for performance improvement from the data collected from mobile devices that exhibits anomalies. This mobile device and cloud training model fully utilizes the resource advantages of the cloud and improves the efficiency of trajectory prediction model iteration for autonomous vehicles.

[0092] like Figure 3 The diagram shows a flowchart of an automated iterative method for a trajectory prediction model according to an embodiment of the present invention, comprising the following steps:

[0093] S21: When the preset abnormal triggering conditions are met, determine the training objective required for the trajectory prediction model of the mobile device;

[0094] S22: According to the training objective, collect sensor data and input it into the trajectory prediction model to obtain the trajectory prediction result;

[0095] S23: Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, construct prediction scene data reflecting the location of the mobile device;

[0096] S24: Send the predicted scene data to the cloud, and update and iterate the trajectory prediction model after receiving the parameters of the trajectory prediction model from the cloud.

[0097] In this embodiment, the trajectory prediction model is automatically iterated from two aspects (cloud and mobile device) to effectively solve the practical problems of long model iteration cycle and low verification efficiency.

[0098] For the execution device of the mobile device, this device is used for mobile device data acquisition and mobile device model inference. It includes: a mobile device data acquisition module, an anomaly triggering module, and a mobile device calculation module. The mobile device calculation module is configured with a vehicle trajectory prediction model. For example, mobile devices are typically vehicles. Figure 2 As shown (the lower half of the vehicle end).

[0099] For step S21, since this method aims to make the model better and better, but the model optimization is divided into different directions, that is, different training objectives, simply put, it includes: supplementing and enriching the scenarios covered by the trajectory prediction model, enabling the vehicle-side trajectory prediction model to cover long-tail scenarios, improving the trajectory prediction model in scenarios where the inference effect does not meet the predetermined requirements, and maintaining the inference ability of the trajectory prediction model in scenarios where the inference effect meets the predetermined requirements.

[0100] As one implementation method, the exception triggering condition includes one or more of the following:

[0101] The event is triggered when the mobile device detects that a predetermined scene is included in a map collected during the journey.

[0102] Triggered when the mobile device detects that the status during travel is a predetermined status data;

[0103] The system triggers when the mobile device predicts that an obstacle falls under a predetermined scenario based on collected map data. The predetermined scenario includes one or more of the following: intersections, roundabouts, and accident-prone areas.

[0104] The predetermined state data includes one or more of the following: rapid acceleration, emergency braking, and sharp turning;

[0105] The obstacles fall under the predetermined conditions, including when the predicted trajectory of the obstacle exceeds the impassable area.

[0106] In this embodiment, the anomaly triggering module is used to automatically trigger events on the mobile device. Simply put, during driving, if potentially dangerous areas are detected on the map, or if the mobile device experiences sudden acceleration or emergency braking, these could pose hidden dangers. Similarly, if the predicted trajectory of an obstacle appears in an impassable area, these all pose certain risks to driving. This method needs to collect data on these potentially dangerous anomaly scenarios. Specifically, under the known normal operation of perception fusion (autonomous driving system: perception fusion -> trajectory prediction -> decision planning), anomaly triggering will occur in at least the following situations:

[0107] (1) Triggering mobile device driving scenarios: road intersections, roundabouts, accident-prone areas (specific information can be obtained from the map data collected by the mobile device data collection module. For example, based on historical information from the map, if a certain intersection has a relatively high frequency of traffic accidents due to narrow roads or traffic transfers, then data for such scenarios will be collected).

[0108] (2) Mobile device status data trigger: sudden acceleration, sudden braking, sudden turning (information can be obtained from the vehicle status data collected by the mobile device data acquisition module. During the operation of the mobile device, sudden braking or sudden turning is usually not possible. When this happens, it can be associated with the sudden appearance of a pedestrian in front. The driver will brake or turn suddenly to avoid the pedestrian. This type of data should also be collected).

[0109] (3) Model inference trajectory triggering: The predicted trajectory of the obstacle exceeds the impassable area (information can be obtained from the map data collected by the mobile device data acquisition module, such as guardrails. Generally speaking, various obstacles placed in traffic roads are placed in specific areas and will not directly affect the driving of the mobile device. However, if these obstacles appear in places that should not appear, they may also cause safety hazards. Therefore, this type of data should also be collected), which may cause a collision.

[0110] An autonomous driving scene flag (scene_flag, default value 0) is added to distinguish between abnormal scenes (scene_flag=1) and normal scenes (scene_flag=0). In general, these triggering conditions correspond to the training objectives.

[0111] For step S22, mobile devices are usually equipped with various sensors to obtain environmental information about the mobile device, which is called the mobile device data acquisition module.

[0112] In one embodiment, the sensor includes:

[0113] Millimeter-wave radar, infrared radar, lidar, Doppler radar, light sensors, rain sensors, monocular cameras, binocular cameras, panoramic cameras, fisheye cameras, dashcams, vehicle attitude sensors, speed sensors, Global Positioning System (GNSS), and Inertial Measurement Unit (IMU). These sensors can acquire various environmental data, including laser data, point clouds, and images.

[0114] The mobile device data acquisition module has data acquisition capabilities. For example, taking a vehicle as an example, the data collected by sensors in an autonomous vehicle is analyzed and processed by a host computer. Specifically, it can be used to collect analog or digital signals from various sensors installed in the autonomous vehicle that perceive the surrounding environment. It can also collect data from the vehicle's computational model for inference through the vehicle's trajectory prediction model, and it can also collect vehicle status data, map data, and driver operation data. The mobile device acquisition module has a built-in data acquisition card (i.e., a computer expansion card that implements data acquisition functions), which can acquire and send data via buses such as USB, PXI, PCI, PCI Express, FireWire (1394), PCMCIA, ISA, Compact Flash, 485, 232, Ethernet, and various wireless networks.

[0115] Furthermore, the mobile device data acquisition module also has data processing capabilities. Specifically, it works in conjunction with the cloud-based data acquisition module to extract valuable data from the collected data that can improve the performance of the vehicle-side trajectory prediction model. This information, combined with the trajectory prediction results, can comprehensively reflect the scenario in which the autonomous vehicle is located, making it more meaningful for training the trajectory prediction model.

[0116] After environmental information is collected, it is input into the trajectory prediction model to obtain trajectory prediction results, which can be processed in the mobile device's computing module.

[0117] As one implementation method, the step of inputting data collected by sensors mounted on the mobile device into the trajectory prediction model to obtain trajectory prediction results includes:

[0118] Predict the behavior patterns of mobile devices under each behavioral modality;

[0119] The probability distribution of multiple predicted trajectories is determined by the horizontal and vertical positions of the trajectory at each prediction time point within the entire prediction duration in the behavioral pattern.

[0120] The predicted trajectory of the maximum probability distribution is determined as the trajectory prediction result.

[0121] In this embodiment, the mobile device computing module can be used to implement functions such as fusion perception, trajectory prediction, and decision planning for autonomous driving mobile devices. The mobile device trajectory prediction model has inference capabilities and can be used to implement trajectory prediction functions for autonomous driving mobile devices. As one implementation method, the mobile device model can be a deep learning model or a non-deep learning model; the following example uses a deep learning model. The "mobile device trajectory prediction model" referred to in this application embodiment is the mobile device model that implements the trajectory function. The "trajectory prediction result" referred to in this application embodiment is the result of inference from the mobile device trajectory prediction model.

[0122] The trajectory prediction results are illustrated as follows: Figure 4 As shown, in the left image, green (G) indicates the highest probability of going straight; in the middle image, magenta (M) indicates the highest probability of turning left; and in the right image, black (B) indicates the highest probability of turning right.

[0123] The target trajectory under each behavioral modality corresponds to each behavioral pattern (m1, m2, ..., m i , ..., m H Under the given prediction duration T, the horizontal and vertical position points (pre-x) of the trajectory at each prediction time point (1, 2, ..., i, ..., T) are given. i ,pre_y i ), recorded as ((pre_x1, pre_y1), (pre_x2, pre_y2),..., (pre_x T ,pre_y T )) mi This corresponds to one of the multiple predicted trajectories of the target vehicle in the diagram (represented by red (r), green (G), blue (b), magenta (M), and black (B) trajectory lines), where mode H = 5. The trajectory probability for each behavioral mode corresponds to a probability value of 0 to 1 represented by the legend, and the five modes correspond to the five colors of the probability values ​​represented by the legend: red, green, blue, magenta, and yellow. The multimodal prediction output is a mixture of probability distributions for multiple modes, and the overall approach is consistent with MDN (Mixture Density Network).

[0124] For multimodal intent and trajectory prediction problems, the optimization objective is to minimize trajectory and distribution errors. Single-modal calculations calculate the mean squared error of the trajectory point's position relative to the true position within the prediction period. Multimodal calculations require considering both trajectory position and position distribution (multi-task learning). The goal is to maximize the probability of the true position in the mixed probability density, using the maximum likelihood function of the joint probability distribution as the loss function. Multimodal partitioning can be implicitly learned based on the model's multi-task loss function, or explicitly defined based on the lateral and longitudinal offsets of the trajectory endpoint and the turning radius (lateral offset exceeding a certain threshold indicates a turn, turning radius exceeding a certain threshold indicates a wide turn, and lateral offset exceeding a certain threshold indicates acceleration / deceleration).

[0125] by Figure 4 For reference, the behavioral modalities, trajectory representations, and color coding schemes in the prediction process are as follows:

[0126] Behavioral modalities: Accelerate straight ahead, normal straight ahead, decelerate straight ahead, turn small curve, turn large curve

[0127] The trajectory is represented by red, green, blue, magenta, and black.

[0128] Color coding [0,0,255] [0,255,0] [255,0,0] [255,0,255] [255,255,255]

[0129] Since the dimension of the target trajectory under each behavioral modality is 2×T and the dimension of the trajectory probability under each behavioral modality is 1, the output dimension of the target trajectory prediction result is (H×2×T)+(H×1). This yields the corresponding trajectory prediction result.

[0130] For step S23, in addition to the trajectory prediction results, the corresponding spatiotemporal synchronization information also needs to be obtained.

[0131] As one implementation method, the spatiotemporal synchronization information includes:

[0132] The environmental data, map data, mobile device status data, and driver operation data are synchronized with the trajectory prediction results in time and space. The environmental data includes: static environment, dynamic environment, communication environment, traffic participants, meteorological environment, and environmental data collected by sensors.

[0133] The map data includes: maps, traffic control information, and navigation information;

[0134] The mobile device status data includes: basic attributes of the mobile device, location of the mobile device, motion status of the mobile device, and human-computer interaction tasks.

[0135] Specifically, the static environment includes: fixed obstacles, buildings, transportation facilities, and roads;

[0136] Dynamic environments include: dynamic traffic lights and traffic police;

[0137] The communication environment includes: signal strength, signal delay time, and electromagnetic interference intensity;

[0138] Traffic participants include: pedestrians, motor vehicles, non-motorized vehicles, and animals;

[0139] Meteorological environment includes: temperature, humidity, light conditions, and weather conditions;

[0140] Basic attributes of mobile devices include: vehicle weight, geometric dimensions, and basic performance.

[0141] The location of the mobile device includes: coordinates and lane position;

[0142] Motion states include: lateral motion state and longitudinal motion state;

[0143] Human-computer interaction tasks include: entertainment and driving tasks.

[0144] The vehicle status data, map data, driver operation data, etc., mentioned above are used as input data for the vehicle-side computing module to perform inference, thereby realizing the functions of fusion perception, trajectory prediction, decision planning, etc. of autonomous vehicles, and obtaining the predicted scene data by predicting the trajectory in the simulated scenario.

[0145] Considering the characteristics of mobile device computing resources, such as high cost, limited computing power, and fast inference speed, mobile device trajectory prediction models can adopt network structures with features such as multi-tasking and lightweight design. Multi-tasking refers to the network structure's ability to share parameters and tasks, while lightweight design means that the network structure can meet computational efficiency and capability requirements within limited storage space and power consumption constraints.

[0146] Multi-task refers to the ability to reuse the results of perception fusion and map feature information (as known inputs) to obtain the results required for multiple tasks through a single model inference. Examples include simultaneously predicting the behavior categories of multiple obstacles (multi-agent), future driving trajectories, and future speed change trends. Lightweight design adapts to the limited computing power of mobile devices while meeting inference efficiency requirements. Mobile device trajectory prediction models can also employ multi-dimensional network structures, which can help uncover the intrinsic connections (e.g., interaction information) between multiple targets.

[0147] In step S24, the predicted scene data determined in the above steps is sent to the cloud, and the parameters of the trajectory prediction model received from the cloud are automatically iterated. That is, the trajectory prediction model currently being used by the mobile device's computing resources is iterated into the trained trajectory prediction model.

[0148] This implementation demonstrates that the training iterations on mobile devices and the cloud, the deployment of multi-tasking, lightweight trajectory prediction models on mobile devices, and the automatic and targeted collection of data valuable for improving the performance of vehicle-side trajectory prediction models based on fused perception results and trajectory prediction results.

[0149] like Figure 5 The diagram shows a flowchart of an automated iterative method for a trajectory prediction model according to an embodiment of the present invention, comprising the following steps:

[0150] S31: When the mobile device meets the preset abnormal triggering conditions, determine the training objective required for the trajectory prediction model of the mobile device, and collect sensor data and input it into the trajectory prediction model according to the training objective to obtain the trajectory prediction result.

[0151] S32: Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, construct prediction scene data reflecting the location of the mobile device;

[0152] S33: Send the predicted scene data to the cloud;

[0153] S34: The cloud receives the predicted scene data determined by the trajectory prediction model of the mobile device;

[0154] S35: The cloud stores the predicted scene data in the cloud scene library, and performs data splitting on all scene data in the cloud scene library to distinguish between normal scene data and abnormal scene data;

[0155] S36: The cloud performs perception annotation on the normal scene data and the abnormal scene data respectively to obtain labeled trajectory prediction training samples;

[0156] S37: The trajectory prediction model is trained in the cloud based on the labeled trajectory prediction training samples;

[0157] S38: The cloud sends the parameters of the trained trajectory prediction model to the mobile device;

[0158] S39: After receiving the parameters of the trajectory prediction model fed back from the cloud, the mobile device updates and iterates the trajectory prediction model.

[0159] In this embodiment, a combination of cloud and mobile device is used, with the mobile device being a vehicle-mounted unit as an example. The overall structure is as follows: Figure 2 As shown.

[0160] Mobile devices selectively collect data valuable for improving the performance of trajectory prediction models (data originates from the mobile device's data acquisition module and anomaly triggering module). This valuable data is then used in the cloud for optimization, followed by training and iteration of the trajectory prediction model. This method allows for the targeted extraction of valuable data based on training objectives, thereby achieving the training goals quickly and effectively. The automatic iteration process of the trajectory prediction model is as follows: Figure 6 As shown.

[0161] As one implementation method for the training objectives, the training objectives include: supplementing and enriching the scenarios covered by the trajectory prediction model; enabling the vehicle-side trajectory prediction model to cover long-tail scenarios; improving the trajectory prediction model's ability to handle scenarios where the inference effect does not meet the predetermined requirements; and maintaining the trajectory prediction model's inference ability to handle scenarios where the inference effect meets the predetermined requirements.

[0162] 1. When the training objective is to supplement and enrich the scenarios covered by the trajectory prediction model, the method includes:

[0163] The mobile device's mobile device acquisition module collects data from various scene types and sends the predicted scene data generated using the data from each scene type to the cloud data acquisition module in the cloud.

[0164] When the cloud data acquisition module determines that the category corresponding to the predicted scene data is missing in the cloud scene library, or the amount of data under the category corresponding to the predicted scene data in the cloud scene library does not reach a preset threshold, the cloud data acquisition module receives the predicted scene data sent by the mobile device acquisition module and determines it as data to be collected to supplement and enrich the scenes covered by the trajectory prediction model.

[0165] In this implementation, the purpose of prioritizing the inference range is to allow the trajectory prediction model to cover (adapt to) as many scenarios as possible. For example... Figure 7 As shown, the mobile device acquisition module constructs a scene using the inference results of the trajectory prediction model and its spatiotemporal synchronization information, and uploads it to the cloud. When the cloud acquisition module determines that there is no scene uploaded by the mobile device data acquisition module in the existing scene library, it collects the trajectory prediction results and its spatiotemporal synchronization information as valuable data for improving the performance of the vehicle-side trajectory prediction model.

[0166] The scenarios in the predicted scenario data include: mobile device features and environmental features;

[0167] The categories of elements of the mobile device itself include: basic attributes including weight, geometric information, and performance information; location information including coordinate information and road location; lateral and longitudinal motion state information; and driving task information including perception and recognition, path planning, human-computer interaction, and network communication.

[0168] The categories of environmental elements include: static environment including obstacles, landscape, traffic facilities, and roads; dynamic environment including dynamic signage facilities and communication information; traffic participants including other mobile devices, pedestrians, and animals; and meteorological information including temperature, humidity, lighting conditions, and weather conditions.

[0169] In this embodiment, in the field of autonomous driving testing, a scenario refers to a dynamic description of the comprehensive interaction process between an autonomous driving mobile device and other factors such as other vehicles, roads, traffic facilities, and weather conditions within a certain time and space range. Figure 8 An example of the element types in the scene is shown.

[0170] The cloud database stores a scene library containing various scenes covered by the mobile device trajectory prediction model. If the cloud data acquisition module compares the scene uploaded by the mobile device data acquisition module with the scenes already in the scene library and finds that the scene is not in the scene library, it means that the mobile device trajectory prediction model cannot yet cover (adapt to) this scene and needs to add this scene to the scene library. At this time, the cloud acquisition module issues a command, and after receiving the command, the mobile device data acquisition module will collect the trajectory prediction result corresponding to this scene and its spatiotemporal synchronization information as valuable data for improving the performance of the vehicle trajectory prediction model.

[0171] Specifically, when the cloud-based data acquisition module compares the scenes uploaded by the mobile device's data acquisition module with the scene library, the scene library is considered to be missing if either of the following two situations occurs:

[0172] 1. The scene category corresponding to this scene is missing from the cloud-based scene library:

[0173] This situation directly indicates that the scene library has not yet covered the category corresponding to the scene. For example, the scene library covers three categories under road type: urban roads, highways, and park roads, but the scene category uploaded by the vehicle data collection module is rural roads. In this case, it can be determined that the scene is missing from the scene library.

[0174] 2. The scene library in the cloud contains a category corresponding to this scene, but the amount of data under this category in the existing scene library has not reached the predetermined quantity:

[0175] This situation indicates that although the scenario library already covers the scenario, the amount of data corresponding to the scenario is still relatively small, while model training requires a sufficient amount of data. In this case, it is still necessary to consider that the scenario library is missing the scenario, and the trajectory prediction results corresponding to the scenario and its spatiotemporal synchronization information should be uploaded to the cloud acquisition module as valuable data for improving the performance of the vehicle trajectory prediction model.

[0176] As another implementation, for mobile devices, the mobile device acquisition module collects data for each scene type and sends the encoded predicted scene data generated using the data for each scene type to the cloud data acquisition module in the cloud.

[0177] After receiving the acquisition command from the cloud data acquisition module, the mobile device acquisition module sends the predicted scene data to the cloud data acquisition module.

[0178] For the cloud, the cloud data acquisition module receives the encoded predicted scene data sent by the mobile device acquisition module of the mobile device;

[0179] Use the corresponding scene encoding library in the cloud scene library to determine whether the encoding of the predicted scene data is stored;

[0180] If the scene encoding library does not store the encoding, the cloud data acquisition module sends an acquisition command to the mobile device acquisition module and receives the predicted scene data uploaded by the mobile device acquisition module based on the acquisition command.

[0181] In this embodiment, considering that the scene contains a large amount of information, uploading the entire scene not only wastes communication resources but also affects the collection efficiency. However, not all scenes are valuable for improving the performance of the mobile device trajectory prediction model (the scene library may already contain the scene library). In this case, the mobile device data acquisition module can encode the scene and upload it to the cloud to save communication resources and speed up data acquisition efficiency.

[0182] In addition to the scene library, the cloud database also stores the corresponding encoding library (the scene library contains the scene codes from the scene library). The cloud data acquisition module compares the scene codes downloaded from the mobile device acquisition module with the encoding library. If it is determined that there is no scene code in the scene encoding library, it can be determined that the mobile device trajectory prediction model cannot cover (adapt to) the scene, so the scene needs to be added to the scene library.

[0183] At this point, the cloud data acquisition module issues an instruction, and the mobile device acquisition module receives the instruction and uses the corresponding trajectory prediction results and their spatiotemporal synchronization information as important data to improve the performance of the mobile device trajectory prediction model.

[0184] Specifically, the vehicle-side data acquisition module should encode the scene according to a predetermined encoding rule, which can be based on encoding scene elements.

[0185] For example, targeting Figure 8The scene shown encodes scene elements according to their order within their parent node elements. For each specific element, the number after the # indicates the current element's order within its parent node elements:

[0186] If the scene includes pedestrians, then from left to right, the code corresponding to external environmental elements is 2, the code corresponding to traffic participants is 3, and the code corresponding to pedestrians is 2. Therefore, the scene code contains the number 232.

[0187] If the scene includes lateral movement, then from left to right, the code corresponding to the vehicle itself is 1, the code corresponding to the movement state is 3, and the code corresponding to the lateral movement state is 1. Therefore, the scene code contains the number 131.

[0188] If the scene contains both pedestrians and lateral movement states, then the scene encoding includes the corresponding data groups (232, 131).

[0189] This implementation demonstrates that scene judgment ensures effective expansion of the cloud-based scene library. Furthermore, considering communication resource consumption, the use of encoding for comparison ensures efficient scene acquisition. It should be noted that the predicted scene data is constructed from the trajectory prediction results and related spatiotemporal synchronization information. Therefore, the trajectory prediction results also store corresponding spatiotemporal synchronization information. During judgment, the corresponding spatiotemporal synchronization information can also be judged independently. Several training objectives described below can determine the corresponding spatiotemporal synchronization information, which will not be elaborated upon here.

[0190] 2. When the training objective is to enable the vehicle trajectory prediction model to cover long-tail scenarios, the mobile device and the cloud jointly perform the processing.

[0191] For the cloud, the cloud data acquisition module receives trajectory prediction results that do not belong to the preset conventional scenarios, which are sent by the mobile device acquisition module for long-tail scenario training.

[0192] For mobile devices, the mobile device acquisition module determines the trajectory prediction results that do not belong to the preset conventional scenarios as data to cover long-tail scenarios, and sends the trajectory prediction results that do not belong to the preset conventional scenarios to the cloud data acquisition module.

[0193] In this embodiment, the purpose of making the vehicle trajectory prediction model cover long-tail scenarios is to enable the vehicle trajectory prediction model to cover (adapt to) rare, sudden, and abnormal long-tail scenarios. For example... Figure 9 As shown, when the mobile device data acquisition module detects that the trajectory prediction results and / or spatiotemporal synchronization information do not belong to the normal scenario, it collects the trajectory prediction results and their spatiotemporal synchronization information as valuable data for improving the performance of the vehicle-side trajectory prediction model.

[0194] The aforementioned conventional scenarios refer to common traffic scenarios that are prevalent in the physical world, such as vehicles driving normally on roads with conventional traffic facilities like traffic lights, traffic signs, lane markings, and shoulders. In contrast, long-tail scenarios are rare, sudden, and unusual traffic scenarios that are seldom or almost impossible to occur in the physical world. Examples include vehicles traveling in the sky, on flowerbeds, or on buildings, or wild animals, buildings, or large floating objects (such as balloons) suddenly appearing on the road. For autonomous vehicles, long-tail scenarios often mean high risks and complex operation. To cope with long-tail scenarios, it is necessary to improve the reasoning ability of the vehicle-side trajectory prediction model when facing them. Correspondingly, the various information corresponding to long-tail scenarios constitutes valuable data for improving the performance of the vehicle-side trajectory prediction model.

[0195] When the trajectory prediction results and / or spatiotemporal synchronization information are detected to be outside the normal scenario, it indicates that the autonomous vehicle is in a rare, sudden, or abnormal long-tail scenario. The inference results and spatiotemporal synchronization information at this time need to be collected as valuable data for improving the performance of the vehicle trajectory prediction model.

[0196] 3. When the training objective is to improve the trajectory prediction model's ability to handle scenarios where the inference performance does not meet predetermined requirements, the inference capability of the trajectory prediction model is tested using a mobile device data acquisition module, including:

[0197] Determine whether the trajectory prediction result of the trajectory prediction model meets the preset expectation;

[0198] The trajectory prediction results are then subjected to a consistency check.

[0199] In this implementation, prioritizing reasoning ability aims to enable the trajectory prediction model to have better reasoning capabilities in scenarios where its own reasoning performance is insufficient.

[0200] Specifically, such as Figure 10 As shown, the following two situations both indicate that the inference performance of the trajectory prediction model is not good enough, and it is necessary to improve the inference ability of the trajectory prediction model when facing the corresponding scenarios:

[0201] (1) Determining whether the trajectory prediction result of the trajectory prediction model meets the preset expectation includes:

[0202] The actual trajectory of the mobile device is obtained based on the sensors;

[0203] Determine whether the subsequent actual trajectory matches the previous trajectory prediction result;

[0204] When there is a mismatch, the trajectory prediction result and / or the corresponding spatiotemporal synchronization information are determined by the mobile device and collected as data for training reasoning ability.

[0205] In this embodiment, the trajectory prediction model infers a certain distance on the map. The trajectory prediction model predicts the target's trajectory for a future time period as a dashed line on the map. After that time period has elapsed, the target's true trajectory, displayed by the fusion results of multiple sensors (visual sensors, LiDAR, millimeter-wave radar, etc.), is a solid line on the map. This solid line is then used as the expected value. Figure 9 It can be seen that the inference results of the trajectory prediction model do not match the expected values ​​well (the overlap between the solid and dashed lines is low). This situation indicates that the inference results of the trajectory prediction model are abnormal. It is possible that the trajectory prediction model's learning effect (inference) on the target behavior in the past period is not good enough. It is necessary to train the vehicle trajectory prediction model to adapt to these target behaviors. Therefore, the inference results and their spatiotemporal synchronization information at this time need to be collected as valuable data for improving the model performance and used for subsequent model training.

[0206] (2) The consistency check of the trajectory prediction results includes:

[0207] Consistency checks are performed between the trajectory prediction results and multiple inference results from the trajectory prediction model based on images and / or laser point clouds and / or millimeter-wave point clouds.

[0208] If the consistency check fails, the trajectory prediction result and / or the corresponding spatiotemporal synchronization information are determined as data for inference ability training by the mobile device.

[0209] In this embodiment, the trajectory prediction results obtained based on different algorithm logics are subjected to consistency checks, and the check results do not reach the predetermined lower limit of consistency.

[0210] The trajectory prediction results based on the map indicate that the obstacle's future trajectory is a left-turn trajectory, while the trajectory prediction results based on images, laser point clouds, and millimeter-wave point clouds all indicate that the obstacle's future trajectory is a right-turn trajectory (or a stationary or straight-going trajectory). A consistency check was performed on the results of these four algorithmic logics. The results showed that the consistency among the four was not good (the map-based result was a left-turn trajectory, while the image, laser point cloud, and millimeter-wave radar-based results were trajectories with other behavioral patterns), failing to meet the predetermined consistency lower limit (e.g., requiring all four to be completely consistent). This situation may be due to the inference results of the map-based trajectory prediction model being inaccurate, or it may be due to the inference results of the trajectory prediction models based on images and / or laser point clouds and / or millimeter-wave point clouds being inaccurate. This indicates that at least one of the four types of trajectory prediction models is not performing well in the current scene, and its inference ability in the current scene needs to be improved. Therefore, the inference results and their spatiotemporal synchronization information at this time need to be collected as valuable data for improving the performance of the vehicle-side trajectory prediction model and used for subsequent model training.

[0211] 4. When the training objective is to maintain the trajectory prediction model's reasoning ability in scenarios where the reasoning performance meets predetermined requirements, a stability test is conducted on the trajectory prediction results using a mobile device data acquisition module, including:

[0212] Determine whether the trajectory prediction result of the trajectory prediction model meets the preset expectation;

[0213] The trajectory prediction results are then subjected to a consistency check.

[0214] In this implementation, prioritizing model stability aims to ensure that the trajectory prediction model continues to perform well in scenarios where its inference capabilities are already excellent.

[0215] Specifically, in the following two cases, it is demonstrated that the inference performance of the vehicle trajectory prediction model is very good, and this inference capability should be maintained:

[0216] (1) Determining whether the trajectory prediction result of the trajectory prediction model meets the preset expectation includes:

[0217] The actual trajectory of the mobile device is obtained based on the sensors;

[0218] Determine whether the subsequent actual trajectory matches the previous trajectory prediction result;

[0219] During matching, the mobile device collects the trajectory prediction results and / or the corresponding spatiotemporal synchronization information as data for training reasoning ability.

[0220] In this embodiment, the trajectory prediction result matches the expected value, and the matching degree reaches a predetermined matching threshold. This situation requires the scene library to include more scenes with very good model inference performance. For example, the trajectory prediction model infers the trajectory of an intersection scene with map information. The predicted trajectory result is that all obstacles turn right and the trajectory line follows the center line of the left and right lanes. According to the high-precision map, the intersection is a right-turn lane. If the high-precision map record is used as the expected value, the matching degree between the trajectory prediction model's inference result and the expected value reaches a good level (e.g., reaching the predetermined matching threshold). This indicates that the trajectory prediction model's inference result for the current scene is very good. The trajectory prediction model needs to maintain this good inference ability. Therefore, it is also necessary to collect the trajectory prediction result and its spatiotemporal synchronization information as valuable data for improving the performance of the vehicle-side trajectory prediction model for subsequent model training.

[0221] (2) The consistency check of the trajectory prediction results includes:

[0222] Consistency checks are performed between the trajectory prediction results and multiple inference results from the trajectory prediction model based on images and / or laser point clouds and / or millimeter-wave point clouds.

[0223] If the consistency check passes, the trajectory prediction result and / or the corresponding spatiotemporal synchronization information are collected by the mobile device as data for training reasoning ability.

[0224] In this embodiment, consistency checks are performed on trajectory prediction results obtained based on different algorithmic logics, and the check results reach a predetermined consistency upper limit. Trajectory prediction results based on maps (e.g., high-precision maps), images, laser point clouds, and millimeter-wave point clouds all indicate that the obstacle's future trajectory is a left-turn trajectory. Consistency checks are performed on the results obtained from these four algorithmic logics, and the trajectory predictions show complete consistency (left-turn trajectory), reaching the predetermined consistency upper limit. This indicates that the trajectory prediction models based on these four types of algorithmic logics have excellent inference performance, and the trajectory prediction needs to maintain this good inference ability. Therefore, the inference results and their spatiotemporal synchronization information at this point need to be collected as valuable data for improving the performance of the trajectory prediction model and used for subsequent model training.

[0225] In summary, the cloud leverages its powerful computing and data storage capabilities to automatically and in real-time complete a series of operations, including generating labeled datasets, training models, and iterating models, accurately extracting data valuable for performance improvement. This mobile device and cloud-based training model fully utilizes the resource advantages of the cloud, improving the efficiency of trajectory prediction model iteration for autonomous vehicles. Furthermore, it allows for targeted model training to achieve specific effects for different training objectives, making the training results controllable.

[0226] Figure 11 This is a schematic diagram of the structure of a vehicle computing system V-150 provided in an embodiment of this application.

[0227] like Figure 11 As shown, the vehicle computing system V-150 includes a processor E-100 coupled to a system bus E-000. The processor E-100 can be any conventional processor, including a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, or a combination thereof. Optionally, the processor E-100 can be a dedicated device such as an Application-Specific Integrated Circuit (ASIC). The processor E-100 can be one or more processors, wherein each processor can include one or more processor cores.

[0228] The system memory E-900 is coupled to the system bus E-000. The data running in the system memory E-900 may include the operating system E-901 and application program E-904 of the vehicle computing system V-150.

[0229] The E-901 operating system consists of a shell (E-902) and a kernel (E-903). The shell (E-902) acts as an interface between the user and the kernel (E-903), forming the outermost layer of the operating system. The shell (E-902) manages the interaction between the user and the operating system, waits for user input, interprets the user's input for the operating system, and processes various operating system outputs.

[0230] The E-903 kernel consists of the parts of the E-901 operating system used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and more.

[0231] Application E-904 includes autonomous driving-related programs E-905, such as programs managing the interaction between the autonomous vehicle 100 and obstacles on the road, programs controlling the driving route or speed of the autonomous driving device, and programs controlling the interaction between the autonomous vehicle 100 and other autonomous driving devices on the road. Application E-904 also exists on the software deployment server system. When application E-904 needs to be executed, the vehicle computing system V-150 can download application E-904 from the software deployment server.

[0232] The system bus E-000 is coupled to the I / O bus E-300 via the bus bridge E-200. The I / O bus E-300 is coupled to the I / O interface E-400. The I / O interface E-400 connects to the USB interface E-500 and various I / O devices for communication, such as input devices, media drives, transceivers, cameras, and sensors. Input devices include, for example, keyboards, mice, and touchscreens; media drives include, for example, CD-ROMs and multimedia interfaces; transceivers are used to send and / or receive radio communication signals; cameras are used to capture still and moving digital video images; and sensors are used to detect the environment surrounding the vehicle computing system V-150 and provide the sensed information to the vehicle computing system V-150.

[0233] The E-800 hard disk drive is coupled to the system bus E-000 via the hard disk drive interface.

[0234] The display adapter E-700 is coupled to the system bus E-000 to drive the display.

[0235] The V-150 vehicle computing system can communicate with the software deployment server via the E-600 network interface. The E-600 network interface is a hardware network interface, such as a network interface card (NIC). The network can be an external network, such as the Internet, an internal network, such as Ethernet or a Virtual Private Network (VPN), or a wireless network, such as a Wi-Fi network or a cellular network.

[0236] The vehicle computing system V-150 may include an on-board execution device, which may include one or more first processors, one or more first memories, and computer instructions stored in the first memory and executable on the first processor. When the first processor executes the computer instructions in the first memory, it performs the functions corresponding to the on-board execution device in the various embodiments provided in this application. The first processor may be configured as one or more general-purpose processors (e.g., CPU, GPU), one or more special-purpose processors (e.g., ASIC), one or more field-programmable gate arrays (FPGA), one or more digital signal processors (DSP), one or more integrated circuits, and / or one or more microcontrollers, etc., in the processor V-151. The first memory may be configured as one or more read-only memories (ROM), one or more random access memories (RAM), one or more flash memories, one or more electrically programmable memories (EPROM), one or more electrically programmable and erasable memories (EEPROM), one or more embedded multimedia cards (eMMC), and / or one or more hard disk drives, etc., in the data storage device V-152. The functions corresponding to the on-board execution device can be implemented as a computer program product; when this computer program product runs on a computer, it implements the functions corresponding to the on-board execution device.

[0237] Figure 12 The image shows a possible example of an autonomous vehicle (using a vehicle as an example of a mobile device) and its onboard execution equipment, such as... Figure 12 As shown, the autonomous vehicle is equipped with an onboard execution device, which includes a first processor, a first memory, and computer instructions stored in the first memory and executable on the first processor. When the first processor executes the computer instructions in the first memory, it performs the following steps: S121, obtaining a trajectory prediction result through inference from the vehicle-side trajectory prediction model; S122, collecting data valuable for improving the performance of the vehicle-side trajectory prediction model based on the trajectory prediction result; S123, iterating the currently used vehicle-side trajectory prediction model into a trained vehicle-side trajectory prediction module.

[0238] Based on the same inventive concept, this application also provides a cloud execution device. Figure 13As shown, the cloud execution device may include one or more second processors, one or more second memories, and computer instructions stored in the second memories and executable on the second processors. When the second processor executes the computer instructions in the second memory, it performs the functions corresponding to the cloud execution device in the various embodiments provided in this application. The second processor may be configured as one or more general-purpose processors (e.g., CPU, GPU), one or more special-purpose processors (e.g., ASIC), one or more field-programmable gate arrays (FPGA), one or more digital signal processors (DSP), one or more integrated circuits, and / or one or more microcontrollers, etc. The second memory may be configured as one or more read-only memories (ROM), one or more random access memories (RAM), one or more flash memories, one or more electrically programmable memories (EPROM), one or more electrically programmable and erasable memories (EEPROM), one or more embedded multimedia cards (eMMC), and / or one or more hard disk drives, etc. The functions corresponding to the cloud execution device can be implemented as a computer program product. When this computer program product runs on a computer, it implements the functions corresponding to the cloud execution device.

[0239] Figure 13 The diagram illustrates a possible example of a cloud-based execution device, including a second processor, a second memory, and computer instructions stored in the second memory and executable on the second processor. When the second processor executes the computer instructions in the second memory, it performs the following steps: S131, collecting data valuable for improving the performance of the vehicle-side model (i.e., the trajectory prediction model); S132, optimizing the data valuable for improving the performance of the vehicle-side model, and training the vehicle-side model using the optimization results; S133, iterating the vehicle-side model currently being used by the onboard execution device to the trained vehicle-side model. Through the above description of the implementation method, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0240] like Figure 14 The diagram shown is a schematic diagram of a cloud execution device provided in an embodiment of the present invention. The system can execute the automated iterative method of trajectory prediction model described in any of the above embodiments and is configured in the terminal.

[0241] This embodiment provides a cloud-based execution device 10, comprising: a cloud data acquisition module 11, a trajectory sample automatic annotation module 12, a trajectory prediction model training module 13, and a trajectory prediction model iteration module 14. The cloud data acquisition module 11 receives prediction scene data determined by the trajectory prediction model of the mobile device and stores the prediction scene data in a cloud scene library. The trajectory sample automatic annotation module 12 includes a data splitting layer, a data driving layer, and a data optimization layer. The data splitting layer is used to split all scene data in the cloud scene library, distinguishing between normal scene data and abnormal scene data. The data driving layer is used to perform perceptual annotation on the normal scene data and the abnormal scene data respectively, obtaining labeled trajectory prediction training samples. The data optimization layer is used to filter the labeled trajectory prediction training samples to obtain smooth trajectory prediction training samples. The trajectory prediction model training module 13 trains the trajectory prediction model in the cloud based on the labeled trajectory prediction training samples. The trajectory prediction model iteration module 14 sends the parameters of the trained trajectory prediction model to the mobile device for updating and iterating the trajectory prediction model of the mobile device.

[0242] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the automated iterative method of trajectory prediction model in any of the above method embodiments;

[0243] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0244] Receive the predicted scenario data determined by the trajectory prediction model of the mobile device;

[0245] The predicted scene data is stored in a cloud scene library, and all scene data in the cloud scene library is split to distinguish between normal scene data and abnormal scene data.

[0246] The normal scene data and the abnormal scene data are respectively labeled to obtain labeled trajectory prediction training samples;

[0247] The trajectory prediction model is trained in the cloud based on the labeled trajectory prediction training samples.

[0248] The parameters of the trained trajectory prediction model are sent to the mobile device to update and iterate the trajectory prediction model of the mobile device.

[0249] like Figure 15The diagram shown is a structural schematic of a mobile device execution device according to an embodiment of the present invention. The system can execute the automated iterative method of trajectory prediction model described in any of the above embodiments and is configured in the terminal.

[0250] The mobile device execution device 20 provided in this embodiment includes: an abnormality triggering module 21, a mobile device acquisition module 22, and a mobile device calculation module 23.

[0251] The abnormality triggering module 21 is used to determine the training objective required for the trajectory prediction model of the mobile device when the preset abnormality triggering conditions are met; the mobile device acquisition module 22 is used to acquire sensor data according to the training objective; the mobile device calculation module 23 is used to input the acquired sensor data into the trajectory prediction model to obtain the trajectory prediction result, construct prediction scene data reflecting the location of the mobile device based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, send the prediction scene data to the cloud, and update and iterate the trajectory prediction model after receiving the parameters of the trajectory prediction model fed back from the cloud.

[0252] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the automated iterative method of trajectory prediction model in any of the above method embodiments;

[0253] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0254] When the preset abnormal triggering conditions are met, the training objective required for the trajectory prediction model of the mobile device is determined.

[0255] According to the training objective, sensor data is collected and input into the trajectory prediction model to obtain trajectory prediction results;

[0256] Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, prediction scene data reflecting the mobile device's location is constructed.

[0257] The predicted scenario data is sent to the cloud, and the trajectory prediction model is updated and iterated after receiving the parameters of the trajectory prediction model from the cloud.

[0258] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the automated iterative method of the trajectory prediction model in any of the above method embodiments.

[0259] This invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute an automated iterative method for a trajectory prediction model applied to a mobile device.

[0260] In some embodiments, the present invention also provides a mobile device, including a body and an electronic device according to any of the foregoing embodiments mounted on the body. The mobile device may be an unmanned vehicle, such as an unmanned sweeper, unmanned floor scrubber, unmanned logistics vehicle, unmanned passenger car, unmanned sanitation vehicle, unmanned minibus / bus, truck, mining truck, etc., or it may be a robot, etc.

[0261] In some embodiments, the present invention also provides a computer program product that, when run on a computer, causes the computer to execute the automated iterative method for trajectory prediction models applied to mobile devices as described in any one of the embodiments of the present invention.

[0262] Figure 16 This is a schematic diagram of the hardware structure of an electronic device for an automated iterative method for trajectory prediction models provided in another embodiment of this application, as shown below. Figure 16 As shown, the device includes:

[0263] One or more processors 1610 and memory 1620, Figure 16 Taking a processor 1610 as an example, the device for an automated iterative method for trajectory prediction models may also include an input device 1630 and an output device 1640.

[0264] The processor 1610, memory 1620, input device 1630, and output device 1640 can be connected via a bus or other means. Figure 16 Taking the example of a connection between China and Israel via a bus.

[0265] The memory 1620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the automated iterative method for trajectory prediction models in the embodiments of this application. The processor 1610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1620, thereby implementing the automated iterative method for trajectory prediction models in the above-described method embodiments.

[0266] The memory 1620 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1620 may optionally include memory remotely located relative to the processor 1610, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0267] Input device 1630 can receive input numerical or character information. Output device 1640 may include display devices such as a display screen.

[0268] The one or more modules are stored in the memory 1620, and when executed by the one or more processors 1610, they execute the automated iterative method of trajectory prediction model in any of the above method embodiments.

[0269] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0270] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0271] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the automated iterative method for trajectory prediction model according to any embodiment of this invention.

[0272] The electronic devices described in this application exist in various forms, including but not limited to:

[0273] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0274] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0275] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0276] (4) Other mobile devices with data processing functions.

[0277] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0278] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0279] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0280] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated iterative method for trajectory prediction models, applied in the cloud, comprising: Receive the predicted scenario data determined by the trajectory prediction model of the mobile device; The predicted scene data is determined by the trajectory prediction results with scene flags output by the trajectory prediction model. The predicted scene data is stored in a cloud scene library, and all scene data in the cloud scene library is split to distinguish between normal scene data and abnormal scene data. The normal scene data and the abnormal scene data are respectively labeled to obtain labeled trajectory prediction training samples; The trajectory prediction model is trained in the cloud based on the labeled trajectory prediction training samples. The parameters of the trained trajectory prediction model are sent to the mobile device to update and iterate the trajectory prediction model of the mobile device. The trajectory prediction result is obtained through the following operations: For each predefined behavior mode, generate the horizontal and vertical position points of the trajectory at each prediction time point within the prediction duration to form the corresponding behavior mode prediction trajectory; the behavior mode includes any one of the following: accelerating straight, normal straight, decelerating straight, turning a small curve, and turning a large curve. Calculate the probability distribution of the predicted trajectory for each behavioral mode, and determine the predicted trajectory with the highest probability distribution as the trajectory prediction result.

2. The method according to claim 1, characterized in that, The process of data splitting for all scene data in the cloud scene library includes: The scene flag is used to split all scene data in the cloud scene library, distinguishing between normal scene data and abnormal scene data.

3. The method according to claim 1, characterized in that, The perception annotation methods include: fusion perception, multi-target tracking, integrated detection and tracking, and pre-trained model optimization, so as to use any one of the perception annotation methods for perception annotation.

4. The method according to claim 1, characterized in that, Training the trajectory prediction model in the cloud based on the labeled trajectory prediction training samples includes: The labeled trajectory prediction training samples are filtered to obtain smooth trajectory prediction training samples for training the trajectory prediction model.

5. The method according to claim 4, characterized in that, The filtering methods include: curve fitting filtering and / or velocity filtering and / or behavior filtering and / or higher-order derivative filtering.

6. The method according to claim 2, characterized in that, The step of storing the predicted scene data in a cloud-based scene library includes: The predicted scene data categories are compared with the cloud scene library. When the cloud scene library lacks the category, or when the amount of data under the category in the cloud scene library does not reach a preset threshold, the predicted scene data is stored in the cloud scene library.

7. The method according to claim 1, characterized in that, The step of sending the parameters of the trained trajectory prediction model to the mobile device to update and iterate the trajectory prediction model of the mobile device includes: The trained trajectory prediction model is tested. If the test results meet the iterative requirements of the training objective, the parameters of the trained trajectory prediction model are sent to the mobile device for iteration.

8. The method according to claim 1, characterized in that, The predicted scenario data determined by the trajectory prediction model of the receiving mobile device includes: The cloud-based data acquisition module receives the encoded predicted scene data sent by the mobile device acquisition module of the mobile device. Use the corresponding scene encoding library in the cloud scene library to determine whether the encoding of the predicted scene data is stored; If the scene encoding library does not store the encoding, the cloud data acquisition module sends an acquisition command to the mobile device acquisition module and receives the predicted scene data uploaded by the mobile device acquisition module based on the acquisition command.

9. The method according to claim 1, characterized in that, The predicted scenario data determined by the trajectory prediction model of the receiving mobile device includes: The cloud data acquisition module receives trajectory prediction results that do not belong to the preset conventional scenarios, which are used for long-tail scenario training, sent by the mobile device acquisition module.

10. An automated iterative method for trajectory prediction models, applied to mobile devices, comprising: When the preset abnormal triggering conditions are met, the training objective required for the trajectory prediction model of the mobile device is determined. According to the training objective, sensor data is collected and input into the trajectory prediction model to obtain trajectory prediction results; Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, prediction scene data reflecting the mobile device's location is constructed. The predicted scenario data is sent to the cloud, and the trajectory prediction model is updated and iterated after receiving the parameters of the trajectory prediction model from the cloud. The step of collecting sensor data and inputting it into the trajectory prediction model according to the training objective to obtain trajectory prediction results includes: For each predefined behavior mode, the horizontal and vertical position points of the trajectory at each prediction time point within the prediction duration are generated to form the corresponding behavior mode prediction trajectory; the behavior mode includes any one of the following: accelerating straight, normal straight, decelerating straight, turning a small curve, and turning a large curve; Calculate the probability distribution of the predicted trajectory for each behavioral mode, and determine the predicted trajectory with the highest probability distribution as the trajectory prediction result.

11. The method according to claim 10, characterized in that, When the training objective is to supplement and enrich the scenarios covered by the trajectory prediction model, the method includes: The mobile device's mobile device acquisition module collects data from various scene types and sends the predicted scene data generated using the data from each scene type to the cloud data acquisition module in the cloud. When the cloud data acquisition module determines that the category corresponding to the predicted scene data is missing in the cloud scene library, or the amount of data under the category corresponding to the predicted scene data in the cloud scene library does not reach a preset threshold, the cloud data acquisition module receives the predicted scene data sent by the mobile device acquisition module and determines it as data to be collected to supplement and enrich the scenes covered by the trajectory prediction model.

12. The method according to claim 11, characterized in that, The scenarios in the predicted scenario data include: mobile device features and environmental features; The categories of elements of the mobile device itself include: basic attributes including weight, geometric information, and performance information; location information including coordinate information and road location; lateral and longitudinal motion state information; and driving task information including perception and recognition, path planning, human-computer interaction, and network communication. The categories of environmental elements include: static environment including obstacles, landscape, traffic facilities, and roads; dynamic environment including dynamic signage facilities and communication information; traffic participants including other mobile devices, pedestrians, and animals; and meteorological information including temperature, humidity, lighting conditions, and weather conditions.

13. The method according to claim 11, characterized in that, When the training objective is to supplement and enrich the scenarios covered by the trajectory prediction model, the method further includes: The mobile device's mobile device acquisition module collects data from various scene types and sends the encoded predicted scene data generated using the data from each scene type to the cloud data acquisition module in the cloud. After receiving the acquisition command from the cloud data acquisition module, the mobile device acquisition module sends the predicted scene data to the cloud data acquisition module.

14. The method according to claim 10, characterized in that, When the training objective is to enable the vehicle trajectory prediction model to cover long-tail scenarios, the method includes: The mobile device acquisition module identifies trajectory prediction results that do not belong to the preset conventional scenarios as data to cover long-tail scenarios, and sends the trajectory prediction results that do not belong to the preset conventional scenarios to the cloud data acquisition module.

15. The method according to claim 10, characterized in that, When the training objective is to improve the reasoning ability of the trajectory prediction model in scenarios where the reasoning performance does not meet predetermined requirements, the reasoning ability of the trajectory prediction model is tested using a mobile device acquisition module, including: Determine whether the trajectory prediction result of the trajectory prediction model meets the preset expectation; The trajectory prediction results are then subjected to a consistency check.

16. The method according to claim 15, characterized in that, The step of determining whether the trajectory prediction result of the trajectory prediction model meets the preset expectation includes: The actual trajectory of the mobile device at a later time step is obtained based on the sensors mounted on the mobile device; Determine whether the actual trajectory at a later time point matches the trajectory prediction result at an earlier time point; When there is a mismatch, the mobile device acquisition module determines the trajectory prediction result at the previous moment as training data for reasoning ability and sends it to the cloud data acquisition module.

17. The method according to claim 15, characterized in that, The consistency check of the trajectory prediction results includes: Consistency checks are performed between the trajectory prediction results and multiple inference results from the trajectory prediction model based on images and / or laser point clouds and / or millimeter-wave point clouds. If the consistency check fails, the mobile device acquisition module will determine the trajectory prediction result as training data for reasoning ability and send it to the cloud data acquisition module.

18. The method according to claim 10, characterized in that, When the training objective is to maintain the trajectory prediction model's reasoning ability in scenarios where the reasoning performance meets predetermined requirements, a stability test is performed on the trajectory prediction results using a mobile device data acquisition module, including: Determine whether the trajectory prediction result of the trajectory prediction model meets the preset expectation; The trajectory prediction results are then subjected to a consistency check.

19. The method according to claim 18, characterized in that, The step of determining whether the trajectory prediction result of the trajectory prediction model meets the preset expectation includes: The actual trajectory of the mobile device at a later time step is obtained based on the sensors mounted on the mobile device; Determine whether the actual trajectory at a later time point matches the trajectory prediction result at an earlier time point; When matching, the mobile device acquisition module determines the trajectory prediction result at the previous moment as the data for inference and sends it to the cloud data acquisition module.

20. The method according to claim 18, characterized in that, The consistency check of the trajectory prediction results includes: Consistency checks are performed between the trajectory prediction results and multiple inference results from the trajectory prediction model based on images and / or laser point clouds and / or millimeter-wave point clouds. If the consistency check passes, the mobile device acquisition module will send the trajectory prediction result, which is determined as data for inference, to the cloud data acquisition module.

21. The method according to claim 10, characterized in that, The abnormal triggering conditions include one or more of the following: The event is triggered when the mobile device detects that a predetermined scene is included in a map collected during the journey. Triggered when the mobile device detects that the status during travel is a predetermined status data; The mobile device triggers the event when it predicts that an obstacle falls under a predetermined condition based on the collected map data.

22. The method according to claim 21, characterized in that, The predetermined scenarios include one or more of the following: intersections, roundabouts, and accident-prone areas; The predetermined state data includes one or more of the following: rapid acceleration, emergency braking, and sharp turning; The obstacles fall under the predetermined conditions, including when the predicted trajectory of the obstacle exceeds the impassable area.

23. The method according to claim 10, characterized in that, The spatiotemporal synchronization information includes one or more of the following: environmental data, map data, mobile device status data, and driver operation data that are synchronized with the trajectory prediction results in time and space.

24. An automated iterative method for trajectory prediction models, comprising: When the mobile device meets the preset abnormal triggering conditions, the training objective required for the trajectory prediction model of the mobile device is determined. According to the training objective, sensor data is collected and input into the trajectory prediction model to obtain the trajectory prediction result. Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, prediction scene data reflecting the mobile device's location is constructed. The step of collecting sensor data and inputting it into the trajectory prediction model to obtain trajectory prediction results, according to the training objective, includes: For each predefined behavior mode, generate the horizontal and vertical position points of the trajectory at each prediction time point within the prediction duration to form the corresponding behavior mode prediction trajectory; the behavior mode includes any one of the following: accelerating straight, normal straight, decelerating straight, turning a small curve, and turning a large curve. Calculate the probability distribution of the predicted trajectory for each behavioral mode, and determine the predicted trajectory with the highest probability distribution as the trajectory prediction result; Send the predicted scenario data to the cloud; The cloud receives the predicted scene data determined by the trajectory prediction model of the mobile device; The cloud stores the predicted scene data in the cloud scene library and performs data splitting on all scene data in the cloud scene library to distinguish between normal scene data and abnormal scene data; The cloud performs perception annotation on the normal scene data and the abnormal scene data respectively to obtain labeled trajectory prediction training samples; The trajectory prediction model is trained in the cloud based on the labeled trajectory prediction training samples; The cloud sends the parameters of the trained trajectory prediction model to the mobile device; After receiving the parameters of the trajectory prediction model from the cloud, the mobile device updates and iterates the trajectory prediction model.

25. A cloud execution device, comprising: The cloud data acquisition module is used to receive the predicted scene data determined by the trajectory prediction model of the mobile device and store the predicted scene data in the cloud scene library; the predicted scene data is determined by the trajectory prediction result with scene flags output by the trajectory prediction model. The trajectory sample automatic annotation module includes a data diversion layer, a data-driven layer, and a data optimization layer. The data diversion layer is used to: divert all scene data in the cloud scene library and distinguish between normal scene data and abnormal scene data. The data-driven layer is used to: perform perceptual annotation on the normal scene data and the abnormal scene data respectively, to obtain labeled trajectory prediction training samples; The data optimization layer is used to: filter the labeled trajectory prediction training samples to obtain smooth trajectory prediction training samples; A trajectory prediction model training module is used to train the trajectory prediction model in the cloud based on the labeled trajectory prediction training samples. The trajectory prediction model iteration module is used to send the parameters of the trained trajectory prediction model to the mobile device to update and iterate the trajectory prediction model of the mobile device. The trajectory prediction result is obtained through the following operations: For each predefined behavior mode, generate the horizontal and vertical position points of the trajectory at each prediction time point within the prediction duration to form the corresponding behavior mode prediction trajectory; the behavior mode includes any one of the following: accelerating straight, normal straight, decelerating straight, turning a small curve, and turning a large curve. Calculate the probability distribution of the predicted trajectory for each behavioral mode, and determine the predicted trajectory with the highest probability distribution as the trajectory prediction result.

26. A mobile device execution device, comprising: The anomaly triggering module is used to determine the training objective required for the trajectory prediction model of the mobile device when preset anomaly triggering conditions are met. A mobile device acquisition module is used to acquire sensor data according to the training objective. The mobile device computing module is used to input the collected sensor data into the trajectory prediction model to obtain the trajectory prediction result. Based on the trajectory prediction result and the spatiotemporal synchronization information related to the trajectory prediction result, it constructs the prediction scene data reflecting the location of the mobile device, sends the prediction scene data to the cloud, and updates and iterates the trajectory prediction model after receiving the parameters of the trajectory prediction model fed back from the cloud. The step of inputting the collected sensor data into the trajectory prediction model to obtain the trajectory prediction result includes: For each predefined behavior mode, generate the horizontal and vertical position points of the trajectory at each prediction time point within the prediction duration to form the corresponding behavior mode prediction trajectory; the behavior mode includes any one of the following: accelerating straight, normal straight, decelerating straight, turning a small curve, and turning a large curve. Calculate the probability distribution of the predicted trajectory for each behavioral mode, and determine the predicted trajectory with the highest probability distribution as the trajectory prediction result.

27. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-9.

28. A mobile device comprising a body and an electronic device according to claim 27 mounted on the body.

29. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-9.

30. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 10-23.

31. A mobile device comprising a body and an electronic device according to claim 30 mounted on the body.

32. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 10-23.

Images