Automatic driving track reconstruction method and reconstruction device based on large language model

Abstract

The invention relates to the technical field of automatic driving, in particular to an automatic driving track reconstruction method and device based on a large language model. The automatic driving trajectory reconstruction method comprises the following steps: acquiring real trajectory data of a current vehicle; and inputting the real trajectory data into the trained trajectory reconstruction model to obtain reconstructed trajectory data of the current vehicle. Wherein the trajectory reconstruction model is a large-scale language model, and the large-scale language model is trained, so that the large-scale language model can still provide excellent prediction capability under the condition that the data quality is not high. By adopting the trained trajectory reconstruction model provided by the invention, the missing points in the trajectory can be identified and filled under the condition that the acquired data is insufficient or the data part is indeed, so that the trajectory reconstruction in the automatic driving is completed, and the stability and the safety of the automatic driving are guaranteed.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving trajectory reconstruction method and reconstruction device based on a large language model. Background Art

[0002] The processing of autonomous driving trajectory data is a key step in ensuring the safety and reliability of the autonomous driving system. However, natural driving trajectory data in autonomous driving also faces a series of challenges, which mainly come from the uncertainty and variability in the data collection process.

[0003] The data collection quality problem of autonomous driving not only limits the effective application of trajectory data in autonomous driving testing, but may also cause the autonomous driving system to be unable to accurately predict and respond to complex driving environment changes in actual applications, thereby affecting overall safety. For example, based on incomplete or inaccurate data collected, the autonomous driving system may not be able to correctly judge the road conditions or the driving status of the vehicle, which may cause safety hazards in high-density traffic environments or severe weather conditions. Therefore, how to effectively process and reconstruct these trajectory data is the key to ensuring the stability and reliability of the autonomous driving system. Summary of the invention

[0004] The present application provides an autonomous driving trajectory reconstruction method and reconstruction device based on a large language model, so as to achieve stable reconstruction of the autonomous driving trajectory when the quality of data collected during autonomous driving is not high, so as to ensure the safety and reliability of autonomous driving.

[0005] In a first aspect, the present application provides an autonomous driving trajectory reconstruction method based on a large language model, the method comprising:

[0006] Get the real trajectory data of the current vehicle;

[0007] Inputting the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle;

[0008] Wherein, the trajectory reconstruction model is a large language model.

[0009] In a possible design, before the real trajectory data is input into the trained trajectory reconstruction model, the following steps are included:

[0010] Obtain historical autonomous driving trajectory data;

[0011] Preprocessing the historical autonomous driving trajectory data to construct a training set;

[0012] The training set is used to train an initial large language model to obtain a trained trajectory reconstruction model.

[0013] In one possible design, the real trajectory data includes position information, speed information, acceleration information, and heading angle information of the autonomous driving vehicle.

[0014] In a possible design, preprocessing the historical autonomous driving trajectory data to construct a training set includes:

[0015] Extracting the vehicle's position information, speed information, and heading angle information from the historical autonomous driving trajectory data;

[0016] Adding a corresponding noise signal combination to the position information, speed information and heading angle information of the vehicle;

[0017] The position information, speed information and heading angle information of the vehicle to which the noise signal combination is added are converted into a preset data format to obtain the training set.

[0018] In a possible design, when training the initial large language model, the loss function used is:

[0019]

[0020] Among them, L i =Loss(f(X i ′;θ)-Y i ′);

[0021] Among them, A l and B l is the low-rank matrix in the LoRA layer, L i is the loss of a single training sample, X i ′ is the noisy input sample data, f(X i ′; θ) is the predicted data output by the large language model, Y i ′ is the actual driving trajectory corresponding to the sample data in the training set.

[0022] In one possible design, it also includes:

[0023] Preprocessing the historical autonomous driving trajectory data to construct a test set;

[0024] The trained trajectory reconstruction model is tested using the test set, and the performance of the trajectory reconstruction model is evaluated based on the test results; wherein the performance evaluation indicators include reconstruction error, prediction accuracy of the model, and performance of the model in a specific scenario.

[0025] In a possible design, the trajectory reconstruction method further includes:

[0026] When the performance of the trajectory reconstruction model after evaluation is not satisfactory to the preset requirements, the trajectory reconstruction model is trained again after adjusting the hyperparameters of the LoRA layer;

[0027] Alternatively, a defect area whose performance of the trajectory reconstruction model does not meet the preset requirements is obtained, a training sample corresponding to the defect area is obtained, and the trajectory reconstruction model is trained again using the training sample corresponding to the defect area until the performance of the trained trajectory reconstruction model meets the preset requirements.

[0028] In a second aspect, the present application also provides an automatic driving trajectory reconstruction device based on a large language model, comprising:

[0029] A data acquisition unit, used to acquire the real trajectory data of the current vehicle;

[0030] The trajectory reconstruction unit is used to input the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle; wherein the trajectory reconstruction model is a large language model.

[0031] In a third aspect, the present application also provides a computer-readable storage medium storing a computer program, which, when running on a processor, executes any of the above-mentioned methods for reconstructing autonomous driving trajectories based on a large language model.

[0032] In a fourth aspect, the present application also provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the autonomous driving trajectory reconstruction method based on a large language model as described in any one of the above items.

[0033] In a possible implementation of the first aspect,

[0034] According to the method for reconstructing the trajectory of an autonomous driving based on a large language model provided in the first aspect above, it includes: obtaining the real trajectory data of the current vehicle; inputting the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle. Among them, the trajectory reconstruction model of the present application is a large language model. By training the large language model, the large language model can still provide excellent prediction capabilities when the data quality is not high. The trained trajectory reconstruction model provided by the present application can identify and fill in the missing points in the trajectory when the collected data is insufficient or the data is partially accurate, so as to complete the trajectory reconstruction in the autonomous driving, thereby ensuring the stability and safety of the autonomous driving.

[0035] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of an autonomous driving trajectory reconstruction method based on a large language model provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the large language model training and trajectory reconstruction process provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a reconstructed trajectory obtained by a trajectory reconstruction method is provided for an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of an autonomous driving trajectory reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0041] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0042] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0043] The autonomous driving trajectory is the prerequisite for ensuring the safety and stability of autonomous driving. During the reconstruction of the autonomous driving trajectory, the continuity and accuracy of data collection are the key influencing factors of trajectory reconstruction.

[0044] In a specific application scenario, the accuracy limitation of the sampling device is a major influencing factor. The resolution, sampling frequency and installation location of the sensor may affect the accuracy of the collected data. For example, low-resolution GPS (Global Positioning System) data may lead to inaccurate vehicle positioning, while high-frequency vibration may affect the data quality of the Inertial Measurement Unit (IMU). In addition, data synchronization issues between different devices may also lead to data inconsistency.

[0045] In a specific application scenario, environmental factors also have a great impact on the accuracy of data collection. Factors such as weather changes, road conditions, and reflection interference from surrounding buildings may affect sensor data. For example, rainy and snowy weather may affect the line of sight and accuracy of cameras and lidars, and strong light or shadow changes may interfere with the image acquisition quality of cameras. These factors may increase the noise in the data, making the trajectory records discontinuous or inaccurate.

[0046] In addition, in a specific application scenario, data loss also has a great impact on trajectory reconstruction. In actual driving environments, sensor data may have breakpoints due to signal loss or equipment failure, resulting in incomplete trajectory data. Data loss not only makes trajectory reconstruction more complicated, but may also have a negative impact on the prediction and decision-making of the autonomous driving system, thereby affecting the safety of autonomous driving.

[0047] The above data quality issues not only limit the effective application of trajectory data in autonomous driving tests, but may also cause the autonomous driving system to be unable to accurately predict and respond to complex driving environment changes in actual applications, thus affecting overall safety. For example, based on incomplete or inaccurate data, the autonomous driving system may not be able to correctly judge the road conditions or the driving status of the vehicle, which may cause safety hazards in high-density traffic environments or severe weather conditions. Therefore, how to effectively process and reconstruct these trajectory data is the key to ensuring the stability and reliability of the autonomous driving system.

[0048] Autonomous driving trajectory reconstruction is a key task in autonomous driving systems. Its goal is to extract and reconstruct accurate vehicle motion trajectories from raw sensor data to support the learning and testing of autonomous driving algorithms. There are three main trajectory reconstruction methods in related technologies: methods based on physical models, Kalman filtering methods, and methods based on data-driven machine learning models.

[0049] Among them, the physical model-based methods use the vehicle's dynamic model and kinematic equations to reconstruct the vehicle trajectory based on physical principles. This type of method requires modeling the vehicle's dynamic characteristics, mechanical properties, driving intentions, etc., and relies on detailed parameterized descriptions. However, in practical applications, these methods often rely on extensive parameter assumptions and simplified models, which may lead to insufficient accuracy and applicability in complex or unknown environments.

[0050] Among them, the Kalman filter method is an estimation technique widely used in real-time data processing and trajectory reconstruction. This method can provide relatively accurate trajectory prediction in a noisy environment. However, the Kalman filter's assumption of a linear system limits its performance in dealing with nonlinear dynamic systems and environments with complex maneuvers. When there are unpredictable maneuvers or environmental disturbances in the vehicle's motion, the effect of the Kalman filter decreases significantly.

[0051] Among them, methods based on data-driven machine learning models include enhanced regression trees, random forests, and neural networks, which perform well in processing complex nonlinear data. This method can learn and identify trajectory patterns from a large amount of training data, thereby achieving remarkable results in trajectory reconstruction tasks. However, a major limitation of this type of method is the need for a large amount of labeled data, especially training data with accurate ground truth. In autonomous driving applications, labeled data is usually difficult to obtain, which limits the application of the model in real-world scenarios.

[0052] It can be seen that there are some key problems in the application of the above-mentioned related technologies in autonomous driving trajectory reconstruction. Although the physical model-based method has a solid foundation in theory, it often relies on too many simplified assumptions and parameter estimates in practical applications, resulting in insufficient accuracy and adaptability when dealing with complex real-world scenarios. The Kalman filter method performs well in dealing with linear systems, but has poor adaptability to nonlinear and non-steady-state systems, especially in the case of drastic environmental changes or large interference, and cannot provide reliable trajectory reconstruction results. Although data-driven machine learning models can achieve high reconstruction accuracy under data-rich conditions, it is usually difficult to obtain a large amount of high-quality training data with true values ​​in reality. This lack of data limits the widespread application of these models.

[0053] The present application aims to solve the above-mentioned problems existing in the reconstruction of autonomous driving trajectories, especially the problems of processing discontinuous, inaccurate or missing data. To this end, the present application introduces a large language model (LLM) to process and reconstruct autonomous driving trajectory data using its powerful pattern recognition and prediction capabilities. LLM has the ability to learn effectively with less training data, and can identify and fill in missing points in the trajectory, correct abnormal data, and provide high-quality trajectory reconstruction results when data is insufficient. Through the application of LLM, the present application provides a method for stably and reliably reconstructing autonomous driving trajectories even when data is incomplete or of low quality, thereby improving the overall safety and reliability of the autonomous driving system.

[0054] Figure 1 For a flow chart of the autonomous driving trajectory reconstruction method based on a large language model provided in the embodiment of the present application, please refer to Figure 1 As shown, the trajectory reconstruction method includes:

[0055] S101, obtaining the actual trajectory data of the current vehicle.

[0056] In some embodiments, the actual trajectory data includes position information, speed information, acceleration information, and heading angle information of the autonomous driving vehicle.

[0057] For example, when obtaining the real trajectory data of the current vehicle, the real trajectory data can be extracted from various sensors and data recording systems of the autonomous driving vehicle. For example, the GPS coordinates of the autonomous driving vehicle are obtained from the GPS positioning module of the autonomous driving vehicle, and the speed, acceleration, heading angle and other information of the vehicle are collected from the inertial sensor IMU.

[0058] In some embodiments, after the raw data is collected, it is necessary to pre-process the collected raw data to convert the raw data into real trajectory data in a preset format. For example, the collected raw data can be parsed and data extracted first, and after the required data is extracted, the raw data can be converted into a format suitable for large model input in the data pre-processing stage.

[0059] In a specific embodiment, the necessary parameters are first extracted from the original data, and then organized into a natural language format that meets the model requirements, such as "determine a continuous state driving trajectory {input_str} based on the driving parameter data sequence (time, x-coordinate, y-coordinate, heading angle, x-axis speed, y-axis speed)", where `{input_str}` represents a specific data sequence. In order to improve processing efficiency, the data is processed in blocks, each block of data contains a fixed number of state parameters, and the processed data is stored in a standardized format for use in subsequent steps. Through the above data processing steps, the original and complex trajectory data can be converted into a format that can be understood by a large language model, laying the foundation for trajectory reconstruction.

[0060] S102, inputting the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle; wherein the trajectory reconstruction model is a large language model.

[0061] In some embodiments, the real trajectory data in a preset format processed in step S101 is input into the trained trajectory reconstruction model, and the trajectory reconstruction capability of the trajectory reconstruction model is used to output the reconstructed trajectory data of the current vehicle.

[0062] In this embodiment, the input layer of the large language model receives time series data, which is usually presented in a format containing parameters such as timestamp, position coordinates, speed, and heading angle, that is, the real trajectory data is obtained after preprocessing. The trained model uses an improved attention mechanism and a recurrent neural network structure to process these time series data and generate trajectory predictions. During the reconstruction process, the model processes the input real trajectory data to restore and reconstruct the complete trajectory, and outputs the prediction result (i.e., the reconstructed trajectory data). The prediction result is compared with the actual trajectory data to generate the reconstructed trajectory data.

[0063] To improve accuracy, large language models combine the loss function optimization parameters in the forward and back propagation processes. The loss function is used to quantify the gap between the predicted trajectory and the true trajectory, and the model parameters are optimized by minimizing the loss function. The final output of the model is the optimized trajectory reconstruction result for further analysis and application.

[0064] Figure 2 For a schematic diagram of the large language model training and trajectory reconstruction process provided in the embodiment of this application, please refer to Figure 2 As shown, in some embodiments, before inputting the real trajectory data into the trained trajectory reconstruction model, a large number of samples need to be used to train the initial large language model. The training process method specifically includes: obtaining historical autonomous driving trajectory data; preprocessing the historical autonomous driving trajectory data to construct a training set; and using the training set to train the initial large language model to obtain a trained trajectory reconstruction model.

[0065] It is understandable that when constructing a training set, sample data also needs to be preprocessed to facilitate recognition and data processing by a large language model, where the preprocessing method for training sample data is similar or the same as the processing direction for the real trajectory data mentioned above. That is, preprocessing the historical autonomous driving trajectory data includes: extracting necessary parameters from the historical autonomous driving trajectory data, and organizing these parameters into a string format that meets the model requirements.

[0066] Specifically, in this embodiment, the Argoverse2 dataset is used to construct a dataset for training the model. The Argoverse2 dataset provides high-quality autonomous driving trajectory data, including vehicle position, speed, acceleration and other information. The Argoverse2 dataset autonomous driving vehicle-related data is divided into a training set, a validation set and a test set at 80%, 10% and 10%.

[0067] In some embodiments, other autonomous driving datasets, such as KITTI or Waymo OpenDataset, can be used to enrich the training and evaluation data sources and improve the generalization ability of the model. Sensor fusion data, including lidar and camera data, can also be introduced to improve the accuracy of trajectory reconstruction.

[0068] In some embodiments, historical autonomous driving trajectory data is preprocessed to construct a training set, including: extracting the vehicle's position information, speed information, and heading angle information from the historical autonomous driving trajectory data; adding a corresponding noise signal combination to the vehicle's position information, speed information, and heading angle information; converting the vehicle's position information, speed information, and heading angle information to which the noise signal combination was added into a preset data format to obtain a training set.

[0069] In this embodiment, in order to construct a training set for training LLM, key time series data are first extracted from the original data. These time series data include timestamps, coordinate positions, heading angles, speeds, etc. Then, by adding noise to these data, common measurement errors and sensor inaccuracies in the real world can be simulated. The process of adding noise includes introducing random perturbations to the position, velocity, and heading angle data to reflect the data uncertainty in the actual environment. The noise creation scheme can combine different types of noise models to better simulate the data uncertainty in the real environment. For position, velocity, and heading angle data, different types of noise combinations are used. Note that the following parameter values ​​are for vehicle trajectory data on urban roads.

[0070] For example, for position data, Gaussian noise and uniform noise are combined, and each coordinate (x, y) of the position will be affected by these two noises. The following formula takes the x coordinate as an example, and the y coordinate is the same:

[0071] Position x ′=Position x +σ x ·Z+(a x +(b x -a x )·U) (1)

[0072] Among them, Position x is the original position coordinate on the x-axis, σ x σ x is the standard deviation of Gaussian noise on the x-axis, σ x You can take 0.1 meters. x and b x are the minimum and maximum values ​​of uniform noise in the x-coordinate, a x and b xcan be -0.2 and 0.2 meters respectively, while Z and U are sampled from the standard normal distribution and uniform distribution respectively.

[0073] For example, for speed data, Gaussian noise and dynamic noise are combined. The intensity of the noise increases with the increase of speed. The following formula takes the speed on the x-axis as an example, and the speed on the y-axis is the same:

[0074]

[0075] Among them, v x is the speed on the x-axis, σ v is the standard deviation of Gaussian noise, σ v 0.3 m / s can be taken as v0 as the reference value of speed, and v0 can be taken as 10 m / s.

[0076] For example, for heading angle data, Gaussian noise and uniform noise are combined, which can be specifically expressed as:

[0077] Heading′=Heading+σ h ·Z+(a h +(b h -a h )·U) (3)

[0078] Among them, Heading is the original heading angle, σ h As the standard deviation of Gaussian noise of heading angle, σ h You can take 0.05 radians, a h With b h Take -0.02 radians and 0.02 radians respectively.

[0079] In some embodiments, other noise processing methods, such as Poisson noise or Laplace noise, can be used to simulate different types of data uncertainty. Data enhancement techniques, such as trajectory smoothing or random cropping, can also be used to expand the diversity of the training data set.

[0080] Through the above preprocessing steps, the data will be converted into an input format acceptable to the model, for example, the data sequence is formatted into the form of "using these driving parameter data sequences (time, x-coordinate, y-coordinate, heading angle, x-axis speed, y-axis speed) to determine a continuous driving trajectory {input_str}", where `{input_str}` represents a specific data sequence for LLM to fine-tune training. In this way, not only is training data with practical application background provided to LLM, but its ability to process trajectory data in real scenarios is also enhanced.

[0081] This embodiment uses noisy training data generated by the Argoverse2 dataset. These data include position coordinates, speed, heading angle, etc., which have been properly preprocessed and injected with noise, and meet the model input format requirements. Next, the large language model is trained. Specifically, the LoRA (Low-Rank Adaptation) method is used to fine-tune the LLM. The training parameters are only one ten-thousandth of the overall parameters, the GPU (graphics processing unit) memory usage is reduced by 2 / 3, and no additional reasoning time is introduced.

[0082] During the LoRA fine-tuning phase, some layers of the pre-trained large language model can be adjusted. Specifically, this embodiment replaces some parameters of the original model with low-rank matrices, which are optimized through training. This process involves inserting LoRA layers in the attention layer or feedforward layer of the Transformer model, which reduce the amount of training parameters of the model through low-rank decomposition, thereby improving computational efficiency. During the training process, noisy sample data is used as input, and the model parameters are optimized by minimizing the loss function to adapt to the specific autonomous driving trajectory reconstruction task.

[0083] In some embodiments, the following formula is used to calculate the loss of each training sample:

[0084] L i =Loss(f(X i ′;θ)-Y i ′) (4)

[0085] Among them, L i is the loss of a single training sample, X i ′ is the noisy input sample data, f(X i ′; θ) is the predicted data output by the large language model, Y i ′ is the actual driving trajectory corresponding to the sample data in the training set.

[0086] When training the initial large language model, the calculation method of the loss function Loss in LoRA is similar to the standard training method. During the optimization process, the goal of this embodiment is to minimize the total loss of all samples. The total loss calculation formula is:

[0087]

[0088] Among them, A l and B l is the low-rank matrix in the LoRA layer, L i is the loss of a single training sample.

[0089] During model training, appropriate learning rates and optimizer configurations are used to ensure that the model can converge quickly and achieve optimal performance. Next, a pre-built validation set is used to evaluate the model's reconstruction capabilities to ensure its accuracy and stability when processing real driving trajectory data. In this way, LoRA fine-tuning not only enhances the model's adaptability to trajectory reconstruction tasks, but also significantly improves its robustness in processing complex data.

[0090] In some embodiments, the historical autonomous driving trajectory data is preprocessed to construct a test set; the trained trajectory reconstruction model is tested using the test set, and the performance of the trajectory reconstruction model is evaluated based on the test results. Among them, the performance evaluation indicators in this embodiment include reconstruction error, prediction accuracy of the model, and performance of the model in a specific scenario. The reconstruction error represents the distance error between the real position and the predicted position reconstructed by the model.

[0091] In some embodiments, when the performance of the trajectory reconstruction model after evaluation is not satisfactory to the preset requirements, the trajectory reconstruction model is trained again after adjusting the hyperparameters of the LoRA layer. Alternatively, a defect area whose performance of the trajectory reconstruction model does not meet the preset requirements is obtained, a training sample corresponding to the defect area is obtained, and the trajectory reconstruction model is trained again using the training sample corresponding to the defect area until the performance of the trained trajectory reconstruction model meets the preset requirements.

[0092] In some embodiments, when the trajectory generated by using the trained trajectory reconstruction model does not meet rationality and data consistency, the model is adjusted before continuing training.

[0093] It is understandable that the large language model can also be further adjusted based on the results of the verification and evaluation feedback. If it is found in the verification and evaluation module that the model's ability to reconstruct real data is insufficient, further adjustments are required. First, performance feedback is obtained from the verification and evaluation results, including reconstruction error, model prediction accuracy, and performance in specific scenarios. This feedback information will help identify weaknesses and deficiencies in the model. Based on this feedback, the training data set can be re-examined, and new data samples may need to be added or existing data may need to be reprocessed to enhance its representativeness.

[0094] In addition, the learning effect of the model can be optimized by adjusting the hyperparameters of the LoRA layer, such as the learning rate, noise intensity, etc. During the re-fine-tuning process, the difficult points in the test set or the areas where the model performs poorly will be trained to improve the overall performance of the model. Specifically, for the areas where the model performs poorly, more similar trajectory data samples are collected. The newly added data should contain trajectory information in these specific scenarios to enhance the model's adaptability to these situations; reprocess the data to improve the quality and diversity of the data. For example, increase the range of data variation through data enhancement technology, or reduce the noise in the data through denoising technology. Apply the adjusted data and hyperparameters to the model for retraining, and re-fine-tune the model using the updated data set and the adjusted LoRA layer settings to optimize the model's ability to reconstruct the real trajectory data. Perform multiple rounds of re-fine-tuning, and gradually improve the performance of the model through continuous feedback and adjustment until the expected reconstruction effect is achieved. Through the above scheme, the trajectory reconstruction ability of the large language model can be continuously optimized based on the feedback from the verification and evaluation module, so that it can better adapt to actual application scenarios.

[0095] In some embodiments, an adaptive fine-tuning method based on an attention mechanism may be used to better capture the characteristics of trajectory data; a domain-adaptive fine-tuning strategy may be applied to adjust specific parameters according to specific scenarios.

[0096] Figure 3 The schematic diagram of the reconstructed trajectory obtained by the trajectory reconstruction method provided in the embodiment of the present application is shown in FIG. Figure 3As shown, in the process of verifying and evaluating the performance of the model, since the real trajectory results are unavailable, the rationality assessment is combined with the data distribution consistency assessment to evaluate the generated trajectory. The rationality assessment mainly verifies the rationality of the generated trajectory in combination with the physical constraints. The physical constraints include ensuring the speed and acceleration of the generated trajectory, and the physical constraints are physically feasible. For example, by checking whether the speed of the trajectory is within the maximum speed range of the vehicle, the implementation scheme of the data distribution consistency assessment is to compare the trajectory generated by the trained large language model with the trajectory distribution existing in the training data. The statistical characteristics of the real trajectory, such as the distribution of position coordinates, speed, and heading angle, are extracted from the Argoverse2 data set. Through these characteristics, a probability distribution model of the trajectory data can be established to capture the main features and laws of the data. This can be achieved by calculating the distance between the generated trajectory features and the real data features. Using the Kolmogorov-Smirnov test, the consistency of the distribution of the reconstructed trajectory and the real trajectory can be effectively detected, so as to effectively evaluate and adjust the model. In another embodiment, the Wasserstein distance or Jensen-Shannon divergence can also be used to evaluate the difference between the generated trajectory and the real data distribution. The Cumulative Distribution Function (CDF) is calculated for each feature data of the real trajectory and the generated trajectory (such as position x and y coordinates, speed, heading angle, etc.). The maximum difference between the CDF of the real data and the generated data is calculated, which is called the KS statistic. According to the results of the KS test, it can be evaluated whether the distribution of the generated trajectory in features such as position, speed and heading angle is consistent with the real trajectory. If the KS statistic value is small and not significant at the allowed significance level (such as 0.05), it means that the distribution of the generated trajectory is similar to the real trajectory, which also means that the trajectory reconstruction effect of the trained model is good. If the KS statistic value is large, it is fed back to the large language model fine-tuning module to adjust the model again. The advantage of this evaluation module is that it does not rely on the specific value of the real trajectory, but indirectly evaluates the quality of the generated trajectory by comparing the data distribution, ensuring that the generated trajectory is as consistent as possible with the actual data in terms of statistical characteristics.

[0097] From the above description, it can be seen that the trajectory reconstruction method provided by this application fully utilizes the advantages of LLM in processing nonlinear time series data, and realizes efficient completion and reconstruction of data through pattern recognition and learning of trajectory data. The pre-training stage of LLM enables it to have rich prior knowledge, while the fine-tuning stage adapts it to the data characteristics of specific tasks, so that it can still provide excellent prediction capabilities when data is limited. Compared with traditional methods, this application can provide more consistent and reliable trajectory reconstruction results in a wider range of scenarios, providing important guarantees for the safety and stability of autonomous driving systems.

[0098] The following provides an autonomous driving trajectory reconstruction device based on a large language model. The trajectory reconstruction device provided below can refer to the trajectory reconstruction method provided above.

[0099] Figure 4 For a schematic diagram of the structure of the automatic driving trajectory reconstruction device provided in the embodiment of the present application, please refer to Figure 4 As shown, the trajectory reconstruction device includes:

[0100] The data acquisition unit 401 is used to acquire the real trajectory data of the current vehicle.

[0101] The trajectory reconstruction unit 402 is used to input the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle; wherein the trajectory reconstruction model is a large language model.

[0102] In some embodiments, the trajectory reconstruction device further includes a model training unit 403, which is used to obtain historical autonomous driving trajectory data; preprocess the historical autonomous driving trajectory data to construct a training set; and use the training set to train the initial large language model to obtain a trained trajectory reconstruction model. For the specific training method, please refer to the above embodiment and will not be repeated here.

[0103] In some embodiments, the trajectory reconstruction device further includes a verification and evaluation unit 404, which is used to test the trained trajectory reconstruction model using a test set, and to perform a performance evaluation on the trajectory reconstruction model according to the test result evaluation, specifically including evaluating, verifying and providing feedback on the generated trajectory. For the specific testing and evaluation method, please refer to the above embodiment, which will not be described in detail here.

[0104] In some embodiments, the present application also provides a computer-readable storage medium storing a computer program, which, when running on a processor, executes the autonomous driving trajectory reconstruction method based on a large language model as described above.

[0105] In some embodiments, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, enables the computer to execute the automatic driving trajectory reconstruction method based on the large language model as described above.

[0106] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0107] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0112] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a memory (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store program code.

[0113] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for reconstructing autonomous driving trajectories based on a large language model, characterized in that: include: Get the real trajectory data of the current vehicle; Inputting the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle; Wherein, the trajectory reconstruction model is a large language model.

2. The method for reconstructing the trajectory of an autonomous driving system based on a large language model according to claim 1, characterized in that: Before inputting the real trajectory data into the trained trajectory reconstruction model, the following steps are included: Obtain historical autonomous driving trajectory data; Preprocessing the historical autonomous driving trajectory data to construct a training set; The training set is used to train an initial large language model to obtain a trained trajectory reconstruction model.

3. The method for reconstructing the automatic driving trajectory based on a large language model according to claim 1 or 2, characterized in that: The real trajectory data includes the position information, speed information, acceleration information and heading angle information of the autonomous driving vehicle.

4. The method for reconstructing the trajectory of an autonomous driving system based on a large language model according to claim 3, characterized in that: The preprocessing of the historical autonomous driving trajectory data to construct a training set includes: Extracting the vehicle's position information, speed information, and heading angle information from the historical autonomous driving trajectory data; Adding a corresponding noise signal combination to the position information, speed information and heading angle information of the vehicle; The position information, speed information and heading angle information of the vehicle to which the noise signal combination is added are converted into a preset data format to obtain the training set.

5. The method for reconstructing the trajectory of an autonomous driving system based on a large language model according to claim 4, characterized in that: When training the initial large language model, the loss function used is: among them,L i =Loss(f(X i ′;θ)-Y i ′); Among them, A l and B l is the low-rank matrix in the LoRA layer, L i is the loss of a single training sample, X i ′ is the noisy input sample data, f(X i ′; θ) is the predicted data output by the large language model, Y i ′ is the actual driving trajectory corresponding to the sample data in the training set.

6. The method for reconstructing automatic driving trajectories based on a large language model according to claim 4, characterized in that: Also includes: Preprocessing the historical autonomous driving trajectory data to construct a test set; The trained trajectory reconstruction model is tested using the test set, and the performance of the trajectory reconstruction model is evaluated based on the test results; wherein the performance evaluation indicators include reconstruction error, prediction accuracy of the model, and performance of the model in a specific scenario.

7. The method for reconstructing automatic driving trajectories based on a large language model according to claim 6, characterized in that: Also includes: When the performance of the trajectory reconstruction model after evaluation is not satisfactory to the preset requirements, the trajectory reconstruction model is trained again after adjusting the hyperparameters of the LoRA layer; Alternatively, a defect area whose performance of the trajectory reconstruction model does not meet the preset requirements is obtained, a training sample corresponding to the defect area is obtained, and the trajectory reconstruction model is trained again using the training sample corresponding to the defect area until the performance of the trained trajectory reconstruction model meets the preset requirements.

8. An automatic driving trajectory reconstruction device based on a large language model, characterized in that: include: A data acquisition unit, used to acquire the real trajectory data of the current vehicle; The trajectory reconstruction unit is used to input the real trajectory data into the trained trajectory reconstruction model to obtain the reconstructed trajectory data of the current vehicle; wherein the trajectory reconstruction model is a large language model.

9. A computer-readable storage medium, characterized in that: It stores a computer program, which, when running on a processor, executes the automatic driving trajectory reconstruction method based on a large language model as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, which, when executed on a computer, enables the computer to execute the automatic driving trajectory reconstruction method based on a large language model as claimed in any one of claims 1 to 7.

Images