Automatic driving strategy generation method and device and vehicle-mounted large model deployment method and device

Abstract

The embodiment of the invention provides an automatic driving strategy generation method and device and a vehicle-mounted large model deployment method and device, and relates to the technical field of automatic driving, the method is applied to an automatic driving system, a vehicle-mounted large model is deployed in the automatic driving system, the vehicle-mounted large model comprises a plurality of network layers, and the method comprises the following steps: calibrating vehicle-mounted data by using the calibration vehicle-mounted data; determining an abnormal weight ratio of each network layer; according to the abnormal weight proportion of the plurality of network layers, the sparseness of each network layer is determined, and the sparseness of each network layer is in negative correlation with the abnormal weight proportion; and the current vehicle-mounted data is processed by using the target weight included in each network layer to obtain an automatic driving strategy, and the ratio of the target weight included in each network layer to all the weights is the sparseness of each network layer. According to the scheme, the deployment of the large language model in the automatic driving system can be realized under the condition that the computing power of the vehicle-mounted chip is insufficient.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method and device for generating an autonomous driving strategy and deploying a large vehicle-mounted model. Background Art

[0002] With the continuous development of autonomous driving technology, the amount of in-vehicle data generated by autonomous vehicles is experiencing explosive growth. This surge in in-vehicle data is improving the perception, decision-making, and control capabilities of autonomous driving systems, enabling safer and more efficient driving. However, this surge in in-vehicle data also poses significant challenges to data storage, transmission, and processing, placing significant computing power demands on in-vehicle chips, and the demand for this computing power is continuously increasing. However, the computing power of current in-vehicle chips cannot keep up with the speed at which in-vehicle sensors acquire data.

[0003] The integration of large language models into autonomous driving systems is essential for improving vehicle intelligence and offers a new approach to enhancing the reasoning, explainability, and decision-making capabilities of autonomous driving systems. However, due to the sheer size of their parameters and the insufficient computing power of onboard chips, large language models face deployment challenges. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a method and apparatus for generating autonomous driving strategies and deploying large onboard models, so as to enable the deployment of large language models in autonomous driving systems when the computing power of onboard chips is insufficient. The specific technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating an autonomous driving strategy, which is applied to an autonomous driving system. A large vehicle model is deployed in the autonomous driving system, and the large vehicle model includes multiple network layers. The method includes:

[0006] Using calibration vehicle data, determine the anomaly weight ratio of each network layer;

[0007] Determining the sparsity of each network layer according to the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio;

[0008] The target weights included in each network layer are used to process the current vehicle data to obtain the autonomous driving strategy. The ratio of the target weights included in each network layer to all weights is the sparsity of each network layer.

[0009] In some embodiments, the step of determining the abnormal weight ratio of each network layer using the calibration vehicle data includes:

[0010] Determining an anomaly score for each weight included in each network layer and an average of anomalies for multiple weights included in each network layer using the calibration vehicle data;

[0011] The abnormal weight ratio of each network layer is determined based on the abnormal scores and abnormal average values ​​of multiple weights included in each network layer.

[0012] In some embodiments, the calibration on-board data includes sub-data linked to each weight included in each network layer;

[0013] The step of determining the anomaly score of each weight included in each network layer and the anomaly average of multiple weights included in each network layer using the calibration vehicle data includes:

[0014] The product of the absolute value of each weight and the second-order norm of the sub-data linked to the weight is used to obtain the abnormal score of each weight included in each network layer;

[0015] The average of the anomaly scores of the multiple weights included in each network layer is calculated to obtain the average anomaly score of the multiple weights included in each network layer.

[0016] In some embodiments, the step of determining the abnormal weight ratio of each network layer based on the abnormal scores and abnormal average values ​​of the multiple weights included in each network layer includes:

[0017] Determine the weights of each network layer that meet a preset condition as abnormal weights, where the preset condition is that the abnormal score of the weight is greater than the abnormal average value of the network layer to which the weight belongs by a first hyperparameter multiple;

[0018] The ratio of the number of abnormal weights included in each network layer to the number of weights included in the network layer is determined to obtain the abnormal weight ratio of each network layer.

[0019] In some embodiments, the step of determining the sparsity of each network layer based on the abnormal weight ratios of the multiple network layers includes:

[0020] The sparsity of each network layer is determined according to the negative correlation between the sparsity and the abnormal weight ratio, the first value range of the sparsity and the average sparsity, and the abnormal weight ratios of the multiple network layers.

[0021] In some embodiments, the minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

[0022] In some embodiments, the step of determining the abnormal weight ratio of each network layer using the calibration vehicle data includes: converting the calibration vehicle data into calibration text data; determining the abnormal weight ratio of each network layer using the calibration text data;

[0023] The step of using the target weight included in each network layer to process the current vehicle data to obtain the autonomous driving strategy includes: converting the current vehicle data into current text data; using the target weight included in each network layer to process the current text data to obtain the autonomous driving strategy.

[0024] In some embodiments, the step of determining the abnormal weight ratio of each network layer using the calibration vehicle data includes: inputting the calibration vehicle data into a vector encoder to obtain a calibration embedded vector; determining the abnormal weight ratio of each network layer using the calibration embedded vector;

[0025] The step of using the target weights included in each network layer to process the current vehicle data to obtain the autonomous driving strategy includes: inputting the current vehicle data into a vector encoder to obtain a current embedded vector; and using the target weights included in each network layer to process the current embedded vector to obtain the autonomous driving strategy.

[0026] In some embodiments, an actuator is further deployed in the autonomous driving system, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning process. After obtaining the autonomous driving strategy, the method further includes:

[0027] Sending the strategic planning program to the executor so that the executor executes the strategic planning program;

[0028] Demonstrate the reasoning ideas of the strategy.

[0029] In some embodiments, the calibration in-vehicle data and the current in-vehicle data include at least one of the following data: human instructions, driving strategy evaluation, system messages, driving scenario data, and historical interaction information.

[0030] In a second aspect, an embodiment of the present application provides a method for deploying a large vehicle model, the method comprising:

[0031] evaluating safety index scores of the plurality of large models when deployed on a first vehicle, and evaluating efficiency index scores of the plurality of large models when deployed on the first vehicle;

[0032] The safety index score and efficiency index score of each large model are weighted averaged to obtain the comprehensive score of each large model;

[0033] The large model with the highest comprehensive score is deployed in the autonomous driving system of the first vehicle, and the autonomous driving system is used to execute any one of the methods provided in the first aspect.

[0034] In some embodiments, the safety index score includes a time-to-collision score;

[0035] The step of evaluating the safety index scores of the plurality of large models when deployed on the first vehicle comprises:

[0036] When executing the driving strategy of the first vehicle using each large model, evaluating a collision occurrence time between the first vehicle corresponding to each large model and each second vehicle;

[0037] A collision time score for each large model is determined based on a safety time threshold and multiple collision occurrence times corresponding to each large model.

[0038] In some embodiments, the step of determining the collision time score of each large model based on the safety time threshold and the multiple collision occurrence times corresponding to each large model includes:

[0039] Determine the minimum collision occurrence time corresponding to each large model from the multiple collision occurrence times corresponding to each large model;

[0040] When the minimum collision occurrence time corresponding to each large model is greater than the safety time threshold, determining the collision time score of each large model to be the maximum score value;

[0041] When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined to be the minimum score value;

[0042] When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0043] In some embodiments, the safety index score includes a speed variance score;

[0044] The step of evaluating the safety index scores of the plurality of large models when deployed on the first vehicle comprises:

[0045] When executing the driving strategy of the first vehicle using each large model, evaluating the speed variance of the first vehicle corresponding to each large model;

[0046] The speed variance score of each large model is determined according to the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and a third value range from the minimum score value to the maximum score value.

[0047] In some embodiments, the step of determining the speed variance score of each large model based on the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and a third value range from the minimum score value to the maximum score value includes:

[0048] Calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation to obtain the speed ratio corresponding to each large model;

[0049] The speed variance score of each large model is determined according to the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and a third value range from the minimum score value to the maximum score value.

[0050] In some embodiments, the efficiency index score includes a time efficiency score;

[0051] The step of evaluating the efficiency index scores of the plurality of large models when deployed on the first vehicle comprises:

[0052] When using each large model to execute the driving strategy of the first vehicle, evaluating the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy;

[0053] The time efficiency score of each large model is determined according to the negative correlation between the consumption time and the time efficiency score, the consumption time corresponding to each large model, and a fourth value range from the minimum score value to the maximum score value.

[0054] In some embodiments, the step of determining the time efficiency score of each large model based on the negative correlation between the consumption time and the time efficiency score, the consumption time corresponding to each large model, and a fourth value range from the minimum score value to the maximum score value includes:

[0055] Calculate the ratio of the consumption time corresponding to each large model to the preset time limit to obtain the time ratio corresponding to each large model;

[0056] The time efficiency score of each large model is determined based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value.

[0057] In a third aspect, an embodiment of the present application provides an autonomous driving strategy generation device, which is applied to an autonomous driving system. A large vehicle model is deployed in the autonomous driving system, and the large vehicle model includes multiple network layers. The device includes:

[0058] A first determination module is used to determine the abnormal weight ratio of each network layer using the calibration vehicle data;

[0059] A second determining module is configured to determine the sparsity of each network layer according to the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio;

[0060] The inference module is used to process the current vehicle data using the target weights included in each network layer to obtain the autonomous driving strategy. The ratio of the target weights included in each network layer to all weights is the sparsity of each network layer.

[0061] In some embodiments, the first determination module is specifically used to: use the calibrated vehicle data to determine the abnormal score of each weight included in each network layer, and the abnormal average value of multiple weights included in each network layer; determine the abnormal weight ratio of each network layer based on the abnormal scores and abnormal average values ​​of multiple weights included in each network layer.

[0062] In some embodiments, the calibration on-board data includes sub-data linked to each weight included in each network layer;

[0063] The first determination module is specifically used to: multiply the absolute value of each weight by the second-order norm of the sub-data linked to the weight to obtain the anomaly score of each weight included in each network layer; calculate the average of the anomaly scores of multiple weights included in each network layer to obtain the anomaly average of the multiple weights included in each network layer.

[0064] In some embodiments, the first determination module is specifically used to: determine the weights included in each network layer that meet preset conditions as abnormal weights, where the preset condition is that the abnormal score of the weight is greater than the abnormal average value of the network layer to which the weight belongs by a first hyperparameter multiple; determine the ratio of the number of abnormal weights included in each network layer to the number of weights included in the network layer, and obtain the abnormal weight ratio of each network layer.

[0065] In some embodiments, the second determination module is specifically used to determine the sparsity of each network layer based on the negative correlation between sparsity and abnormal weight ratio, the first value range and average sparsity of sparsity, and the abnormal weight ratio of the multiple network layers.

[0066] In some embodiments, the minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

[0067] In some embodiments, the first determination module is specifically configured to: convert the calibration vehicle data into calibration text data; and determine the abnormal weight ratio of each network layer using the calibration text data;

[0068] The reasoning module is specifically used to: convert the current vehicle data into current text data; use the target weight included in each network layer to process the current text data to obtain an autonomous driving strategy.

[0069] In some embodiments, the first determination module is specifically configured to: input the calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and determine the abnormal weight ratio of each network layer using the calibration embedded vector;

[0070] The inference module is specifically used to: input the current vehicle data into the vector encoder to obtain the current embedded vector; use the target weight included in each network layer to process the current embedded vector to obtain the autonomous driving strategy.

[0071] In some embodiments, the autonomous driving system further comprises an actuator, the autonomous driving strategy comprises a strategy planning program and a strategy reasoning approach; and the device further comprises:

[0072] An execution module is used to send the strategy planning program to the executor after obtaining the autonomous driving strategy, so that the executor executes the strategy planning program; and demonstrate the strategy reasoning ideas.

[0073] In some embodiments, the calibration in-vehicle data and the current in-vehicle data include at least one of the following data: human instructions, driving strategy evaluation, system messages, driving scenario data, and historical interaction information.

[0074] In a fourth aspect, an embodiment of the present application provides a vehicle-mounted large model deployment device, the device comprising:

[0075] an evaluation module, configured to evaluate safety index scores of the plurality of large models when deployed on a first vehicle, and to evaluate efficiency index scores of the plurality of large models when deployed on the first vehicle;

[0076] The weighting module is used to perform a weighted average of the safety index score and efficiency index score of each large model to obtain a comprehensive score for each large model;

[0077] A deployment module is used to deploy the large model with the highest comprehensive score in the automatic driving system of the first vehicle, and the automatic driving system is used to execute any one of the methods provided in the first aspect.

[0078] In some embodiments, the safety index score includes a time-to-collision score;

[0079] The evaluation module is specifically configured to: evaluate the collision time between the first vehicle and each second vehicle corresponding to each large model when each large model is used to execute the driving strategy of the first vehicle; and determine a collision time score for each large model based on a safety time threshold and multiple collision times corresponding to each large model.

[0080] In some embodiments, the evaluation module is specifically configured to:

[0081] Determine the minimum collision occurrence time corresponding to each large model from the multiple collision occurrence times corresponding to each large model;

[0082] When the minimum collision occurrence time corresponding to each large model is greater than the safety time threshold, determining the collision time score of each large model to be the maximum score value;

[0083] When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined to be the minimum score value;

[0084] When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0085] In some embodiments, the safety index score includes a speed variance score;

[0086] The evaluation module is specifically configured to: evaluate the speed variance of the first vehicle corresponding to each large model when each large model is used to execute the driving strategy of the first vehicle; and determine the speed variance score of each large model based on the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and a third value range from a minimum score value to a maximum score value.

[0087] In some embodiments, the evaluation module is specifically used to: calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation to obtain the speed ratio corresponding to each large model; determine the speed variance score of each large model based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and a third value range from the minimum score value to the maximum score value.

[0088] In some embodiments, the efficiency index score includes a time efficiency score;

[0089] The evaluation module is specifically used to: when using each large model to execute the driving strategy of the first vehicle, evaluate the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy; determine the time efficiency score of each large model based on the negative correlation between the consumed time and the time efficiency score, the consumed time corresponding to each large model, and a fourth value range from the minimum score value to the maximum score value.

[0090] In some embodiments, the evaluation module is specifically used to: calculate the ratio of the consumption time corresponding to each large model to the preset time limit to obtain the time ratio corresponding to each large model; determine the time efficiency score of each large model based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value.

[0091] In a fifth aspect, an embodiment of the present application provides an autonomous driving system, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0092] Memory for storing computer programs;

[0093] The processor is configured to implement any one of the methods provided in the first aspect when executing a program stored in the memory.

[0094] In a sixth aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0095] Memory for storing computer programs;

[0096] The processor is configured to implement any one of the methods provided in the second aspect when executing a program stored in the memory.

[0097] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any method provided in the first aspect, or implements any method provided in the second aspect.

[0098] In an eighth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods provided in the first aspect, or execute any of the methods provided in the second aspect.

[0099] Beneficial effects of the embodiments of the present application:

[0100] In the technical solution provided by the embodiments of this application, the onboard large model is deployed within the autonomous driving system on the vehicle. The autonomous driving system uses calibrated onboard data to determine the abnormal weight ratio of each network layer included in the onboard large model, and then determines the sparsity of each network layer, so that the sparsity of each network layer is negatively correlated with the abnormal weight ratio of each network layer. The abnormal weights in the network layer have a strong correlation with the computing power required by the onboard large model, while the abnormal weights have little impact on the accuracy of the autonomous driving strategy. In an embodiment of the present application, the sparsity of each network layer is negatively correlated with the abnormal weight ratio of each network layer, that is, the larger the abnormal weight ratio of a network layer, the fewer the normal weights (that is, target weights) involved in the calculation in the network layer, and the target weights involved in the calculation in each network layer match the abnormal weight ratio. This non-uniform layer-by-layer sparsity method is used to exclude the abnormal weights in each network layer, and to use the normal weights in each network layer to process real-time vehicle data to obtain an autonomous driving strategy. While ensuring the accuracy of the autonomous driving strategy, the weights involved in the calculation of the large on-board model are greatly reduced, thereby reducing the overall computing power requirements of the large on-board model, and realizing the deployment of a large language model in an autonomous driving system when the computing power of the on-board chip is insufficient.

[0101] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0103] Figure 1 A schematic diagram of a first flow chart of the method for generating an autonomous driving strategy provided in an embodiment of the present application;

[0104] Figure 2A schematic structural diagram of a vector encoder provided in an embodiment of the present application;

[0105] Figure 3 A schematic diagram of the structure of a large vehicle model provided in an embodiment of the present application;

[0106] Figure 4 for Figure 1 A schematic diagram of step S101 in FIG.

[0107] Figure 5 A schematic diagram of determining anomaly scores according to an embodiment of the present application;

[0108] Figure 6 A second flow chart of the method for generating an autonomous driving strategy provided in an embodiment of the present application;

[0109] Figure 7 A schematic diagram of a pruning strategy based on layer-by-layer abnormal distribution provided in an embodiment of the present application;

[0110] Figure 8 A flow chart of a method for deploying a large vehicle model according to an embodiment of the present application;

[0111] Figure 9 A schematic diagram of the structure of the autonomous driving strategy generation device provided in an embodiment of the present application;

[0112] Figure 10 A schematic diagram of the structure of a vehicle-mounted large model deployment device provided in an embodiment of the present application;

[0113] Figure 11 A schematic diagram of the structure of an autonomous driving system provided in an embodiment of the present application;

[0114] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0115] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0116] With the continuous advancement of autonomous driving technology, the amount of in-vehicle data generated is exploding. Currently, a single autonomous vehicle can generate tens of terabytes (TB) of data per day. This data primarily comes from multiple sensors, such as lidar, millimeter-wave radar, and cameras. For example, a high-resolution lidar can generate millions of data points per second, and multiple cameras operating simultaneously also generate a large amount of image data. For example, a self-driving car equipped with multiple advanced sensors can generate over 5TB of data in a single hour of driving. This massive amount of in-vehicle data is crucial for the training and optimization of autonomous driving systems. By analyzing and processing this in-vehicle data, autonomous driving systems can continuously improve their perception, decision-making, and control capabilities, achieving safer and more efficient driving. However, this explosive growth in in-vehicle data also poses significant challenges to data storage, transmission, and processing, and places enormous computing power demands on in-vehicle chips.

[0117] As demand for in-vehicle computing power continues to increase, the computing power of automotive chips continues to rise. However, this computing power still cannot keep up with the speed at which in-vehicle sensors acquire data, resulting in an awkward mismatch between in-vehicle data and computing power. Following this logic, as demand for computing power increases, the computing power of in-vehicle chips must be increased. To meet the computing power demands of future vehicles for autonomous driving, the only way is to continuously increase the computing power of in-vehicle chips and develop high-end chips. This involves adopting new process technologies, new instruction set architectures, new Internet Protocols (IP), improving chip yields, and keeping pace with the technology and products of leading computing companies. However, due to numerous constraints, the computing power of in-vehicle chips is limited in the short timeframe required to achieve this technological leap.

[0118] The integration of large language models with autonomous driving systems is essential for improving vehicle intelligence and is a new approach to enhancing the system's reasoning, interpretability, and decision-making capabilities. The integration of large language models with autonomous driving systems can be achieved using the following methods:

[0119] The first method is to call a pre-trained large language model through an application programming interface (API). The API-based method has shown flexibility in using pre-trained large language models to complete various driving tasks.

[0120] This approach relies on cloud server connections to deploy large language models, leveraging state-of-the-art pre-trained large language models to achieve optimal performance. However, this approach relies on remote deployment of large language models and its implementation depends on network conditions and server data processing speed, limiting its effectiveness in complex driving scenarios.

[0121] The second approach involves training and deploying large language models locally in the vehicle. This approach relies on local computing resources to deploy large language models, addressing the cloud computing and network quality requirements of autonomous driving systems. However, due to the billions of parameters in large language models, this approach places high demands on local computing resources. The limited computing power of onboard chipsets has limited practical application in autonomous vehicles.

[0122] Currently, there are several solutions for integrating large language models with autonomous driving systems:

[0123] (1) Language Model Predictive Control (Language MPC) driving decision-making system: Utilizes a large language model to reason about complex scenarios, derives an autonomous driving policy, and then converts the autonomous driving policy into a control signal using a parameter matrix. This approach demonstrates the potential of large language models in bridging the gap between abstract reasoning and concrete vehicle control.

[0124] (2) Generative Pre-trained Transformer-Driver (GPT-Driver): This system reformulates motion planning as a natural language modeling task by converting heterogeneous scene inputs into language tokens. GPT-Driver is a multimodal large-model language framework that receives a sequence of image frames and natural language (i.e., questions) as input, generates responses to human inquiries, and predicts the next control signal.

[0125] (3) Large Language Model-Driver (LLM-Driver): This approach uses a large language model to reason about the scene and derive an autonomous driving strategy, which is then converted into control signals using a parameter matrix. This approach enables real-time question-and-answering about the driving environment and enhances the system’s ability to explain its decisions to the user.

[0126] (4) Language Model Drive (LMDrive): This system uses a visual encoder to process images and generate visual tags. LMDrive is an end-to-end closed-loop autonomous driving framework designed to process multimodal sensor data in conjunction with natural language instructions.

[0127] While numerous solutions exist for integrating large language models with autonomous driving systems, their implementation for end-to-end autonomous driving computation remains in its infancy. Existing integration solutions focus on convolutional neural networks (CNNs) to integrate large language models with autonomous driving systems, remaining largely unexplored. These solutions rely on cloud server connections or local computing resources to deploy large language models, which presents the limitations mentioned above. Furthermore, these approaches often incur extended computation times, hindering real-time data processing.

[0128] To solve the above problems, the present invention provides a method for generating an autonomous driving strategy. Figure 1 As shown, it is applied to an autonomous driving system, in which a large vehicle model is deployed. The large vehicle model includes multiple network layers. The method includes the following steps.

[0129] Step S101, using the calibration vehicle data, determine the abnormal weight ratio of each network layer;

[0130] Step S102, determining the sparsity of each network layer based on the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio;

[0131] In step S103, the current vehicle data is processed using the target weights included in each network layer to obtain an autonomous driving strategy. The ratio of the target weights included in each network layer to all weights is the sparsity of each network layer.

[0132] In the technical solution provided by the embodiments of this application, the onboard large model is deployed within the autonomous driving system on the vehicle. The autonomous driving system uses calibrated onboard data to determine the abnormal weight ratio of each network layer included in the onboard large model, and then determines the sparsity of each network layer, so that the sparsity of each network layer is negatively correlated with the abnormal weight ratio of each network layer. The abnormal weights in the network layer have a strong correlation with the computing power required by the onboard large model, while the abnormal weights have little impact on the accuracy of the autonomous driving strategy. In an embodiment of the present application, the sparsity of each network layer is negatively correlated with the abnormal weight ratio of each network layer, that is, the larger the abnormal weight ratio of a network layer, the fewer the normal weights (that is, target weights) involved in the calculation in the network layer, and the target weights involved in the calculation in each network layer match the abnormal weight ratio. This non-uniform layer-by-layer sparsity method is used to exclude the abnormal weights in each network layer, and to use the normal weights in each network layer to process real-time vehicle data to obtain an autonomous driving strategy. While ensuring the accuracy of the autonomous driving strategy, the weights involved in the calculation of the large on-board model are greatly reduced, thereby reducing the overall computing power requirements of the large on-board model, and realizing the deployment of a large language model in an autonomous driving system when the computing power of the on-board chip is insufficient.

[0133] In the embodiments of this application, the autonomous driving system is a system on a vehicle that implements autonomous driving decisions. A vehicle equipped with the autonomous driving system can be referred to as an autonomous vehicle. The onboard big model is the large language model deployed within the autonomous driving system. In other words, the onboard big model is the large language model deployed on the vehicle. The number of network layers included in the onboard big model can be set based on actual needs.

[0134] In step S101, the calibration vehicle data is the vehicle data used to calibrate the sparsity of multiple network layers included in the vehicle-mounted large model. The calibration vehicle data may include, but is not limited to, human instructions, driving strategy evaluation, system messages, driving scenario data, historical interaction information, etc.

[0135] Human commands are natural language input directly from the user to the onboard big model. Human commands can include the user's desired requirements for the autonomous driving subject. For example, the user can input a human command to the onboard big model: "Overtake the vehicle ahead."

[0136] Driving strategy evaluation is the user's feedback to the onboard big model on the effectiveness of the autonomous driving strategy. That is, after the onboard big model processes the onboard data such as human instructions input by the user and outputs the autonomous driving strategy, the user inputs the driving strategy evaluation to the onboard big model based on the autonomous driving strategy output by the onboard big model. For example, the onboard big model outputs the autonomous driving strategy of "no vehicle ahead, accelerate at 10m / s 2Increase the vehicle speed to 80 km / h”; based on this autonomous driving strategy, the user can input a driving strategy evaluation into the onboard large model: “This autonomous driving strategy is rated 4 points.”

[0137] System messages provide instructions or context to both the user and the autonomous driving system. They are a set of high-level guidelines or rules that outline how the vehicle's big model should operate and make decisions. These high-level system messages define the basic driving logic, including task definition, traffic rules to be followed, description of decision states, and overall goals or metrics to be optimized. System messages serve as a foundational framework, guiding the behavior and decision-making process of autonomous vehicles on the road. Without well-designed system messages, the vehicle's big model may make incorrect assumptions or adopt unintended strategies.

[0138] Driving scenario data describes the driving scenario and may include, but is not limited to, route information, vehicle information, pedestrian information, and information about the vehicle's environment. Raw driving scenario data can include images, videos, analog signals, digital signals, and other types of data. Vehicle sensors can convert raw driving scenario data into textual descriptions, namely driving scenario data. Driving scenario data follows a predefined structure and uses natural language to provide a detailed textual description of the current driving scene, such as "You are in the leftmost lane of a two-way highway" or "There is a car 50 meters ahead of your current position." This textual description intuitively translates the complex spatial and temporal relationships between different road users and the vehicle into a natural language format, enabling the onboard big model to reason about driving scenario information. The purpose of driving scenario data is to provide the onboard big model with a comprehensive representation of the situation, enabling it to make appropriate decisions under current traffic conditions.

[0139] Historical interaction information is the historical interaction data between the user and the autonomous driving system. The autonomous driving system can be configured with a memory module that stores profiles of different users to enhance the driving experience for all users. Each time a user engages the autonomous driving system, the system records their interactions with the system. Upon completion of the trip, the system updates the user's historical interaction data to the corresponding profile in the memory module. The specific content of this historical interaction data varies for each autonomous driving system. This historical user interaction data can be used as a reference for user preferences and fed into the onboard large-scale model to guide the autonomous driving system in enhancing the user experience.

[0140] In the embodiment of the present application, the abnormal weight ratio is the ratio of abnormal weights. The autonomous driving system can use events or periodic triggers to execute step S101, obtain calibration vehicle data, and use the calibration vehicle data to determine the abnormal weight ratio of each network layer, so that the autonomous driving system can obtain layer-by-layer abnormality distribution. The layer-by-layer abnormality distribution includes the abnormal weight ratios of multiple network layers, for example, the layer-by-layer abnormality distribution ROI = [D1, D2, ..., D n ], D m Represents the abnormal weight ratio of network layer m, m = 1, 2, ..., n, and n represents the number of network layers included in the vehicle-mounted large model.

[0141] For example, when the autonomous driving system obtains human quality and needs to perform a driving task, the autonomous driving system executes step S101 to obtain the abnormal weight ratio and sparsity that are suitable for the current driving task, thereby improving the accuracy of the autonomous driving strategy.

[0142] For another example, the autonomous driving system periodically executes step S101. In this way, when executing a driving task, the last acquired abnormal weight ratio and sparsity can be directly used to execute step S103, thereby improving the efficiency of generating the autonomous driving strategy.

[0143] In the above step S102, the sparsity of the network layer represents the proportion of weights involved in the calculation in the network layer, which can be expressed by the following formula (1).

[0144] P m =λ 1m / λ 2m (1)

[0145] In formula (1), P m represents the sparsity of network layer m, λ 1m Indicates the number of weights involved in the calculation in network layer m, / λ 2m Represents the total number of weights in network layer m.

[0146] The sparsity of a network layer is negatively correlated with the abnormal weight ratio. In an example, the negative correlation between the sparsity of a network layer and the abnormal weight ratio can be expressed by the following formula (2) or formula (3).

[0147] P m =k1 / D m (2)

[0148] P m =k2(1-D m ) (3)

[0149] In formula (2) and formula (3), P m represents the sparsity of network layer m, D mIt represents the abnormal weight ratio of network layer m, k1 and k2 represent negative correlation hyperparameters, and the values ​​of k1 and k2 can be set according to actual needs.

[0150] The negative correlation between the sparsity of the network layer and the abnormal weight ratio can also be expressed in other ways, which are not limited to this.

[0151] In an embodiment of the present application, for each network layer included in the on-board large model, the autonomous driving system can determine the sparsity of the network layer based on the negative correlation between sparsity and abnormal weight ratio, and the abnormal weight ratio of the network layer.

[0152] In step S103 above, the target weights are the weights involved in the calculation. The number of target weights included in different network layers can be the same or different. Current vehicle data is the vehicle data collected in real time during the execution of the current driving task. Current vehicle data may include, but is not limited to, human instructions, driving strategy evaluations, system messages, driving scenario data, and historical interaction information.

[0153] The autonomous driving system obtains the current vehicle data and inputs the current vehicle data into the on-board large model. The on-board large model prunes according to the sparsity of each network layer to obtain the target weight included in each network layer. Then, the target weight included in each network layer is used to infer the current vehicle data to obtain the autonomous driving strategy.

[0154] In the embodiment of the present application, the autonomous driving system eliminates abnormal weights (i.e., unnecessary weights) included in each network layer according to the sparsity of each network layer, which is equivalent to removing the low-rank adaptive module. This reduces the volume of the on-board large model and the amount of calculation of the on-board large model while maintaining the performance of the on-board large model, improving the accuracy and utilization of the on-board large model, and reducing the phenomenon of calculation jams of the on-board large model.

[0155] In some embodiments, step S101 may include converting the calibration vehicle data into calibration text data; and using the calibration text data, determining the abnormal weight ratio of each network layer. Step S103 may include converting the current vehicle data into current text data; and processing the current text data using the target weights included in each network layer to obtain an autonomous driving strategy.

[0156] The in-vehicle large model is a large language model, and its input data is general text data. That is, the layer-by-layer anomaly distribution ROI is calculated using general text data. In-vehicle data includes images, videos, analog signals, digital signals, and other types of data. In the context of autonomous driving, in-vehicle data consists of vector representations of driving scenarios. Significant differences exist between in-vehicle data and general text data, which can lead to significantly different results for the layer-by-layer anomaly distribution ROI calculation, potentially affecting the pruning results of the in-vehicle large model.

[0157] In an embodiment of the present application, the autonomous driving system converts the calibration vehicle data into text data (i.e., calibration text data) and converts the current vehicle data into text data (i.e., current text data). The autonomous driving system uses the calibration text data to calculate a layer-by-layer anomaly distribution ROI, and then uses the layer-by-layer anomaly distribution ROI to prune the vehicle-mounted large model, and then infers the current text data, ensuring that the pruned vehicle-mounted large model is more compatible with the downstream autonomous driving task, and ensuring that the pruning process can be customized to the specific needs and characteristics of the autonomous driving task, resulting in a more effective pruning effect.

[0158] In an embodiment of the present application, when the on-board large model supports vector representation of the driving scene, the autonomous driving system directly uses the calibrated on-board data to calculate the layer-by-layer abnormal distribution ROI, and then uses the layer-by-layer abnormal distribution ROI to prune the on-board large model, and then directly performs inference processing on the current on-board data, without the need to convert the calibrated on-board data and the current on-board data into text data, thereby improving the generation efficiency of the autonomous driving strategy.

[0159] In some embodiments, step S101 may include: inputting the calibrated vehicle data into a vector encoder to obtain a calibrated embedded vector; and using the calibrated embedded vector, determining the abnormal weight ratio of each network layer. Step S103 may include: inputting the current vehicle data into a vector encoder to obtain a current embedded vector; and processing the current embedded vector using the target weights included in each network layer to obtain an autonomous driving strategy.

[0160] In the embodiment of the present application, the autonomous driving system can also deploy a vector encoder, and the structure of the vector encoder can be set according to actual needs. In one example, the vector encoder can be used Figure 2 As shown in the structure, the vector encoder includes multiple multilayer perceptrons (MLPs), multiple MLPs connected to a cross attention layer, the cross attention layer connected to 7 self-attention layers, and 7 self-attention layers connected to a cross attention layer.

[0161] based on Figure 2The vector encoder shown in the figure allows the autonomous driving system to convert on-board data such as route information, vehicle information, pedestrian information, and the vehicle's environment into tokens and input them into the vector encoder. In the vector encoder, each token is input into an MLP for processing. Each token (i.e., vector) processed by the MLP is then projected into a latent space through a cross-attention layer, resulting in multiple latent vectors. Vehicle characteristics are then added to each latent vector to represent the vehicle state. Seven self-attention layers and a cross-attention layer then convert the vehicle state into an embedded vector compatible with the large on-board module. After the vector encoder outputs the embedded vector, the autonomous driving system inserts it into a text prompt embedding, which is then input into the large on-board model for inference processing. Alternatively, the autonomous driving system uses the embedded vector to calculate layer-by-layer anomaly distribution ROIs.

[0162] In the embodiment of the present application, the vector encoder has higher sensitivity than other components of the on-board large model. Using the vector encoder to process on-board data effectively reduces computing requirements while enabling the on-board large model to demonstrate better perception, motion prediction and language comprehension capabilities.

[0163] In some embodiments, step S101 may include: converting the calibration vehicle data into calibration text data; inputting the calibration text data into a vector encoder to obtain a calibration embedded vector; and using the calibration embedded vector, determining the abnormal weight ratio of each network layer. Step S103 may include: converting the current vehicle data into current text data; inputting the current text data into a vector encoder to obtain a current embedded vector; and processing the current embedded vector using the target weights included in each network layer to obtain an autonomous driving strategy.

[0164] In an embodiment of the present application, the autonomous driving system converts the calibration vehicle data into calibration text data, converts the calibration text data into tokens and inputs them into a vector encoder.

[0165] In an embodiment of the present application, the autonomous driving system converts the calibration vehicle data and the current vehicle data into. The autonomous driving system uses the calibration text data to calculate the layer-by-layer abnormal distribution ROI, and then uses the layer-by-layer abnormal distribution ROI to prune the vehicle-mounted large model, and then infers the current text data, ensuring that the pruned vehicle-mounted large model is more consistent with the downstream autonomous driving task, and ensuring that the pruning process can be customized according to the specific needs and characteristics of the autonomous driving task, resulting in a more effective pruning effect. In addition, the vector encoder has a higher sensitivity. Using the vector encoder to process the text data corresponding to the vehicle data effectively reduces the computing requirements while enabling the vehicle-mounted large model to demonstrate better perception, action prediction, and language understanding capabilities.

[0166] In the embodiments of this application, actuators may also be deployed within the autonomous driving system. These actuators may include, but are not limited to, cockpit systems, intelligent driving systems, and chassis systems. The actuators can serve as a bridge between the output text of the onboard large model and the current autonomous driving strategy. The autonomous driving strategy may include a strategy planning program, which is the code that implements the autonomous driving strategy.

[0167] After obtaining the autonomous driving policy (i.e., the policy planning program), the autonomous driving system sends the policy planning program to the actuator. The actuator executes the policy planning program, interacting with the current state of the vehicle to enable the vehicle to implement the desired driving behavior in the real environment or simulated environment.

[0168] In embodiments of the present application, a display may be deployed within the autonomous driving system, or the autonomous driving system may be connected to a device with a display. The autonomous driving strategy may also include a strategy reasoning process, which is the thought process used to infer the strategy planning program. This strategy reasoning process helps the large onboard model generate more reasonable autonomous driving strategies.

[0169] After obtaining the autonomous driving policy (i.e., the policy planning program and policy reasoning), the autonomous driving system sends the policy planning program to the actuator. The actuator executes the policy planning program, interacting with the current state of the vehicle, enabling the vehicle to implement the desired driving behavior in a real or simulated environment. The autonomous driving system also sends the policy reasoning to the display, presenting the policy reasoning in text form. This helps users understand the obtained autonomous driving policy and subsequently evaluate it. The driving policy evaluation is then input into the onboard large model to optimize the autonomous driving policy.

[0170] The following combination Figure 3 The large vehicle model architecture shown provides a detailed explanation of the autonomous driving strategy generation method provided in the embodiment of this application.

[0171] The autonomous driving system draws historical interaction information from the memory module, combines it with human commands, driving strategy assessments, system messages, and driving scenario data, and then feeds this information into the vehicle's large onboard model. The model's multiple network layers perform inference processing on this input data, outputting a policy planning program and policy reasoning. The actuator executes the policy planning program, enabling the vehicle to implement the intended driving behavior in real-world environments or simulated environments such as Navigate on Autopilot (NOA) and Car Learning to Act (CARLA).

[0172] At the same time, the autonomous driving system records the historical interaction information between the user and the autonomous driving system. When the trip (i.e., the driving task) is completed, the autonomous driving system can update the historical interaction information to the memory module.

[0173] In some embodiments, as Figure 4 As shown, the above step S101 may include:

[0174] Step S1011, using the calibration vehicle data, determining the anomaly score of each weight included in each network layer, and the anomaly average of multiple weights included in each network layer;

[0175] In this embodiment of the present application, for each weight included in each network layer, the autonomous driving system can use the calibration vehicle data to determine an anomaly score for that weight. The anomaly score of a weight indicates the degree of anomaly of the weight, with a larger anomaly score indicating a higher degree of anomaly of the weight.

[0176] In some embodiments, the calibration vehicle data includes sub-data linked to each weight included in each network layer. The autonomous driving system may multiply the absolute value of each weight by the second-order norm of the sub-data linked to the weight to obtain an anomaly score for each weight included in each network layer; and calculate the average of the anomaly scores of the multiple weights included in each network layer to obtain an average anomaly value for the multiple weights included in each network layer.

[0177] For example, for a large vehicle-mounted model with n network layers, for each weight included in each network layer, the autonomous driving system can use the following formula (4) to determine the abnormality score of the weight.

[0178]

[0179] In formula (4), Represents the weights included in network layer m The anomaly score, X j Indicates link to weight All sub-data (i.e. input features), represents the weight of the i-th row and j-th column of the network layer m, ||||2 represents the second-order norm, and || represents the absolute value. Figure 5 The anomaly score is determined as shown in the following example. The network layer m includes the weight w 11 ~w 33 , calibration vehicle data including X 11 、X 21 、X 31 、X 41 etc. link to w 11 、w 21 、w 31 Input features X1, X 12 、X22 、X 32 、X 42 etc. link to w 12 、w 22 、w 32 Input features X2, X 13 、X 23 、X 33 、X 43 etc. link to w 13 、w 23 、w 33 Input feature X3. With input feature 1, w 11 For example, the weight w 31 The anomaly score is:

[0180] The autonomous driving system can also use other methods to determine the anomaly score of each weight included in each network layer. For example, for a large vehicle-mounted model with n network layers, the autonomous driving system can use the following formula (5) to determine the anomaly score of each weight included in each network layer.

[0181]

[0182] In formula (5), Represents the weights included in network layer m The anomaly score, X j Indicates link to weight All sub-data (i.e. input features), represents the weight of the i-th row and j-th column of the network layer m, |||| represents the first-order norm, and || represents the absolute value.

[0183] In the embodiment of the present application, the method for determining the abnormality score of the weight is not limited, as long as the larger the abnormality score is, the higher the abnormality degree of the weight is.

[0184] Step S1012 : determining the abnormal weight ratio of each network layer according to the abnormal scores and abnormal average values ​​of the multiple weights included in each network layer.

[0185] After obtaining the anomaly scores of multiple weights included in each network layer and the average anomaly value of each network layer, the autonomous driving system can compare the anomaly score of each weight included in each network layer with the average anomaly value of the network layer to determine whether the weight is an abnormal weight and the proportion of abnormal weights in the network layer, i.e., the abnormal weight proportion.

[0186] In some embodiments, the autonomous driving system can determine the weights included in each network layer that meet preset conditions as abnormal weights, where the preset condition is that the abnormal score of the weight is greater than the abnormal average value of the network layer to which the weight belongs by a first hyperparameter multiple; determine the ratio of the number of abnormal weights included in each network layer to the number of weights included in the network layer, and obtain the abnormal weight ratio of each network layer.

[0187] In the embodiment of the present application, the autonomous driving system pre-configures a first hyperparameter, which is a hyperparameter used to determine the abnormality threshold. The preset conditions are: in, Represents the weights included in network layer m The anomaly score, represents the average abnormality of network layer m, and M represents the first hyperparameter. The value of M can be set according to actual needs.

[0188] For each network layer, the autonomous driving system can determine the network layer includes more than The weight of is used as the abnormal weight, and then the first number of abnormal weights included in the network layer and the total number of weights included in the network layer (such as the second number) are determined. The ratio of the first number to the second number is calculated to obtain the abnormal weight ratio of each network layer.

[0189] In the embodiment of the present application, the determination of the abnormal weight ratio of each network layer can be expressed by the following formula (6).

[0190]

[0191] In formula (6), D m represents the abnormal weight ratio of network layer m, Represents the weights included in network layer m The anomaly score, represents the average abnormal value of network layer m, M represents the first hyperparameter, E in Indicates the number of dimensions of the input layer, that is, the number of rows of the weight matrix included in the network layer m, E out Represents the number of dimensions of the output layer, that is, the number of columns of the weight matrix included in the network layer m, I() represents the indicator function, when the preset conditions are met, I returns 1, otherwise I returns 0. Based on this, we can obtain the layer-by-layer abnormal distribution ROI = [D1, D2, ..., D n ],like Figure 5 shown.

[0192] In the embodiments of the present application, the autonomous driving system may also use other methods to determine the abnormal weight ratio of each network layer, which is not limited to this. As long as the abnormal weight ratio is determined to be larger, the number of abnormal weights of the network layer is greater. For example, the autonomous driving system may determine the weights of each network layer that meet the preset conditions as abnormal weights, and determine the ratio of the number of abnormal weights included in each network layer to the number of weights included in all network layers to obtain the abnormal weight ratio of each network layer. Specifically, it can be expressed by the following formula (7).

[0193]

[0194] In formula (7), D m represents the abnormal weight ratio of network layer m, Represents the weights included in network layer m The anomaly score, It represents the abnormal average value of network layer m, M represents the first hyperparameter, H represents the total number of all weights included in network layer 1 to network layer n, and I() represents the indicator function. When the preset condition is met, I returns 1, otherwise I returns 0.

[0195] In the embodiment of the present application, the autonomous driving system quantifies the abnormal distribution between each network layer based on the pruning of the layer-by-layer abnormal distribution, and obtains the layer-by-layer abnormal distribution ROI = [D1, D2, ..., D n By quantifying the anomaly distribution at each layer, the autonomous driving system can match the sparsity of each layer with its anomaly weight ratio, thereby ensuring the preservation of key features that are critical to maintaining model performance.

[0196] In some embodiments, as Figure 6 As shown, a method for generating an autonomous driving strategy is also provided, which is applied to an autonomous driving system. A large vehicle model is deployed in the autonomous driving system. The large vehicle model includes multiple network layers. The method may include the following steps.

[0197] Step S601, using the calibration vehicle data, determines the abnormal weight ratio of each network layer; the same as the above step S101.

[0198] Step S602: determining the sparsity of each network layer based on the negative correlation between the sparsity and the abnormal weight ratio, the first value range of the sparsity and the average sparsity, and the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio;

[0199] In the embodiment of the present application, the negative correlation between sparsity and abnormal weight ratio can be expressed by the above formula (2) or formula (3). The average sparsity is the average sparsity of all network layers included in the vehicle-mounted large model, which is also the target sparsity of the vehicle-mounted large model. The first value range is the value range of sparsity, and the first value range can be expressed as [P 小 , P 大 ], the first value range can effectively prevent the sparsity difference between network layers from being too large. In one example, the autonomous driving system can introduce a hyperparameter, such as the second hyperparameter δ, the minimum value P of the first value range 小 is the difference between the average sparsity and the second hyperparameter, the maximum value P of the first value range 大 is the sum of the average sparsity and the second hyperparameter, that is, the first value range can be expressed as [P-δ, P+δ], where P is the average sparsity. The first value range can also be expressed in other ways, which are not limited to this.

[0200] In the embodiment of the present application, the autonomous driving system can adjust the negatively correlated hyperparameters (such as k1 and k2) so that when calculating the sparsity of each network layer according to the negative correlation between sparsity and abnormal weight ratio and the abnormal weight ratio of each network layer, the minimum sparsity calculated is greater than or equal to P 小 , the calculated maximum sparsity is less than or equal to P 大 In addition, the autonomous driving system can adjust the calculated sparsity of each network layer so that the average of the sparsity of all network layers is equal to the average sparsity.

[0201] In the embodiment of the present application, the autonomous driving system can also directly calculate the sparsity of each network layer according to the negative correlation between sparsity and abnormal weight ratio and the abnormal weight ratio of each network layer; if the calculated sparsity of a network layer is less than P 小 , then the sparsity of the network layer is set to P 小 ; If the calculated sparsity of a network layer is greater than P 大 , then the sparsity of the network layer is set to P 大 If the calculated sparsity of a network layer is less than P 小 , then the sparsity of the network layer is set to P 小 ; If the calculated sparsity of a network layer is greater than or equal to P 小 , and the sparsity of the network layer is less than or equal to P 大 , the sparsity of the network layer remains unchanged. In addition, the autonomous driving system can adjust the calculated sparsity of each network layer so that the average sparsity of all network layers is equal to the average sparsity.

[0202] In the embodiment of the present application, the autonomous driving system may also use other methods to determine the sparsity of each network layer, which is not limited to this.

[0203] Step S603: Process the current vehicle data using the target weights of each network layer to obtain an autonomous driving strategy. The ratio of the target weights of each network layer to all weights is the sparsity of each network layer. This is the same as step S103 above.

[0204] In the technical solution provided in the embodiment of the present application, the autonomous driving system determines the sparsity of each network layer based on the negative correlation between sparsity and the abnormal weight ratio, the first value range of sparsity and the average sparsity, so that the sparsity of each network layer is within the first value range, and the average sparsity of all network layers is maintained at the average sparsity, thereby preventing the sparsity difference between network layers from being too large, and avoiding the average sparsity of all network layers from being too large or too small, avoiding excessive pruning of the on-board large model, affecting the accuracy of the on-board large model, and avoiding excessive pruning of the on-board large model, resulting in excessive computational load required for the on-board large model, and causing the on-board large model calculation to be stuck.

[0205] The following combination Figure 2 The vector encoder shown and Figure 7 The pruning strategy based on layer-by-layer abnormal distribution shown is used to provide a detailed description of the autonomous driving strategy generation method provided in an embodiment of the present application.

[0206] The autonomous driving system can convert the calibrated vehicle data such as route information, vehicle information, pedestrian information and the vehicle's environmental information into tokens and input them into the vector encoder. The autonomous driving system inputs the embedded vector output by the vector encoder and activates the pruning strategy based on layer-by-layer anomaly distribution. The pruning strategy based on layer-by-layer anomaly distribution uses the embedded vector to determine the layer-by-layer anomaly distribution ROI. The process of determining the layer-by-layer anomaly distribution ROI by the pruning strategy can be seen above. Figure 5 and Figure 6 Description of the part.

[0207] After obtaining the layer-by-layer anomaly distribution ROI, the autonomous driving system converts calibrated onboard data, such as route information, vehicle information, pedestrian information, and the vehicle's environmental information, into tokens and inputs them into a vector encoder. The autonomous driving system then inserts the embedded vectors output by the vector encoder into a text prompt embedding, which is then fed into the onboard large model for inference processing. The large model outputs a policy planning program and policy reasoning ideas, and sends the policy planning program to actuators such as the cockpit system, intelligent driving system, and chassis system to implement the corresponding driving behavior.

[0208] The embodiment of the present application proposes a pruning strategy based on layer-by-layer anomaly distribution, which is an efficient autonomous driving framework strategy. This framework strategy adopts a weighted layer-by-layer pruning method based on outliers (i.e., anomaly weights), uses the target weights in each network layer for reasoning, and can compress the data input of the large vehicle model without fine-tuning. By allocating non-uniform sparsity based on layer-by-layer anomaly distribution and incorporating the vehicle data of the driving environment into the calibration and pruning process, this method significantly reduces the computational requirements of the large vehicle model while maintaining the performance of the large vehicle model. The specific advantages are as follows:

[0209] (1) The embodiment of the present application proposes an autonomous driving framework with the help of a large language model pruning strategy. The autonomous driving framework uses layer-by-layer sparsity based on outlier weighting to compress the input data of the large on-board model, which can solve a series of problems such as the computational complexity of the large on-board model and insufficient hardware computing power.

[0210] (2) The embodiment of the present application utilizes a model pruning method, which can remove unnecessary weights for autonomous driving (i.e., abnormal weights). That is, the abnormal weights are not involved in the calculation, but the model performance can be maintained, the accuracy of the large on-board model can be improved, the model size can be reduced, the utilization rate of the model can be improved, and the calculation jamming of the large on-board model can be reduced.

[0211] (3) The present embodiment effectively addresses the computational limitations of large language models in vehicles by employing layer-by-layer sparsity weighted by outliers. By processing and reasoning complex environmental data and natural language input from multiple sensor modalities, autonomous vehicles can make more complex and efficient decisions.

[0212] (4) The embodiment of the present application applies a vector encoder in the autonomous driving system. Compared with other large model components, the vector encoder has higher sensitivity. The application of the vector encoder can effectively reduce computing requirements while demonstrating better perception, action prediction and language understanding capabilities.

[0213] (5) The embodiments of the present application combine pruning technology with large language models, and are expected to develop efficient and stable autonomous driving and cockpit systems that can handle complex scenarios.

[0214] The present application also provides a method for deploying a large vehicle model. Figure 8 As shown, the method includes the following steps:

[0215] Step S801, evaluating safety index scores when multiple large models are deployed on a first vehicle, and evaluating efficiency index scores when multiple large models are deployed on the first vehicle;

[0216] Step S802: performing a weighted average of the safety index score and the efficiency index score of each large model to obtain a comprehensive score for each large model;

[0217] In step S803, the large model with the highest comprehensive score is deployed in the autonomous driving system of the first vehicle, and the autonomous driving system is used to execute the above-mentioned autonomous driving strategy generation method.

[0218] The technical solution provided in the embodiments of this application uses safety and efficiency indicators to evaluate each large model, and the one with the highest score is selected for installation on the vehicle. This approach not only reduces the amount of data input for the large model, but also significantly mitigates some of the uncertainty associated with the large model installation process, thereby improving the performance of the large model in autonomous driving tasks.

[0219] In the embodiment of the present application, the vehicle-mounted large model deployment method can be applied to servers, clusters, and other devices. The following description is based on electronic devices as the execution subject, which is not limiting. When executing the vehicle-mounted large model deployment method, the first vehicle on which the vehicle-mounted large model (i.e., the large language model) needs to be deployed is located in an autonomous driving simulation environment, and the autonomous driving simulation environment may also include n other vehicles (such as a second vehicle), where n is greater than or equal to 1, that is, the autonomous driving simulation environment includes n+1 vehicles.

[0220] In the above step S801, the safety index is used to measure the ability of the vehicle to maintain a safe distance and avoid collision. The safety index may include but is not limited to time to collision (TTC), speed variance (SV), etc. The efficiency index is used to evaluate the efficiency of the autonomous driving strategy within the predetermined time limit T limit The ability to complete autonomous driving tasks within a certain timeframe. Efficiency indicators include but are not limited to time efficiency (TE).

[0221] The electronic device is configured with a safety indicator evaluation method and an efficiency indicator evaluation method. For each large model, the electronic device evaluates the safety indicator score (i.e., safety indicator score) of the large model when deployed on the first vehicle using the safety indicator evaluation method, and evaluates the efficiency indicator score (i.e., efficiency indicator score) of the large model when deployed on the first vehicle using the efficiency indicator evaluation method.

[0222] The following describes how to evaluate TTC, SV, and TE.

[0223] (1)TTC.

[0224] When the safety index includes collision time, that is, the safety index score includes a collision time score, in the above-mentioned step S801, the step of evaluating the safety index score when multiple large models are deployed on the first vehicle can be: when using each large model to execute the driving strategy of the first vehicle, evaluating the collision time between the first vehicle corresponding to each large model and each second vehicle; and determining the collision time score of each large model based on the safety time threshold and the multiple collision times corresponding to each large model.

[0225] In the embodiment of the present application, the safety time threshold T safe The size of can be set according to actual needs, for example, T safe It can be 2 seconds, 3 seconds, etc.

[0226] For each large model, when the large model is deployed on the first vehicle and the driving strategy of the first vehicle is executed using the large model, the electronic device can use the following formula (8) to determine the time of collision between the first vehicle and each second vehicle.

[0227]

[0228] In formula (8), Tc i represents the time of collision between the first vehicle and the second vehicle i, P0 represents the position of the first vehicle, P i represents the position of the second vehicle i, v0 represents the longitudinal velocity of the first vehicle, v i represents the longitudinal velocity of the second vehicle i, i=1,…,n.

[0229] During all time steps t in which the driving strategy of the first vehicle is executed using the large model, the electronic device can calculate the collision occurrence time Tc between the first vehicle and each second vehicle according to formula (8): i That is, for each second vehicle i, at each time step t, the electronic device can calculate a Tc i .

[0230] After the large model is used to execute the driving strategy of the first vehicle, the electronic device may determine a collision time score of the large model according to a safety time threshold and multiple collision occurrence times corresponding to the large model.

[0231] In one example, the electronic device can determine the minimum collision occurrence time corresponding to each large model from multiple collision occurrence times corresponding to each large model; when the minimum collision occurrence time corresponding to each large model is greater than the safety time threshold, the collision time score of each large model is determined to be the maximum score value; when the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined to be the minimum score value; when the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0232] In the embodiment of the present application, the second value range is the value range of the collision time score. For example, the safety time threshold is 2 seconds, the maximum score is 100, and the minimum score is 0. For each large model, the electronic device can use the following formula (9) and formula (10) to determine the collision time score of the large model.

[0233]

[0234] In formula (9) and formula (10), TTC is the collision time score of the large model, T min represents the minimum collision time corresponding to the large model; n represents the number of the second vehicle, represents the time when the collision between the first vehicle and the second vehicle i occurs at time step t.

[0235] In the embodiment of the present application, if the collision time between the first vehicle and the second vehicle is greater than or equal to 0, it means that the first vehicle and the second vehicle may collide, and if the collision time between the first vehicle and the second vehicle is less than 0, it means that the first vehicle and the second vehicle cannot collide. The electronic device can determine the minimum collision time T corresponding to the large model from the collision time corresponding to the large model that is greater than or equal to 0. min .

[0236] In formula (10), only (maximum score value - 1 / T min ) is used as an example to represent the positive correlation between the minimum collision time and the safety index score, which is not limiting. For example, the positive correlation between the minimum collision time and the safety index score can also be expressed as (T min / safe time threshold)×maximum score value, such as (T min / 2)×100.

[0237] In an embodiment of the present application, the electronic device controls the collision time score of each large model between the minimum score value and the maximum score value, thereby avoiding the collision time score of the large model being too large or too small, improving the accuracy of the safety index score of the large model, and improving the accuracy of the assessment of the large model.

[0238] (2)SV.

[0239] When the safety index includes speed variance, that is, the safety index score includes the speed variance score, in the above-mentioned step S801, the step of evaluating the safety index score when multiple large models are deployed on the first vehicle can be: when using each large model to execute the driving strategy of the first vehicle, evaluating the speed variance of the first vehicle corresponding to each large model; determining the speed variance score of each large model based on the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and the third value range from the minimum score value to the maximum score value.

[0240] In the embodiment of the present application, the third value range is the value range of the speed variance score. The third value range can be the same as the second value range mentioned above, so as to unify the benchmarks of the speed variance score and the collision time score, so as to facilitate the management of the evaluation criteria of the large model. The third value range can also be different from the second value range mentioned above, such as dividing the second value range and the third value range according to the importance of the benchmarks of the speed variance score and the collision time score. For example, the importance of the speed variance score is higher than that of the collision time score. The second value range is 0 to 100, and the third value range is 0 to 200, so as to facilitate the accurate evaluation of the large model.

[0241] For each large model, when the large model is deployed on the first vehicle and the driving strategy of the first vehicle is executed using the large model, the electronic device can use the following formula (11) to determine the speed variance of the first vehicle corresponding to the large model.

[0242]

[0243] In formula (11), S0 represents the speed variance of the first vehicle corresponding to the large model, T represents the time required to complete the driving strategy of the first vehicle using the large model, t represents the time step before the driving strategy of the first vehicle is completed using the large model, v0 represents the longitudinal speed of the first vehicle, δ0 represents the average longitudinal speed of the first vehicle corresponding to the large model, and δ0 can be determined using the following formula (12).

[0244]

[0245] In formula (12), δ0 represents the average longitudinal speed of the first vehicle corresponding to the large model, T represents the time required to complete the driving strategy of the first vehicle using the large model, t represents the time step before the driving strategy of the first vehicle is completed using the large model, and v0 represents the longitudinal speed of the first vehicle.

[0246] After the large model is used to execute the driving strategy of the first vehicle, the electronic device can determine the speed variance score of the large model based on the negative correlation between the speed variance and the speed variance score, the third value range, and the speed variance corresponding to the large model.

[0247] In one example, the electronic device can calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation to obtain the speed ratio corresponding to each large model; based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and the third value range from the minimum score value to the maximum score value, determine the speed variance score of each large model.

[0248] In the embodiment of the present application, the maximum score value is 100, the minimum score value is 0, and the third value range is 0 to 100. For each large model, the electronic device can use the following formula (13) to determine the speed variance score of the large model.

[0249]

[0250] In formula (13), SV is the speed variance score of the large model, S0 represents the speed variance of the first vehicle corresponding to the large model; S safe Indicates the preset safety speed deviation. The preset safety speed deviation can be set according to actual needs. In formula (13), only the maximum score value × (1-S0 / S safe ) represents the negative correlation between the speed ratio and the speed variance score, that is, the negative correlation between the speed variance and the speed variance score as an example, and does not play a limiting role.

[0251] For example, the negative correlation between speed variance and speed variance score can also be expressed as:

[0252]

[0253] At this time, taking the maximum score value as 100 as an example, the speed variance score of the large model can also be determined using the following formula (14).

[0254]

[0255] In formula (14), SV is the speed variance score of the large model, S0 represents the speed variance of the first vehicle, S safeIndicates the preset safety speed deviation.

[0256] In an embodiment of the present application, the electronic device controls the speed variance score of each large model between the minimum score value and the maximum score value, thereby avoiding the speed variance score of the large model being too large or too small, improving the accuracy of the safety index score of the large model, and improving the accuracy of the assessment of the large model.

[0257] (3)TE.

[0258] When the efficiency index includes time efficiency, that is, the efficiency index score includes the time efficiency score, in the above-mentioned step S801, the step of evaluating the efficiency index score when multiple large models are deployed on the first vehicle can be: when using each large model to execute the driving strategy of the first vehicle, evaluating the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy; according to the negative correlation between the consumed time and the time efficiency score, the consumed time corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value, determine the time efficiency score of each large model.

[0259] In the embodiment of the present application, the fourth value range is the value range of the time efficiency score. The fourth value range can be the same as the second value range mentioned above, so as to unify the benchmarks of the time efficiency score and the collision time score, so as to facilitate the management of the evaluation criteria of the large model. The fourth value range can also be different from the second value range mentioned above, such as dividing the second value range and the fourth value range according to the importance of the benchmarks of the time efficiency score and the collision time score. For example, the importance of the time efficiency score is higher than that of the collision time score. The second value range is 0 to 100, and the fourth value range is 0 to 200, so as to facilitate the accurate evaluation of the large model.

[0260] For each large model, the large model is deployed on the first vehicle, and the driving strategy of the first vehicle is executed using the large model. The time T consumed by the first vehicle to complete the operation corresponding to the driving strategy is, that is, the time T required to complete the driving strategy of the first vehicle using the large model.

[0261] After the driving strategy of the first vehicle is executed using the large model, the electronic device can determine the time efficiency score of the large model based on the negative correlation between the consumption time and the time efficiency score, the fourth value range, and the consumption time corresponding to the large model.

[0262] In one example, the electronic device can calculate the ratio of the consumption time corresponding to each large model to the preset time limit to obtain the time ratio corresponding to each large model; based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value, determine the time efficiency score of each large model.

[0263] In the embodiment of the present application, the maximum score value is 100, the minimum score value is 0, and the fourth value range is 0 to 100. For each large model, the electronic device can use the following formula (15) to determine the time efficiency score of the large model.

[0264]

[0265] In formula (15), TE is the time efficiency score of the large model, T represents the consumption time corresponding to the large model, and T limit Indicates the scheduled time limit. The scheduled time limit can be set according to actual needs. In formula (15), only the maximum score value × (1-T / T limit ) represents the negative correlation between the time ratio and the time efficiency score, that is, the negative correlation between the consumed time and the time efficiency score as an example, and does not serve as a limitation.

[0266] For example, the negative correlation between consumption time and time efficiency score can also be expressed as:

[0267]

[0268] At this time, taking the maximum score value as 100 as an example, the time efficiency score of the large model can also be determined using the following formula (16).

[0269]

[0270] In formula (16), TE is the time efficiency score of the large model, T represents the speed variance of the first vehicle, and T limit Indicates a scheduled time limit.

[0271] In an embodiment of the present application, the electronic device controls the time efficiency score of each large model between the minimum score value and the maximum score value, thereby avoiding the time efficiency score of the large model being too large or too small, improving the accuracy of the efficiency index score of the large model, and improving the accuracy of the assessment of the large model.

[0272] In step S802 above, the electronic device pre-configures weights for each indicator. Using the pre-configured weights, the electronic device performs a weighted average of the security and efficiency scores for each large model to obtain a comprehensive score for each large model. Specifically, the following formula (17) can be used to determine the comprehensive score for each large model.

[0273] Score=w TTC ×TTC+ w SV ×SV+w TE ×TE (17)

[0274] In formula (17), Score represents the comprehensive score of the large model, w TTC represents the weight of the collision time score, w SV represents the weight of the speed variance score, w TE represents the weight of the time efficiency score, TTC represents the collision time score corresponding to the large model, SV represents the speed variance score corresponding to the large model, and TE represents the time efficiency score corresponding to the large model.

[0275] In step S83, after obtaining the comprehensive scores for each large model, the electronic device selects the large model with the highest comprehensive score and deploys the selected large model within the autonomous driving system of the first vehicle. Subsequently, the large model serves as the onboard large model of the first vehicle, and the autonomous driving system of the first vehicle can use this onboard large model to execute the aforementioned autonomous driving strategy generation method.

[0276] In the embodiment of the present application, while reducing the amount of data input to the large language model, a large language model on-vehicle evaluation scheme is proposed, and TTC, SV, and TE are proposed. Through objective experimental evaluation, each type of large model suitable for vehicle installation is scored, and the large language model with a higher score is selected as the vehicle-mounted large model installation scheme.

[0277] The main goal of the on-board large model selection strategy is to evaluate the performance of the intelligent agent based on the large language model in interpreting human instructions in driving scenarios, and the large language model uses the provided functional primitives to generate code for motion planning.

[0278] In the embodiments of this application, two rule-based baseline strategies are employed: an intelligent driver model and a minimum lane change-induced overall braking principle. The former describes a rule that updates the vehicle acceleration based on the specific speed and relative speed of the vehicle and the object directly ahead to avoid collisions. The latter is an extension of the former, executing a lane change if the anticipated new lane offers a more favorable driving scenario and the maneuver can be performed safely. These baseline strategies can be considered to operate randomly because they are independent of human instructions and only follow pre-defined rules.

[0279] Large language models are able to generate coherent solutions for various autonomous driving tasks without additional fine-tuning. However, in terms of code generation (such as generating policy planning programs), especially for complex road scenarios, large language models may produce suboptimal results. The autoregressive nature of these large language models poses a huge challenge to autonomous driving because the tags generated early in the historical interaction sequence cannot be modified in the same round of iteration. This limitation restricts the ability of large language models to optimize the initial impact. The embodiment of the present application proposes a vehicle-mounted large model selection strategy (such as Figure 8This can effectively improve some uncertainties in the process of loading the large vehicle model and improve the performance of the large vehicle model in autonomous driving test tasks.

[0280] In the technical solution provided in the embodiment of the present application, the on-board big model plays the role of the decision-making "brain" in the autonomous driving system, and the perception or positioning module plays the role of the vehicle's "eyes" in the autonomous driving system. The on-board big model does not directly affect the perception or positioning module. The information collected by the perception or positioning module is used as a reference to guide the high-level decision-making process. By receiving processed data from these perception or positioning modules, the on-board big model can enhance the ability to make wise decisions, thereby significantly improving the performance of autonomous driving vehicles. In the downstream link, the vehicle's control module (i.e., actuator) plays the role of "hands" in the autonomous driving system, and the vehicle's control module executes the driving strategy obtained from the decision-making process based on the on-board big model. Combined with this architecture, the autonomous driving system can execute Figure 1 The autonomous driving strategy generation method shown (i.e., pruning strategy based on layer-by-layer abnormal distribution) is used to generate autonomous driving strategies.

[0281] The inventors used the technical solution provided in the embodiments of the present application to conduct experimental evaluations on the LLaMA7B large model and the LLaMA13B large model, and found that the large language model using the pruning strategy based on layer-by-layer anomaly distribution reduced the data input by 0.4 times compared to the original large language model that did not use the pruning strategy based on layer-by-layer anomaly distribution, while achieving 3 times the end-to-end inference speed in the inference engine.

[0282] Corresponding to the above-mentioned autonomous driving strategy generation method, the embodiment of the present application provides an autonomous driving strategy generation device, such as Figure 9 As shown, it is applied to an autonomous driving system, in which a large vehicle model is deployed. The large vehicle model includes multiple network layers. The device includes:

[0283] The first determination module 901 is used to determine the abnormal weight ratio of each network layer using the calibration vehicle data;

[0284] A second determining module 902 is configured to determine the sparsity of each network layer based on the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio;

[0285] The reasoning module 903 is used to process the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy. The ratio of the target weights included in each network layer to all weights is the sparsity of each network layer.

[0286] In some embodiments, the first determination module 901 is specifically used to: use the calibrated vehicle data to determine the abnormal score of each weight included in each network layer, and the abnormal average value of multiple weights included in each network layer; determine the abnormal weight ratio of each network layer based on the abnormal scores and abnormal average values ​​of multiple weights included in each network layer.

[0287] In some embodiments, the calibration onboard data includes sub-data linked to each weight included in each network layer;

[0288] The first determination module 901 is specifically used to: multiply the absolute value of each weight by the second-order norm of the sub-data linked to the weight to obtain the anomaly score of each weight included in each network layer; calculate the average of the anomaly scores of multiple weights included in each network layer to obtain the anomaly average of the multiple weights included in each network layer.

[0289] In some embodiments, the first determination module 901 is specifically used to: determine the weights included in each network layer that meet preset conditions as abnormal weights, where the preset condition is that the abnormal score of the weight is greater than the abnormal average value of the network layer to which the weight belongs by a first hyperparameter multiple; determine the ratio of the number of abnormal weights included in each network layer to the number of weights included in the network layer, and obtain the abnormal weight ratio of each network layer.

[0290] In some embodiments, the second determination module 902 is specifically used to determine the sparsity of each network layer based on the negative correlation between sparsity and abnormal weight ratio, the first value range of sparsity and the average sparsity, and the abnormal weight ratio of multiple network layers.

[0291] In some embodiments, the minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

[0292] In some embodiments, the first determination module 901 is specifically configured to: convert the calibration vehicle data into calibration text data; and determine the abnormal weight ratio of each network layer using the calibration text data;

[0293] The inference module is specifically used to: convert the current vehicle data into current text data; use the target weights included in each network layer to process the current text data to obtain the autonomous driving strategy.

[0294] In some embodiments, the first determination module 901 is specifically configured to: input the calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and determine the abnormal weight ratio of each network layer using the calibration embedded vector;

[0295] The inference module is specifically used to: input the current vehicle data into the vector encoder to obtain the current embedded vector; use the target weights included in each network layer to process the current embedded vector to obtain the autonomous driving strategy.

[0296] In some embodiments, an actuator is further deployed within the autonomous driving system, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning process; the autonomous driving strategy generation device further includes:

[0297] The execution module is used to send the strategy planning program to the actuator after obtaining the autonomous driving strategy, so that the actuator can execute the strategy planning program; it also demonstrates the strategy reasoning ideas.

[0298] In some embodiments, the calibration vehicle data and the current vehicle data include at least one of the following data: human instructions, driving strategy evaluation, system messages, driving scenario data, and historical interaction information.

[0299] In the technical solution provided by the embodiments of this application, the onboard large model is deployed within the autonomous driving system on the vehicle. The autonomous driving system uses calibrated onboard data to determine the abnormal weight ratio of each network layer included in the onboard large model, and then determines the sparsity of each network layer, so that the sparsity of each network layer is negatively correlated with the abnormal weight ratio of each network layer. The abnormal weights in the network layer have a strong correlation with the computing power required by the onboard large model, while the abnormal weights have little impact on the accuracy of the autonomous driving strategy. In an embodiment of the present application, the sparsity of each network layer is negatively correlated with the abnormal weight ratio of each network layer, that is, the larger the abnormal weight ratio of a network layer, the fewer the normal weights (that is, target weights) involved in the calculation in the network layer, and the target weights involved in the calculation in each network layer match the abnormal weight ratio. This non-uniform layer-by-layer sparsity method is used to exclude the abnormal weights in each network layer, and to use the normal weights in each network layer to process real-time vehicle data to obtain an autonomous driving strategy. While ensuring the accuracy of the autonomous driving strategy, the weights involved in the calculation of the large on-board model are greatly reduced, thereby reducing the overall computing power requirements of the large on-board model, and realizing the deployment of a large language model in an autonomous driving system when the computing power of the on-board chip is insufficient.

[0300] Corresponding to the above-mentioned vehicle-mounted large model deployment method, the embodiment of the present application also provides a vehicle-mounted large model deployment device, such as Figure 10 As shown, the device includes:

[0301] An evaluation module 1001 is configured to evaluate safety index scores when multiple large models are deployed on a first vehicle, and to evaluate efficiency index scores when multiple large models are deployed on the first vehicle;

[0302] The weighting module 1002 is used to perform a weighted average of the safety index score and the efficiency index score of each large model to obtain a comprehensive score for each large model;

[0303] The deployment module 1003 is used to deploy the large model with the highest comprehensive score in the automatic driving system of the first vehicle, and the automatic driving system is used to execute any of the above-mentioned automatic driving strategy generation methods.

[0304] In some embodiments, the safety index score includes a time-to-collision score;

[0305] The evaluation module 1001 is specifically used to: when using each large model to execute the driving strategy of the first vehicle, evaluate the collision time between the first vehicle corresponding to each large model and each second vehicle; and determine the collision time score of each large model based on the safety time threshold and the multiple collision times corresponding to each large model.

[0306] In some embodiments, the evaluation module 1001 is specifically configured to:

[0307] Determine the minimum collision occurrence time corresponding to each large model from the multiple collision occurrence times corresponding to each large model;

[0308] When the minimum collision occurrence time corresponding to each large model is greater than the safety time threshold, the collision time score of each large model is determined to be the maximum score value;

[0309] When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined to be the minimum score value;

[0310] When the minimum collision time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision time and the safety index score, the minimum collision time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0311] In some embodiments, the safety index score includes a speed variance score;

[0312] Evaluation module 1001 is specifically configured to: evaluate the speed variance of the first vehicle corresponding to each large model when each large model is used to execute the driving strategy of the first vehicle; and determine the speed variance score of each large model based on the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and a third value range from the minimum score value to the maximum score value.

[0313] In some embodiments, the evaluation module 1001 is specifically used to: calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation to obtain the speed ratio corresponding to each large model; determine the speed variance score of each large model based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and a third value range from the minimum score value to the maximum score value.

[0314] In some embodiments, the efficiency indicator score includes a time efficiency score;

[0315] Evaluation module 1001 is specifically used to: when using each large model to execute the driving strategy of the first vehicle, evaluate the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy; determine the time efficiency score of each large model based on the negative correlation between the consumed time and the time efficiency score, the consumed time corresponding to each large model, and a fourth value range from the minimum score value to the maximum score value.

[0316] In some embodiments, the evaluation module 1001 is specifically used to: calculate the ratio of the consumption time corresponding to each large model to the preset time limit to obtain the time ratio corresponding to each large model; determine the time efficiency score of each large model based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value.

[0317] The technical solution provided in the embodiments of this application uses safety and efficiency indicators to evaluate each large model, and the one with the highest score is selected for installation on the vehicle. This approach not only reduces the amount of data input for the large model, but also significantly mitigates some of the uncertainty associated with the large model installation process, thereby improving the performance of the large model in autonomous driving tasks.

[0318] Corresponding to the above-mentioned automatic driving strategy generation method, the embodiment of the present application also provides an automatic driving system, such as Figure 11 As shown, it includes a processor 1101 , a communication interface 1102 , a memory 1103 and a communication bus 1104 , wherein the processor 1101 , the communication interface 1102 , and the memory 1103 communicate with each other via the communication bus 1104 .

[0319] Memory 1103, used for storing computer programs;

[0320] The processor 1101 is configured to implement any of the above-mentioned autonomous driving strategy generation methods when executing the program stored in the memory 1103.

[0321] Corresponding to the above-mentioned vehicle-mounted large model deployment method, the embodiment of the present application also provides an electronic device, such as Figure 12 As shown, it includes a processor 1201 , a communication interface 1202 , a memory 1203 and a communication bus 1204 , wherein the processor 1201 , the communication interface 1202 and the memory 1203 communicate with each other via the communication bus 1204 .

[0322] Memory 1203, used for storing computer programs;

[0323] The processor 1201 is configured to implement any of the above-mentioned vehicle-mounted large model deployment methods when executing the program stored in the memory 1203.

[0324] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, the figure shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0325] The communication interface is used for communication between the above electronic device and other devices.

[0326] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0327] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0328] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements any of the above-mentioned autonomous driving strategy generation methods, or implements any of the above-mentioned vehicle-mounted large model deployment methods.

[0329] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned autonomous driving strategy generation methods, or execute any of the above-mentioned vehicle-mounted large model deployment methods.

[0330] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0331] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0332] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device, autonomous driving system, electronic device, storage medium, and program product embodiments are generally similar to the method embodiments, so their descriptions are simplified. For relevant portions, reference can be made to the descriptions of the method embodiments.

[0333] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A method for generating an autonomous driving strategy, characterized in that: Applied to an autonomous driving system, wherein a large vehicle model is deployed in the autonomous driving system, and the large vehicle model includes multiple network layers, the method includes: Using calibration vehicle data, determine the anomaly weight ratio of each network layer; Determining the sparsity of each network layer according to the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio; The target weights included in each network layer are used to process the current vehicle data to obtain the autonomous driving strategy. The ratio of the target weights included in each network layer to all weights is the sparsity of each network layer.

2. The method according to claim 1, characterized in that The step of determining the abnormal weight ratio of each network layer using the calibration vehicle data includes: Determining an anomaly score for each weight included in each network layer and an average of anomalies for multiple weights included in each network layer using the calibration vehicle data; The abnormal weight ratio of each network layer is determined based on the abnormal scores and abnormal average values ​​of multiple weights included in each network layer.

3. The method according to claim 2, characterized in that The calibration on-board data includes sub-data linked to each weight included in each network layer; The step of determining the anomaly score of each weight included in each network layer and the anomaly average of multiple weights included in each network layer using the calibration vehicle data includes: The product of the absolute value of each weight and the second-order norm of the sub-data linked to the weight is used to obtain the abnormal score of each weight included in each network layer; The average of the anomaly scores of the multiple weights included in each network layer is calculated to obtain the average anomaly score of the multiple weights included in each network layer.

4. The method according to claim 2, characterized in that The step of determining the abnormal weight ratio of each network layer according to the abnormal scores and abnormal average values ​​of the multiple weights included in each network layer includes: Determine the weights of each network layer that meet a preset condition as abnormal weights, where the preset condition is that the abnormal score of the weight is greater than the abnormal average value of the network layer to which the weight belongs by a first hyperparameter multiple; The ratio of the number of abnormal weights included in each network layer to the number of weights included in the network layer is determined to obtain the abnormal weight ratio of each network layer.

5. The method according to claim 1, wherein The step of determining the sparsity of each network layer according to the abnormal weight ratio of the multiple network layers includes: The sparsity of each network layer is determined according to the negative correlation between the sparsity and the abnormal weight ratio, the first value range of the sparsity and the average sparsity, and the abnormal weight ratios of the multiple network layers.

6. The method according to claim 5, characterized in that The minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

7. The method according to claim 1, characterized in that The step of determining the abnormal weight ratio of each network layer using the calibration vehicle data includes: converting the calibration vehicle data into calibration text data; determining the abnormal weight ratio of each network layer using the calibration text data; The step of using the target weight included in each network layer to process the current vehicle data to obtain the autonomous driving strategy includes: converting the current vehicle data into current text data; using the target weight included in each network layer to process the current text data to obtain the autonomous driving strategy.

8. The method according to claim 1, characterized in that The step of determining the abnormal weight ratio of each network layer using the calibration vehicle data includes: inputting the calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and determining the abnormal weight ratio of each network layer using the calibration embedded vector; The step of using the target weights included in each network layer to process the current vehicle data to obtain the autonomous driving strategy includes: inputting the current vehicle data into a vector encoder to obtain a current embedded vector; and using the target weights included in each network layer to process the current embedded vector to obtain the autonomous driving strategy.

9. The method according to claim 1, characterized in that An actuator is also deployed in the autonomous driving system, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning idea. After obtaining the autonomous driving strategy, the method further includes: Sending the strategic planning program to the executor so that the executor executes the strategic planning program; Demonstrate the reasoning ideas of the strategy.

10. The method according to any one of claims 1 to 9, characterized in that The calibration vehicle data and the current vehicle data include at least one of the following data: human instructions, driving strategy evaluation, system messages, driving scenario data, and historical interaction information.

11. A method for deploying a large vehicle model, characterized in that: The method comprises: evaluating safety index scores of the plurality of large models when deployed on a first vehicle, and evaluating efficiency index scores of the plurality of large models when deployed on the first vehicle; The safety index score and efficiency index score of each large model are weighted averaged to obtain the comprehensive score of each large model; The large model with the highest comprehensive score is deployed in the autonomous driving system of the first vehicle, and the autonomous driving system is used to execute the method described in any one of claims 1 to 10.

12. The method according to claim 11, characterized in that The safety index score includes a collision time score; The step of evaluating the safety index scores of the plurality of large models when deployed on the first vehicle comprises: When each large model is used to execute the driving strategy of the first vehicle, a collision time between the first vehicle corresponding to each large model and each second vehicle is evaluated; and a collision time score for each large model is determined based on a safety time threshold and multiple collision times corresponding to each large model.

13. The method according to claim 12, characterized in that The step of determining the collision time score of each large model based on the safety time threshold and the multiple collision occurrence times corresponding to each large model includes: Determine the minimum collision occurrence time corresponding to each large model from the multiple collision occurrence times corresponding to each large model; When the minimum collision occurrence time corresponding to each large model is greater than the safety time threshold, determining the collision time score of each large model to be the maximum score value; When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined to be the minimum score value; When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

14. The method according to claim 11, characterized in that The safety index score includes a speed variance score; The step of evaluating the safety index scores of the plurality of large models when deployed on the first vehicle comprises: When executing the driving strategy of the first vehicle using each large model, evaluating the speed variance of the first vehicle corresponding to each large model; The speed variance score of each large model is determined according to the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and a third value range from the minimum score value to the maximum score value.

15. The method according to claim 14, characterized in that The step of determining the speed variance score of each large model based on the negative correlation between the speed variance and the speed variance score, the speed variance corresponding to each large model, and a third value range from the minimum score value to the maximum score value includes: Calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation to obtain the speed ratio corresponding to each large model; determine the speed variance score of each large model based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and the third value range from the minimum score value to the maximum score value.

16. The method according to claim 11, characterized in that The efficiency index score includes a time efficiency score; The step of evaluating the efficiency index scores of the plurality of large models when deployed on the first vehicle comprises: When using each large model to execute the driving strategy of the first vehicle, evaluating the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy; The time efficiency score of each large model is determined according to the negative correlation between the consumption time and the time efficiency score, the consumption time corresponding to each large model, and a fourth value range from the minimum score value to the maximum score value.

17. The method according to claim 16, characterized in that The step of determining the time efficiency score of each large model based on the negative correlation between the consumption time and the time efficiency score, the consumption time corresponding to each large model, and a fourth value range from the minimum score value to the maximum score value includes: Calculate the ratio of the consumption time corresponding to each large model to the preset time limit to obtain the time ratio corresponding to each large model; The time efficiency score of each large model is determined based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value.

18. An automatic driving strategy generating device, characterized in that: Applied to an autonomous driving system, wherein a large vehicle model is deployed in the autonomous driving system, the large vehicle model includes multiple network layers, and the device includes: A first determination module is used to determine the abnormal weight ratio of each network layer using the calibration vehicle data; A second determining module is configured to determine the sparsity of each network layer according to the abnormal weight ratios of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio; The inference module is used to process the current vehicle data using the target weights included in each network layer to obtain the autonomous driving strategy. The ratio of the target weights included in each network layer to all weights is the sparsity of each network layer.

19. A vehicle-mounted large model deployment device, characterized in that: The device comprises: an evaluation module, configured to evaluate safety index scores of the plurality of large models when deployed on a first vehicle, and to evaluate efficiency index scores of the plurality of large models when deployed on the first vehicle; The weighting module is used to perform a weighted average of the safety index score and efficiency index score of each large model to obtain a comprehensive score for each large model; A deployment module, configured to deploy the large model with the highest comprehensive score in the autonomous driving system of the first vehicle, wherein the autonomous driving system is configured to execute the method described in any one of claims 1-10.

Images