An automatic driving strategy generation and vehicle-mounted large model deployment method and device

Abstract

The embodiment of the application provides a kind of automatic driving strategy generation, vehicle-mounted large model deployment method and device, it is related to the field of automatic driving technology, this method is applied to automatic driving system, automatic driving system is deployed vehicle-mounted large model, vehicle-mounted large model includes multiple network layers, method includes: using calibration vehicle-mounted data, determine the abnormal weight proportion of each network layer;According to the abnormal weight proportion of multiple network layers, determine the sparsity of each network layer, the sparsity of each network layer is negatively related to abnormal weight proportion;Using the target weight included in each network layer, process current vehicle-mounted data, obtain automatic driving strategy, the proportion of target weight included in each network layer and all weights is the sparsity of each network layer.The scheme can realize the deployment of large language model in automatic driving system in the case where the computing power of vehicle-mounted chip is insufficient.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method and apparatus for generating autonomous driving strategies and deploying large-scale vehicle models. Background Technology

[0002] With the continuous development of autonomous driving technology, the amount of onboard data is experiencing explosive growth. This surge in onboard data enhances the perception, decision-making, and control capabilities of autonomous driving systems, leading to safer and more efficient driving. However, this explosive growth also presents significant challenges to data storage, transmission, and processing, placing enormous demands on the computing power of onboard chips and driving up the overall demand for onboard computing power. Currently, the computing power of onboard chips is still insufficient to keep pace with the speed at which onboard sensors acquire data.

[0003] The integration of large language models with autonomous driving systems is indispensable for improving the intelligence level of vehicles and is also a new method to enhance the reasoning, interpretability, and decision-making capabilities of autonomous driving systems. However, due to the massive scale of model parameters in large language models and the insufficient computing power of onboard chips, large language models face deployment difficulties. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for generating autonomous driving strategies and deploying large onboard models, so as to enable the deployment of large language models in autonomous driving systems when the computing power of onboard chips is insufficient. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide an autonomous driving strategy generation method, applied to an autonomous driving system, wherein an on-board large model is deployed within the autonomous driving system, the on-board large model comprising multiple network layers, and the method includes:

[0006] Using calibrated vehicle data, the proportion of outlier weights in each network layer is determined;

[0007] Based on the proportion of abnormal weights in the multiple network layers, the sparsity of each network layer is determined, and the sparsity of each network layer is negatively correlated with the proportion of abnormal weights.

[0008] By utilizing the target weights included in each network layer, the current vehicle data is processed to obtain an autonomous driving strategy. The ratio of the target weights to all weights included in each network layer is the sparsity of each network layer.

[0009] In some embodiments, the step of determining the proportion of outlier weights for each network layer using calibrated vehicle data includes:

[0010] Using calibrated onboard data, the outlier score of each weight in each network layer and the outlier average of multiple weights in each network layer are determined.

[0011] The proportion of abnormal weights in each network layer is determined based on the abnormal scores and average values ​​of the multiple weights included in each network layer.

[0012] In some embodiments, the calibration vehicle data includes sub-data linked to each weight included in each network layer;

[0013] The step of using calibrated vehicle data to determine the outlier score of each weight in each network layer, and the outlier average of multiple weights in each network layer, includes:

[0014] The anomaly score for each weight in each network layer is obtained by multiplying the absolute value of each weight by the second norm of the subdata linked to that weight.

[0015] Calculate the average of the outlier scores of the multiple weights included in each network layer to obtain the average outlier score of the multiple weights included in each network layer.

[0016] In some embodiments, the step of determining the proportion of abnormal weights for each network layer based on the abnormal scores and the average abnormal values ​​of the multiple weights included in each network layer includes:

[0017] The weights of each network layer that meet a preset condition are identified as abnormal weights. The preset condition is that the abnormal score of the weight is greater than the average abnormal score of the network layer to which the weight belongs by a first hyperparameter multiple.

[0018] The ratio of the number of outlier weights in each network layer to the total number of weights in that network layer is determined to obtain the outlier weight ratio of each network layer.

[0019] In some embodiments, the step of determining the sparsity of each network layer based on the proportion of outlier weights of the plurality of network layers includes:

[0020] The sparsity of each network layer is determined based on the negative correlation between sparsity and the proportion of outlier weights, the first range of sparsity values ​​and the average sparsity, and the proportion of outlier weights of the multiple network layers.

[0021] In some embodiments, the minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

[0022] In some embodiments, the step of determining the outlier weight ratio of each network layer using calibration vehicle data includes: converting the calibration vehicle data into calibration text data; and using the calibration text data to determine the outlier weight ratio of each network layer.

[0023] The step of processing the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy includes: converting the current vehicle data into current text data; and processing the current text data using the target weights included in each network layer to obtain an autonomous driving strategy.

[0024] In some embodiments, the step of determining the abnormal weight ratio of each network layer using calibration vehicle data includes: inputting calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and using the calibration embedded vector to determine the abnormal weight ratio of each network layer.

[0025] The step of processing the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy includes: inputting the current vehicle data into a vector encoder to obtain a current embedded vector; and processing the current embedded vector using the target weights included in each network layer to obtain an autonomous driving strategy.

[0026] In some embodiments, the autonomous driving system further deploys an actuator, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning approach; after obtaining the autonomous driving strategy, the method further includes:

[0027] The strategy planning program is sent to the executor so that the executor executes the strategy planning program.

[0028] The reasoning process of the strategy is demonstrated.

[0029] In some embodiments, the calibration vehicle data and the current vehicle data include at least one of the following: human commands, driving strategy evaluations, system messages, driving scenario data, and historical interaction information.

[0030] Secondly, embodiments of this application provide a method for deploying a large vehicle-mounted model, the method comprising:

[0031] The safety metrics scores of multiple large models deployed on the first vehicle are evaluated, as are the efficiency metrics scores of multiple large models deployed on the first vehicle.

[0032] The safety and efficiency scores of each large model are weighted and averaged to obtain the comprehensive score of each large model.

[0033] The large model with the highest overall score is deployed within the autonomous driving system of the first vehicle, which is used to execute any of the methods provided in the first aspect.

[0034] In some embodiments, the safety performance rating includes a time-of-collision rating;

[0035] The steps for evaluating safety metrics scores when multiple large models are deployed on a first vehicle include:

[0036] When executing the driving strategy of the first vehicle using each large model, the collision time between the first vehicle and each second vehicle corresponding to each large model is evaluated;

[0037] The collision time score for each large model is determined based on the safety time threshold and the multiple collision occurrence times corresponding to each large model.

[0038] In some embodiments, the step of determining the collision time score of each large model based on a safety time threshold and multiple collision occurrence times corresponding to each large model includes:

[0039] Determine the minimum collision occurrence time for each large model from among the multiple collision occurrence times corresponding to each large model;

[0040] When the minimum collision occurrence time corresponding to each large model is greater than the safe time threshold, the collision time score of each large model is determined as the maximum score value.

[0041] When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined as the minimum score value.

[0042] When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0043] In some embodiments, the safety metric score includes a velocity variance score;

[0044] The steps for evaluating safety metrics scores when multiple large models are deployed on a first vehicle include:

[0045] When executing the driving strategy of the first vehicle using each large model, evaluate the speed variance of the first vehicle corresponding to each large model;

[0046] Based on the negative correlation between velocity variance and velocity variance score, the velocity variance of each large model, and the third range of values ​​from the minimum score to the maximum score, the velocity variance score of each large model is determined.

[0047] In some embodiments, the step of determining the velocity variance score of each large model based on the negative correlation between velocity variance and velocity variance score, the velocity variance corresponding to each large model, and a third value range from the minimum score to the maximum score includes:

[0048] Calculate the ratio of the velocity variance corresponding to each large model to the preset safe velocity deviation to obtain the velocity ratio corresponding to each large model;

[0049] Based on the negative correlation between the velocity ratio and the velocity variance score, the velocity ratio corresponding to each large model, and the third range of values ​​from the minimum score to the maximum score, the velocity variance score of each large model is determined.

[0050] In some embodiments, the efficiency index score includes a time efficiency score;

[0051] The step of evaluating the efficiency metric score when multiple large models are deployed on the first vehicle includes:

[0052] When executing the driving strategy of the first vehicle using each large model, the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy is evaluated.

[0053] Based on the negative correlation between time consumption and time efficiency score, the time consumption corresponding to each large model, and the fourth range of values ​​from the minimum score to the maximum score, the time efficiency score of each large model is determined.

[0054] In some embodiments, the step of determining the time efficiency score of each large model based on the negative correlation between the time consumed and the time efficiency score, the time consumed for each large model, and a fourth value range from the minimum score to the maximum score includes:

[0055] Calculate the ratio of the time consumed by each large model to the preset time limit to obtain the time ratio for each large model;

[0056] Based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth range of values ​​from the minimum score to the maximum score, the time efficiency score of each large model is determined.

[0057] Thirdly, embodiments of this application provide an autonomous driving strategy generation device applied to an autonomous driving system, wherein an in-vehicle large model is deployed within the autonomous driving system, the in-vehicle large model comprising multiple network layers, and the device comprising:

[0058] The first determination module is used to determine the proportion of abnormal weights for each network layer using calibration vehicle data;

[0059] The second determining module is used to determine the sparsity of each network layer based on the abnormal weight ratio of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio.

[0060] The inference module is used to process the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy. The ratio of the target weights to all weights in each network layer is the sparsity of each network layer.

[0061] In some embodiments, the first determining module is specifically configured to: use calibrated vehicle data to determine the abnormal score of each weight included in each network layer, and the abnormal average value of multiple weights included in each network layer; and determine the abnormal weight ratio of each network layer based on the abnormal score and abnormal average value of the multiple weights included in each network layer.

[0062] In some embodiments, the calibration vehicle data includes sub-data linked to each weight included in each network layer;

[0063] The first determining module is specifically used to: multiply the absolute value of each weight by the second norm of the sub-data linked to that weight to obtain the anomaly score of each weight included in each network layer; calculate the average of the anomaly scores of the multiple weights included in each network layer to obtain the anomaly average of the multiple weights included in each network layer.

[0064] In some embodiments, the first determining module is specifically used to: determine the weights of each network layer that meet a preset condition as abnormal weights, wherein the preset condition is that the abnormal score of the weight is greater than the first hyperparameter multiple of the abnormal average value of the network layer to which the weight belongs; and determine the ratio of the number of abnormal weights in each network layer to the number of weights in the network layer to obtain the abnormal weight ratio of each network layer.

[0065] In some embodiments, the second determining module is specifically used to: determine the sparsity of each network layer based on the negative correlation between sparsity and the proportion of outlier weights, the first value range of sparsity and the average sparsity, and the proportion of outlier weights of the plurality of network layers.

[0066] In some embodiments, the minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

[0067] In some embodiments, the first determining module is specifically configured to: convert calibration vehicle data into calibration text data; and use the calibration text data to determine the abnormal weight ratio of each network layer.

[0068] The inference module is specifically used to: convert current vehicle data into current text data; and process the current text data using the target weights included in each network layer to obtain an autonomous driving strategy.

[0069] In some embodiments, the first determining module is specifically used to: input calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and use the calibration embedded vector to determine the abnormal weight ratio of each network layer.

[0070] The inference module is specifically used to: input the current vehicle data into the vector encoder to obtain the current embedded vector; and process the current embedded vector using the target weights included in each network layer to obtain an autonomous driving strategy.

[0071] In some embodiments, the autonomous driving system further includes an actuator, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning approach; the device further includes:

[0072] The execution module is used to send the strategy planning program to the executor after obtaining the autonomous driving strategy, so that the executor executes the strategy planning program and demonstrates the strategy reasoning logic.

[0073] In some embodiments, the calibration vehicle data and the current vehicle data include at least one of the following: human commands, driving strategy evaluations, system messages, driving scenario data, and historical interaction information.

[0074] Fourthly, embodiments of this application provide a vehicle-mounted large model deployment device, the device comprising:

[0075] The evaluation module is used to evaluate the safety index scores of multiple large models deployed on the first vehicle, and to evaluate the efficiency index scores of multiple large models deployed on the first vehicle.

[0076] The weighting module is used to perform a weighted average of the safety and efficiency scores of each large model to obtain a comprehensive score for each large model.

[0077] The deployment module is used to deploy the largest model with the highest overall score within the autonomous driving system of the first vehicle, the autonomous driving system being used to execute any of the methods provided in the first aspect.

[0078] In some embodiments, the safety performance rating includes a time-of-collision rating;

[0079] The evaluation module is specifically used to: evaluate the collision occurrence time between the first vehicle and each second vehicle corresponding to each large model when executing the driving strategy of the first vehicle using each large model; and determine the collision time score of each large model based on the safe time threshold and the multiple collision occurrence times corresponding to each large model.

[0080] In some embodiments, the evaluation module is specifically used for:

[0081] Determine the minimum collision occurrence time for each large model from among the multiple collision occurrence times corresponding to each large model;

[0082] When the minimum collision occurrence time corresponding to each large model is greater than the safe time threshold, the collision time score of each large model is determined as the maximum score value.

[0083] When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined as the minimum score value.

[0084] When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0085] In some embodiments, the safety metric score includes a velocity variance score;

[0086] The evaluation module is specifically used to: evaluate the speed variance of the first vehicle corresponding to each large model when executing the driving strategy of the first vehicle using each large model; and determine the speed variance score of each large model based on the negative correlation between speed variance and speed variance score, the speed variance corresponding to each large model, and the third value range from the minimum score value to the maximum score value.

[0087] In some embodiments, the evaluation module is specifically used to: calculate the ratio of the velocity variance corresponding to each large model to the preset safe velocity deviation, thereby obtaining the velocity ratio corresponding to each large model; and determine the velocity variance score of each large model based on the negative correlation between the velocity ratio and the velocity variance score, the velocity ratio corresponding to each large model, and the third value range from the minimum score to the maximum score.

[0088] In some embodiments, the efficiency index score includes a time efficiency score;

[0089] The evaluation module is specifically used to: when executing the driving strategy of the first vehicle using each large model, evaluate the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy; and determine the time efficiency score of each large model based on the negative correlation between the consumed time and the time efficiency score, the consumed time corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value.

[0090] In some embodiments, the evaluation module is specifically used to: calculate the ratio of the time consumed by each large model to the preset time limit, and obtain the time ratio for each large model; and determine the time efficiency score for each large model based on the negative correlation between the time ratio and the time efficiency score, the time ratio for each large model, and the fourth value range from the minimum score to the maximum score.

[0091] Fifthly, embodiments of this application provide an autonomous driving system, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0092] Memory, used to store computer programs;

[0093] The processor, when executing a program stored in memory, implements any of the methods provided in the first aspect.

[0094] Sixthly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0095] Memory, used to store computer programs;

[0096] The processor, when executing a program stored in memory, implements any of the methods provided in the second aspect.

[0097] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods provided in the first aspect or any of the methods provided in the second aspect.

[0098] Eighthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods provided in the first aspect, or to perform any of the methods provided in the second aspect.

[0099] Beneficial effects of the embodiments in this application:

[0100] In the technical solution provided in this application, the in-vehicle large model is deployed within the autonomous driving system on the vehicle. The autonomous driving system uses calibrated in-vehicle data to determine the proportion of outlier weights in each network layer of the in-vehicle large model, thereby determining the sparsity of each network layer, such that the sparsity of each network layer is negatively correlated with the proportion of outlier weights in each network layer. The outlier weights in the network layers are strongly correlated with the computational power required for the in-vehicle large model, while the outlier weights have a relatively small impact on the accuracy of the autonomous driving strategy. In this embodiment, the sparsity of each network layer is negatively correlated with the proportion of abnormal weights in each network layer. That is, the larger the proportion of abnormal weights in a network layer, the fewer normal weights (i.e., target weights) participate in the calculation in that network layer. The proportion of target weights and abnormal weights participating in the calculation in each network layer is matched. By using this non-uniform layer-by-layer sparsity method, abnormal weights in each network layer are eliminated, and normal weights in each network layer are used to process real-time vehicle data to obtain autonomous driving strategies. While ensuring the accuracy of autonomous driving strategies, the number of weights participating in the calculation in the large vehicle model is greatly reduced, the overall computing power requirement of the large vehicle model is reduced, and the deployment of large language models in autonomous driving systems is realized when the computing power of vehicle chips is insufficient.

[0101] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0102] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0103] Figure 1 This is a schematic diagram of a first flowchart of an autonomous driving strategy generation method provided in an embodiment of this application;

[0104] Figure 2A schematic diagram of a vector encoder provided in an embodiment of this application;

[0105] Figure 3 A schematic diagram of a vehicle-mounted large model architecture provided in an embodiment of this application;

[0106] Figure 4 for Figure 1 A schematic diagram of step S101;

[0107] Figure 5 A schematic diagram illustrating the determination of abnormal scores provided in an embodiment of this application;

[0108] Figure 6 This is a second flowchart illustrating the autonomous driving strategy generation method provided in an embodiment of this application.

[0109] Figure 7 A schematic diagram of a pruning strategy based on layer-by-layer anomaly distribution provided in an embodiment of this application;

[0110] Figure 8 A schematic flowchart of a vehicle-mounted large model deployment method provided in an embodiment of this application;

[0111] Figure 9 A schematic diagram of the structure of the autonomous driving strategy generation device provided in the embodiments of this application;

[0112] Figure 10 A schematic diagram of a vehicle-mounted large model deployment device provided in an embodiment of this application;

[0113] Figure 11 A schematic diagram of the structure of an autonomous driving system provided in an embodiment of this application;

[0114] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0115] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0116] With the continuous development of autonomous driving technology, the amount of onboard data generated by autonomous vehicles is experiencing explosive growth. Currently, an autonomous vehicle can generate tens of terabytes (TB) of data per day. This data mainly comes from multiple sensors such as LiDAR, millimeter-wave radar, and cameras. For example, a high-resolution LiDAR can generate millions of data points per second, and multiple cameras working simultaneously also generate a large amount of image data. Taking an autonomous vehicle equipped with multiple advanced sensors as an example, it may generate more than 5TB of data during an hour of driving. This massive amount of onboard data is crucial for the training and optimization of autonomous driving systems. By analyzing and processing this onboard data, autonomous driving systems can continuously improve their perception, decision-making, and control capabilities, thereby achieving safer and more efficient driving. However, the explosive growth of onboard data also brings enormous challenges to data storage, transmission, and processing, posing a huge computational challenge to onboard chips.

[0117] As the demand for automotive computing power continues to increase, the computing power of automotive chips is constantly rising. However, the computing power of automotive chips still cannot keep up with the speed at which automotive sensors acquire data, resulting in an awkward situation of mismatch between automotive data and automotive computing power. Following this logic that increasing computing power demand necessitates increasing the computing power of automotive chips, to meet the future computing power requirements of autonomous driving vehicles, the only solution is to continuously increase the computing power of automotive chips and develop high-end chips. This includes adopting new process technologies, new instruction set architectures, new network protocols (Internet Protocol, IP), improving chip yield, and closely following the product roadmap of major computing power companies in terms of technology and products. However, due to various limitations, it is impossible to achieve this technological leap in a short period of time, and the computing power of automotive chips is limited.

[0118] The integration of large language models with autonomous driving systems is indispensable for improving vehicle intelligence and represents a novel approach to enhancing system reasoning, interpretability, and decision-making capabilities. The integration of large language models with autonomous driving systems can be achieved through the following methods:

[0119] The first approach involves calling a pre-trained large language model via an Application Programming Interface (API). This API-based method has demonstrated flexibility in utilizing pre-trained large language models to accomplish various driving tasks.

[0120] This approach relies on cloud server connections to deploy large language models, enabling it to fully leverage state-of-the-art pre-trained large language models for optimal performance. However, the large language models are deployed remotely, and the method's effectiveness is limited by network conditions and server data processing speed, thus restricting its effectiveness in complex driving scenarios.

[0121] The second approach involves training and deploying large language models locally within the vehicle. This method relies on local computing resources to deploy large language models, addressing the requirements of autonomous driving systems for cloud computing resources and network communication quality. However, since large language models can have billions of parameters, this method places high demands on local computing resources, while the computing power of onboard chips is limited, thus restricting its practical application in autonomous vehicles.

[0122] Currently, the integration solutions for large language models and autonomous driving systems include the following:

[0123] (1) Language Model Predictive Control (Language MPC): This method utilizes a large language model to reason about complex scenarios, derives an autonomous driving strategy, and then converts the autonomous driving strategy into control signals through a parameter matrix. This approach demonstrates the potential of large language models to bridge the gap between abstract reasoning and concrete vehicle control.

[0124] (2) Generative Pre-trained Transformer-Driver (GPT-Driver): By converting heterogeneous scene inputs into linguistic tags, motion planning is re-expressed as a natural language modeling task. GPT-Driver is a multimodal large model language framework that takes a series of image frames and natural language (i.e., questions) as input, generates responses to human inquiries, and predicts the control signals for the next step.

[0125] (3) Large Language Model-Driver (LLM-Driver): This method uses a large language model to reason about the scenario, obtains the autonomous driving strategy, and then converts the autonomous driving strategy into control signals through a parameter matrix. This method can perform real-time question answering in the driving environment and enhances the system's ability to explain its decisions to the user.

[0126] (4) Language Model Drive (LMDrive): This system uses a visual encoder to process images and generate visual labels. LMDrive is an end-to-end closed-loop autonomous driving framework designed to combine natural language commands with multimodal sensor data.

[0127] Despite numerous solutions for integrating large language models with autonomous driving systems, the application of large language models for end-to-end autonomous driving computation remains in its early stages. Existing integration solutions focus on using Convolutional Neural Networks (CNNs) to integrate large language models with autonomous driving systems, and this approach has not been fully explored. These solutions rely on cloud server links or local computing resources to deploy large language models, and they face the limitations mentioned above. Furthermore, these methods lead to increased computation time, hindering real-time data processing.

[0128] To address the aforementioned problems, embodiments of this application provide a method for generating autonomous driving strategies, such as... Figure 1 As shown, this method is applied to an autonomous driving system, in which an in-vehicle large model is deployed. The in-vehicle large model includes multiple network layers, and the method includes the following steps.

[0129] Step S101: Using the calibrated vehicle data, determine the proportion of abnormal weights for each network layer;

[0130] Step S102: Determine the sparsity of each network layer based on the abnormal weight ratio of multiple network layers. The sparsity of each network layer is negatively correlated with the abnormal weight ratio.

[0131] Step S103: Process the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy. The ratio of the target weights to all weights included in each network layer is the sparsity of each network layer.

[0132] In the technical solution provided in this application, the in-vehicle large model is deployed within the autonomous driving system on the vehicle. The autonomous driving system uses calibrated in-vehicle data to determine the proportion of outlier weights in each network layer of the in-vehicle large model, thereby determining the sparsity of each network layer, such that the sparsity of each network layer is negatively correlated with the proportion of outlier weights in each network layer. The outlier weights in the network layers are strongly correlated with the computational power required for the in-vehicle large model, while the outlier weights have a relatively small impact on the accuracy of the autonomous driving strategy. In this embodiment, the sparsity of each network layer is negatively correlated with the proportion of abnormal weights in each network layer. That is, the larger the proportion of abnormal weights in a network layer, the fewer normal weights (i.e., target weights) participate in the calculation in that network layer. The proportion of target weights and abnormal weights participating in the calculation in each network layer is matched. By using this non-uniform layer-by-layer sparsity method, abnormal weights in each network layer are eliminated, and normal weights in each network layer are used to process real-time vehicle data to obtain autonomous driving strategies. While ensuring the accuracy of autonomous driving strategies, the number of weights participating in the calculation in the large vehicle model is greatly reduced, the overall computing power requirement of the large vehicle model is reduced, and the deployment of large language models in autonomous driving systems is realized when the computing power of vehicle chips is insufficient.

[0133] In this embodiment, the autonomous driving system is a system that enables autonomous driving decisions on a vehicle, and a vehicle equipped with an autonomous driving system can be called an autonomous driving vehicle. The on-board large model is a large language model deployed within the autonomous driving system; that is, the on-board large model is a large language model deployed on the vehicle. The number of network layers included in the on-board large model can be set according to actual needs.

[0134] In step S101 above, the vehicle-mounted data being calibrated refers to the vehicle-mounted data used to calibrate the sparsity of multiple network layers within the large vehicle-mounted model. The vehicle-mounted data being calibrated may include, but is not limited to, human commands, driving strategy evaluations, system messages, driving scenario data, and historical interaction information.

[0135] Human commands are natural language input directly from the user to the onboard model, and can include the user's desired requests to the autonomous driving system. For example, a user could input the command "overtake the vehicle in front" into the onboard model.

[0136] Driving strategy evaluation is the user's feedback on the effectiveness of the autonomous driving strategy represented by the onboard big data model. Specifically, after the onboard big data model processes the user's input of human commands and other onboard data and outputs an autonomous driving strategy, the user inputs a driving strategy evaluation into the onboard big data model based on this output. For example, the onboard big data model might output an autonomous driving strategy of "No vehicles ahead, proceed at an acceleration of 10 m / s²". 2Increase vehicle speed to 80km / h; based on this autonomous driving strategy, users can input a driving strategy evaluation into the onboard model: "The autonomous driving strategy scores 4 points".

[0137] System messages provide users and autonomous driving systems with pairs of instructions or context. They are a high-level set of guidelines or rules that outline how the onboard model should operate and make decisions. These high-level system messages define the basic driving logic, encompassing task definitions, traffic rules to be followed, descriptions of decision states, and overall goals or metrics to be optimized. As a fundamental framework, system messages guide the behavior and decision-making processes of autonomous vehicles on the road. Without well-designed system messages, the onboard model may make incorrect assumptions or adopt unintended strategies.

[0138] Driving scenario data describes driving scenarios and may include, but is not limited to, route information, vehicle information, pedestrian information, and environmental information of the vehicle itself. Raw driving scenario data can include images, videos, analog signals, digital signals, and other types of data. Vehicle sensors can convert this raw driving scenario data into textual descriptions, i.e., driving scenario data. These descriptions follow a predefined structure and use natural language to provide detailed textual descriptions of the current driving scenario, such as descriptive statements like "You are in the far left lane of a two-way highway" or "There is a vehicle 50 meters ahead of your current position." This textual description intuitively converts the complex spatial and temporal relationships between different road users and the vehicle into natural language format, enabling the onboard large-scale model to reason about driving scenario information. The purpose of driving scenario data is to provide a comprehensive representation of the scenario for the onboard large-scale model, enabling it to make appropriate decisions under current traffic conditions.

[0139] Historical interaction information refers to the historical interaction data between the user and the autonomous driving system. The autonomous driving system can be configured with a memory module that stores profiles for different users to improve the driving experience for all users. Each time a user uses the autonomous driving system, the system records the interaction between that user and the system. After the trip, the system updates the user's historical interaction data into the corresponding profile in the memory module. The specific content of the historical interaction data varies depending on the autonomous driving system. This historical interaction data can serve as a reference for user preferences, input into the onboard model, and guide the autonomous driving system to improve the user experience.

[0140] In this embodiment, the anomaly weight ratio refers to the proportion of anomalies. The autonomous driving system can use events or periodic triggers to execute step S101, acquire calibration vehicle data, and use this data to determine the anomaly weight ratio of each network layer. Thus, the autonomous driving system can obtain a layer-by-layer anomaly distribution. The layer-by-layer anomaly distribution includes the anomaly weight ratios of multiple network layers, for example, a layer-by-layer anomaly distribution ROI = [D1, D2, ..., D...]. n ], D m This represents the proportion of abnormal weights in network layer m, where m = 1, 2, ..., n, and n represents the number of network layers included in the large vehicle model.

[0141] For example, when an autonomous driving system acquires human quality and needs to perform a driving task, the autonomous driving system executes step S101 to obtain the abnormal weight ratio and sparsity adapted to the current driving task, thereby improving the accuracy of the autonomous driving strategy.

[0142] For example, the autonomous driving system periodically executes step S101. In this way, when performing a driving task, step S103 can be executed directly using the last obtained anomaly weight ratio and sparsity, thus improving the generation efficiency of the autonomous driving strategy.

[0143] In step S102 above, the sparsity of the network layer represents the proportion of weights involved in the calculation in the network layer, which can be expressed by the following formula (1).

[0144] P m =λ 1m / λ 2m (1)

[0145] In formula (1), P m λ represents the sparsity of network layer m. 1m The number of weights involved in the calculation in network layer m is represented by / λ. 2m This represents the total number of weights in network layer m.

[0146] The sparsity of a network layer is negatively correlated with the proportion of outlier weights. In one example, the negative correlation between the sparsity of a network layer and the proportion of outlier weights can be expressed by the following formula (2) or formula (3).

[0147] P m =k1 / D m (2)

[0148] P m =k2(1-D m (3)

[0149] In formulas (2) and (3), P m D represents the sparsity of network layer m. mThis represents the proportion of abnormal weights in network layer m, and k1 and k2 represent negative correlation hyperparameters. The values ​​of k1 and k2 can be set according to actual needs.

[0150] The negative correlation between the sparsity of network layers and the proportion of outlier weights can also be represented in other ways, without limitation.

[0151] In this embodiment of the application, for each network layer included in the vehicle-mounted large model, the autonomous driving system can determine the sparsity of the network layer based on the negative correlation between sparsity and the proportion of abnormal weights, as well as the proportion of abnormal weights of the network layer.

[0152] In step S103 above, the target weights are the weights used in the calculation. The number of target weights in different network layers can be the same or different. Current vehicle data refers to the vehicle data collected in real time during the execution of the current driving task. Current vehicle data may include, but is not limited to, human commands, driving strategy evaluations, system messages, driving scenario data, and historical interaction information.

[0153] The autonomous driving system acquires the current vehicle data and inputs it into the vehicle large model. The vehicle large model is pruned according to the sparsity of each network layer to obtain the target weights included in each network layer. Then, the target weights included in each network layer are used to perform inference processing on the current vehicle data to obtain the autonomous driving strategy.

[0154] In this embodiment, the autonomous driving system eliminates abnormal weights (i.e. unnecessary weights) in each network layer according to the sparsity of each network layer. This is equivalent to removing the low-rank adaptive module. This reduces the size and computational load of the vehicle-mounted large model while maintaining its performance, improving its accuracy and utilization, and reducing computational lag.

[0155] In some embodiments, step S101 may be: converting the calibration vehicle data into calibration text data; using the calibration text data, determining the anomaly weight ratio of each network layer. Step S103 may be: converting the current vehicle data into current text data; using the target weights included in each network layer, processing the current text data to obtain an autonomous driving strategy.

[0156] The in-vehicle large model is a large language model, and its input data is general text data. This means that the layer-by-layer anomaly distribution ROI is calculated using general text data. However, in-vehicle data includes various types of data such as images, videos, analog signals, and digital signals. In other words, in the context of autonomous driving, in-vehicle data consists of vector representations of the driving scene. The significant differences between in-vehicle data and general text data can lead to vastly different calculation results for the layer-by-layer anomaly distribution ROI, potentially affecting the pruning results of the in-vehicle large model.

[0157] In this embodiment, the autonomous driving system converts the calibrated onboard data into text data (i.e., calibrated text data) and the current onboard data into text data (i.e., current text data). The autonomous driving system uses the calibrated text data to calculate the layer-by-layer anomaly distribution ROI, and then uses the layer-by-layer anomaly distribution ROI to prune the large onboard model, and then performs inference on the current text data. This ensures that the pruned large onboard model is more compatible with the downstream autonomous driving task, and ensures that the pruning process can be customized for the specific needs and characteristics of the autonomous driving task, resulting in a more effective pruning effect.

[0158] In this embodiment, when the vehicle-mounted large model supports vector representation of driving scenarios, the autonomous driving system directly uses calibrated vehicle-mounted data to calculate the layer-by-layer anomaly distribution ROI, and then uses the layer-by-layer anomaly distribution ROI to prune the vehicle-mounted large model, and then directly performs inference processing on the current vehicle-mounted data, without having to convert the calibrated vehicle-mounted data and the current vehicle-mounted data into text data, thus improving the generation efficiency of autonomous driving strategies.

[0159] In some embodiments, step S101 may be: inputting calibration vehicle data into a vector encoder to obtain a calibration embedded vector; using the calibration embedded vector to determine the anomaly weight ratio of each network layer. Step S103 may be: inputting current vehicle data into a vector encoder to obtain a current embedded vector; using the target weights included in each network layer to process the current embedded vector to obtain an autonomous driving strategy.

[0160] In this embodiment, the autonomous driving system can also deploy a vector encoder, the structure of which can be configured according to actual needs. In one example, the vector encoder can employ... Figure 2 The structure shown includes multiple multilayer perceptrons (MLPs), multiple MLPs connected to a cross-attention layer, the cross-attention layer connected to seven self-attention layers, and the seven self-attention layers connected to the cross-attention layer.

[0161] based on Figure 2The vector encoder shown allows the autonomous driving system to convert onboard data such as route information, vehicle information, pedestrian information, and environmental information into tokens, which are then input into the vector encoder. In the vector encoder, each type of token is processed by a separate MLP. Each token (i.e., a vector) processed by the MLP is then processed through a cross-attention layer and projected into a latent space, resulting in multiple latent vectors. Vehicle features are added to each latent vector, representing the vehicle's state. Subsequently, seven self-attention layers and one cross-attention layer convert the vehicle state into embedded vectors compatible with the onboard module. After the vector encoder outputs the embedded vectors, the autonomous driving system inserts them into a text prompt embedding. This text prompt is then input into the onboard model for inference processing, or the autonomous driving system uses the embedded vectors to calculate layer-by-layer anomaly distribution ROIs.

[0162] In this embodiment, the vector encoder is more sensitive to other components of the vehicle-mounted large model. By using the vector encoder to process vehicle data, the computational requirements are effectively reduced, enabling the vehicle-mounted large model to exhibit better perception, action prediction, and language understanding capabilities.

[0163] In some embodiments, step S101 may be: converting calibration vehicle data into calibration text data; inputting the calibration text data into a vector encoder to obtain a calibration embedded vector; and using the calibration embedded vector to determine the anomaly weight ratio of each network layer. Step S103 may be: converting current vehicle data into current text data; inputting the current text data into a vector encoder to obtain a current embedded vector; and processing the current embedded vector using the target weights included in each network layer to obtain an autonomous driving strategy.

[0164] In this embodiment, the autonomous driving system converts the calibration vehicle data into calibration text data, and then converts the calibration text data into tokens before inputting them into the vector encoder.

[0165] In this embodiment, the autonomous driving system converts calibrated onboard data and current onboard data. The system uses the calibrated text data to calculate the layer-by-layer anomaly distribution ROI, and then uses this ROI to prune the large onboard model. This pruning is then used to infer the current text data, ensuring that the pruned onboard model is more aligned with the downstream autonomous driving task. The pruning process is customized to meet the specific needs and characteristics of the autonomous driving task, resulting in a more effective pruning effect. Furthermore, vector encoders have higher sensitivity. Using vector encoders to process the text data corresponding to the onboard data effectively reduces computational requirements while enabling the large onboard model to exhibit superior perception, action prediction, and language understanding capabilities.

[0166] In this embodiment, actuators may also be deployed within the autonomous driving system. These actuators may include, but are not limited to, cockpit systems, intelligent driving systems, and chassis systems. The actuators can act as a bridge between the text output by the onboard large model and the current autonomous driving strategy. The autonomous driving strategy may include a strategy planning program, which is code that implements the autonomous driving strategy.

[0167] After obtaining the autonomous driving strategy (i.e., the strategy planner), the autonomous driving system sends the strategy planner to the actuator. The actuator executes the strategy planner, enabling the code to interact with the current state of the vehicle, allowing the vehicle to perform the intended driving behavior in a real or simulated environment.

[0168] In this embodiment, a display may also be deployed within the autonomous driving system, or the autonomous driving system may be connected to a device with a display. The autonomous driving strategy may also include a strategy reasoning approach, which is the thought process that derives the strategy planning procedure. This strategy reasoning approach helps the onboard large model generate a more reasonable autonomous driving strategy.

[0169] After obtaining the autonomous driving strategy (i.e., the strategy planning program and the strategy reasoning logic), the autonomous driving system sends the strategy planning program to the executor. The executor executes the strategy planning program, enabling the code to interact with the current state of the vehicle, allowing the vehicle to perform the expected driving behavior in a real or simulated environment. Additionally, the autonomous driving system sends the strategy reasoning logic to the display, presenting it in text form. This helps the user understand the obtained autonomous driving strategy, allowing for its evaluation. The evaluation is then input into the onboard large-scale model to optimize the autonomous driving strategy.

[0170] The following is combined with Figure 3 The vehicle-mounted large model architecture shown herein provides a detailed explanation of the autonomous driving strategy generation method provided in this application embodiment.

[0171] The autonomous driving system retrieves historical interaction information from the memory module and combines it with human commands, driving strategy evaluations, system messages, and driving scenario data, inputting all of this into the onboard large model. The onboard large model, comprising multiple network layers, processes the input data for inference, outputting a policy planning program and policy reasoning logic. The actuator executes the policy planning program, enabling the vehicle to perform the intended driving behaviors in real-world environments or simulated environments such as Navigate on Autopilot (NOA) and Car Learning to Act (CARLA).

[0172] Meanwhile, the autonomous driving system records historical interaction information between the user and the autonomous driving system. When the trip (i.e. the driving task) ends, the autonomous driving system can update the historical interaction information into the memory module.

[0173] In some embodiments, such as Figure 4 As shown, step S101 above may include:

[0174] Step S1011: Using the calibrated vehicle data, determine the outlier score of each weight in each network layer, and the outlier average of multiple weights in each network layer.

[0175] In this embodiment, for each weight included in each network layer, the autonomous driving system can use calibrated onboard data to determine the anomaly score of that weight. The anomaly score of a weight indicates the degree of anomaly of the weight; the higher the anomaly score, the higher the degree of anomaly of the weight.

[0176] In some embodiments, calibrating onboard data includes sub-data linked to each weight included in each network layer. The autonomous driving system can obtain an anomaly score for each weight included in each network layer by multiplying the absolute value of each weight by the second norm of the sub-data linked to that weight; and calculate the average of the anomaly scores of the multiple weights included in each network layer to obtain the anomaly average of the multiple weights included in each network layer.

[0177] For example, for a large vehicle model with n network layers, the autonomous driving system can use the following formula (4) to determine the anomaly score of each weight included in each network layer.

[0178]

[0179] In formula (4), This indicates the weights included in network layer m. Abnormal scores, X j Indicates the link to weight All sub-data (i.e., input features), Let ||||2| represent the weights in the i-th row and j-th column of network layer m, where ||||2 represents the second norm and || represents the absolute value. For example... Figure 5 The method for determining outlier scores shown includes the weights w in network layer m. 11 ~w 33 Calibrate onboard data including X 11 X 21 X 31 X 41 Wait for the link to w 11 w 21 w 31 Input features X1, X 12 X22 X 32 X 42 Wait for the link to w 12 w 22 w 32 Input features X2, X 13 X 23 X 33 X 43 Wait for the link to w 13 w 23 w 33 The input feature X3. With input features 1 and w 11 For example, weight w 31 The abnormal score is:

[0180] The autonomous driving system can also use other methods to determine the anomaly score of each weight included in each network layer. For example, for a large vehicle model with n network layers, the autonomous driving system can use the following formula (5) to determine the anomaly score of each weight included in each network layer.

[0181]

[0182] In formula (5), This indicates the weights included in network layer m. Abnormal scores, X j Indicates the link to weight All sub-data (i.e., input features), Let |||| represent the weights in the i-th row and j-th column of network layer m, |||| represent the first-order norm, and || represent the absolute value.

[0183] In this embodiment of the application, the method for determining the abnormal score of the weight is not limited, as long as the larger the abnormal score is, the higher the degree of abnormality of the weight.

[0184] Step S1012: Determine the proportion of abnormal weights for each network layer based on the abnormal scores and average values ​​of the multiple weights included in each network layer.

[0185] Having obtained the abnormal scores of multiple weights included in each network layer, as well as the average abnormal score of each network layer, the autonomous driving system can compare the abnormal score of each weight in each network layer with the average abnormal score of that network layer to determine whether the weight is an abnormal weight and to determine the proportion of abnormal weights in that network layer. This is known as the abnormal weight proportion.

[0186] In some embodiments, the autonomous driving system can determine the weights of each network layer that meet preset conditions as abnormal weights. The preset condition is that the abnormal score of the weight is greater than the average abnormal value of the network layer to which the weight belongs by a first hyperparameter multiple. The system can also determine the ratio of the number of abnormal weights in each network layer to the total number of weights in that network layer to obtain the abnormal weight ratio of each network layer.

[0187] In this embodiment, the autonomous driving system is pre-configured with a first hyperparameter, which is a hyperparameter used to determine anomaly thresholds. The preset conditions are: in, This indicates the weights included in network layer m. Abnormal scores, This represents the outlier average value of network layer m, where M represents the first hyperparameter. The value of M can be set according to actual needs.

[0188] For each network layer, the autonomous driving system can determine the network layer that includes, and is greater than, The weights are used as abnormal weights, and then the first number of abnormal weights included in the network layer and the total number of weights included in the network layer (such as the second number) are determined. The ratio of the first number to the second number is calculated to obtain the abnormal weight ratio of each network layer.

[0189] In this embodiment of the application, the abnormal weight ratio of each network layer can be determined by the following formula (6).

[0190]

[0191] In formula (6), D m This represents the proportion of outlier weights in network layer m. This indicates the weights included in network layer m. Abnormal scores, E represents the outlier average of network layer m, where M represents the first hyperparameter, and E represents the first hyperparameter. in E represents the number of dimensions of the input layer, that is, the number of rows in the weight matrix of network layer m. out This represents the number of dimensions of the output layer, i.e., the number of columns in the weight matrix of network layer m. I() represents an indicator function; it returns 1 if a preset condition is met, and 0 otherwise. Based on this, the layer-by-layer anomaly distribution ROI = [D1, D2, ..., D...] can be obtained. n ],like Figure 5 As shown.

[0192] In this embodiment, the autonomous driving system can also use other methods to determine the abnormal weight ratio of each network layer. There is no limitation on this method, as long as the larger the abnormal weight ratio is, the more abnormal weights there are in the network layer. For example, the autonomous driving system can determine the weights that meet the preset conditions included in each network layer as abnormal weights, and determine the ratio of the number of abnormal weights included in each network layer to the total number of weights included in all network layers to obtain the abnormal weight ratio of each network layer. Specifically, it can be expressed by the following formula (7).

[0193]

[0194] In formula (7), D m This represents the proportion of outlier weights in network layer m. This indicates the weights included in network layer m. Abnormal scores, The value represents the outlier average of network layer m, M represents the first hyperparameter, H represents the total number of weights in network layers 1 to n, and I() represents the indicator function. I returns 1 when a preset condition is met, otherwise I returns 0.

[0195] In this embodiment, the autonomous driving system quantifies the anomaly distribution between each network layer based on layer-by-layer anomaly distribution pruning, and obtains the layer-by-layer anomaly distribution ROI = [D1, D2, ..., D]. n By quantifying the anomaly distribution of each network layer, the autonomous driving system can match the sparsity of each network layer with the proportion of its anomaly weights, thereby ensuring the preservation of key features that are crucial to maintaining model performance.

[0196] In some embodiments, such as Figure 6 As shown, an autonomous driving strategy generation method is also provided, which is applied to an autonomous driving system. The autonomous driving system deploys an on-board large model, which includes multiple network layers. The method may include the following steps.

[0197] Step S601: Using the calibrated vehicle data, determine the abnormal weight ratio of each network layer; the same as step S101 above.

[0198] Step S602: Based on the negative correlation between sparsity and the ratio of outlier weights, the first range of sparsity values ​​and the average sparsity, and the ratio of outlier weights of the multiple network layers, determine the sparsity of each network layer, wherein the sparsity of each network layer is negatively correlated with the ratio of outlier weights.

[0199] In this embodiment, the negative correlation between sparsity and the proportion of outlier weights can be expressed by the above formula (2) or formula (3). The average sparsity is the average sparsity of all network layers included in the vehicle-mounted large model, and it is also the target sparsity of the vehicle-mounted large model. The first value range is the range of sparsity values, which can be expressed as [P 小 P 大 By using a first value range, excessive sparsity differences between network layers can be effectively prevented. In one example, an autonomous driving system could introduce a hyperparameter, such as a second hyperparameter δ, where the minimum value P within the first value range is... 小 P is the difference between the average sparsity and the second hyperparameter, and the maximum value P in the first range. 大 The first value range is the sum of the average sparsity and the second hyperparameter, which can be expressed as [P-δ, P+δ], where P is the average sparsity. The first value range can also be represented in other ways, without limitation.

[0200] In this embodiment, the autonomous driving system can adjust the negative correlation hyperparameters (such as k1 and k2) so that when calculating the sparsity of each network layer according to the negative correlation between sparsity and the ratio of outlier weights and the ratio of outlier weights for each network layer, the minimum sparsity calculated is greater than or equal to P. 小 The calculated maximum sparsity is less than or equal to P. 大 In addition, the autonomous driving system can adjust the sparsity of each calculated network layer so that the average sparsity of all network layers is equal to the average sparsity.

[0201] In this embodiment of the application, the autonomous driving system can also directly calculate the sparsity of each network layer based on the negative correlation between sparsity and the proportion of outlier weights, and the proportion of outlier weights for each network layer; if the calculated sparsity of a network layer is less than P... 小 Then the sparsity of the network layer is set to P. 小 If the sparsity of a network layer is greater than P 大 Then the sparsity of the network layer is set to P. 大 If the sparsity of a calculated network layer is less than P 小 Then the sparsity of the network layer is set to P. 小 If the calculated sparsity of a network layer is greater than or equal to P 小 And the sparsity of this network layer is less than or equal to P. 大 If the sparsity of the network layer remains unchanged, then the autonomous driving system can adjust the calculated sparsity of each network layer so that the average sparsity of all network layers is equal to the average sparsity.

[0202] In this embodiment of the application, the autonomous driving system may also use other methods to determine the sparsity of each network layer, and there is no limitation on this.

[0203] Step S603: Using the target weights included in each network layer, process the current vehicle data to obtain an autonomous driving strategy. The ratio of the target weights to all weights in each network layer represents the sparsity of each network layer. This is the same as step S103 above.

[0204] In the technical solution provided in this application embodiment, the autonomous driving system determines the sparsity of each network layer based on the negative correlation between sparsity and the proportion of outlier weights, the first range of sparsity values, and the average sparsity. This ensures that the sparsity of each network layer is within the first range, and the average sparsity of all network layers is maintained at the average sparsity. This prevents excessive sparsity differences between network layers and avoids the average sparsity of all network layers being too large or too small. It also avoids over-pruning of the large vehicle model, which would affect the accuracy of the large vehicle model, and also avoids under-pruning of the large vehicle model, which would lead to excessive computational load and cause computational lag.

[0205] The following is combined with Figure 2 The vector encoder shown and Figure 7 The pruning strategy based on layer-by-layer anomaly distribution shown in the illustration provides a detailed description of the autonomous driving strategy generation method provided in this application embodiment.

[0206] The autonomous driving system can convert calibration onboard data, such as route information, vehicle information, pedestrian information, and environmental information, into tokens, which are then input into the vector encoder. The autonomous driving system inputs the embedded vectors output from the vector encoder and activates a pruning strategy based on layer-by-layer anomaly distribution. This pruning strategy uses the embedded vectors to determine the layer-by-layer anomaly distribution ROI. The process of determining the layer-by-layer anomaly distribution ROI using the pruning strategy is described above. Figure 5 and Figure 6 Partial description.

[0207] After obtaining the layer-by-layer anomaly distribution ROI, the autonomous driving system can convert calibration onboard data such as route information, vehicle information, pedestrian information, and the vehicle's environmental information into tokens, which are then input into the vector encoder. The autonomous driving system inserts the embedded vector output by the vector encoder into a text prompt embedding, and then inputs this text prompt into the onboard big model for inference processing. The onboard big model outputs a strategy planning program and a strategy inference logic, and sends the strategy planning program to actuators such as the cockpit system, intelligent driving system, and chassis system to implement the corresponding driving behaviors.

[0208] This application proposes a pruning strategy based on layer-by-layer anomaly distribution, a highly efficient autonomous driving framework strategy. This strategy employs a weighted layer-by-layer pruning method based on outliers (i.e., outlier weights), utilizing the target weights in each network layer for inference, thus compressing the data input volume of large-scale in-vehicle models without fine-tuning. By allocating non-uniform sparsity based on layer-by-layer anomaly distribution and incorporating in-vehicle data from the driving environment into the calibration and pruning process, this method significantly reduces the computational requirements of large-scale in-vehicle models while maintaining their performance. Specifically, it offers the following advantages:

[0209] (1) This application proposes an autonomous driving framework based on a large language model pruning strategy. This autonomous driving framework uses layer-by-layer sparsity based on outlier weighting to compress the input data of the vehicle-mounted large model, which can solve a series of problems such as the computational complexity of the vehicle-mounted large model and insufficient hardware computing power.

[0210] (2) The embodiments of this application utilize the model pruning method, which can remove unnecessary weights (i.e. abnormal weights) for autonomous driving. In other words, abnormal weights do not participate in the calculation, but can maintain model performance, improve the accuracy of the vehicle-mounted large model, reduce model size, improve model utilization, and reduce the phenomenon of vehicle-mounted large model calculation lag.

[0211] (3) The embodiments of this application effectively solve the computational limitation problem of large language models in vehicles by adopting layer-by-layer sparsity weighted by outlier calculation. By processing complex environmental data and natural language input from multiple sensing modalities and reasoning about them, autonomous vehicles can make more complex and efficient decisions.

[0212] (4) The embodiments of this application use a vector encoder in the autonomous driving system. Compared with other large model components, the vector encoder has higher sensitivity. By using a vector encoder, the computational requirements can be effectively reduced while demonstrating better perception, action prediction and language understanding capabilities.

[0213] (5) The embodiments of this application combine pruning technology with large language patterns, which is expected to develop an efficient and stable autonomous driving and cockpit system that can handle complex scenarios.

[0214] This application also provides a method for deploying a large vehicle-mounted model, such as... Figure 8 As shown, the method includes the following steps:

[0215] Step S801: Evaluate the safety index score when multiple large models are deployed on the first vehicle, and evaluate the efficiency index score when multiple large models are deployed on the first vehicle.

[0216] Step S802: Calculate the weighted average of the safety index score and efficiency index score of each large model to obtain the comprehensive score of each large model.

[0217] Step S803: Deploy the large model with the highest comprehensive score into the autonomous driving system of the first vehicle. The autonomous driving system is used to execute the above-mentioned autonomous driving strategy generation method.

[0218] In the technical solution provided in this application, each large model is evaluated using safety and efficiency indicators, and the large model with the highest score is selected for installation on the vehicle. This method of determining the onboard large model reduces the amount of data input required and effectively mitigates some uncertainties during the installation process, thereby improving the performance of the onboard large model in autonomous driving tasks.

[0219] In this embodiment, the vehicle-mounted large model deployment method can be applied to servers, clusters, and other devices. The following description uses electronic devices as the execution subject and is not intended to be limiting. When executing the vehicle-mounted large model deployment method, the first vehicle to which the vehicle-mounted large model (i.e., the large language model) needs to be deployed is located in an autonomous driving simulation environment. This autonomous driving simulation environment may also include n other vehicles (such as a second vehicle), where n is greater than or equal to 1; that is, the autonomous driving simulation environment includes n+1 vehicles.

[0220] In step S801 above, safety metrics are used to measure the vehicle's ability to maintain a safe distance and avoid collisions. Safety metrics may include, but are not limited to, Time to Collision (TTC) and Speed ​​Variance (SV). Efficiency metrics are used to evaluate the autonomous driving strategy within the predetermined time limit T. limit The ability to complete autonomous driving tasks within a short timeframe, with efficiency metrics including but not limited to time efficiency (TE).

[0221] The electronic device is configured with evaluation methods for both safety and efficiency indicators. For each large model, the electronic device evaluates the safety indicator score (i.e., safety indicator score) when the large model is deployed on the first vehicle using the safety indicator evaluation method, and evaluates the efficiency indicator score (i.e., efficiency indicator score) when the large model is deployed on the first vehicle using the efficiency indicator evaluation method.

[0222] The evaluation methods for TTC, SV, and TE are explained below.

[0223] (1)TTC.

[0224] When the safety index includes collision time, that is, when the safety index score includes collision time score, the step S801 above, which evaluates the safety index score when multiple large models are deployed on the first vehicle, can be: when executing the driving strategy of the first vehicle using each large model, evaluate the collision occurrence time between the first vehicle and each second vehicle corresponding to each large model; and determine the collision time score of each large model based on the safety time threshold and the multiple collision occurrence times corresponding to each large model.

[0225] In this embodiment, the safety time threshold T safe The size can be set according to actual needs, for example, T safe It can be 2 seconds, 3 seconds, etc.

[0226] For each large model, when the large model is deployed on the first vehicle and the driving strategy of the first vehicle is executed using the large model, the electronic device can use the following formula (8) to determine the collision time between the first vehicle and each second vehicle.

[0227]

[0228] In formula (8), Tc i P represents the time of the collision between the first vehicle and the second vehicle i, P0 represents the position of the first vehicle, and P... i v represents the position of the second vehicle i, v0 represents the longitudinal velocity of the first vehicle, v i Let i represent the longitudinal velocity of the second vehicle i, where i = 1, ..., n.

[0229] Within all time steps t of executing the driving strategy of the first vehicle using this large model, the electronic device can calculate the collision time Tc between the first vehicle and each of the second vehicles according to formula (8). i In other words, for each second vehicle i, at each time step t, the electronic device can calculate a Tc. i .

[0230] After the driving strategy of the first vehicle is executed using the large model, the electronic device can determine the collision time score of the large model based on the safety time threshold and the multiple collision occurrence times corresponding to the large model.

[0231] In one example, the electronic device can determine the minimum collision occurrence time for each large model from multiple collision occurrence times corresponding to each large model; when the minimum collision occurrence time for each large model is greater than the safety time threshold, the collision time score for each large model is determined as the maximum score value; when the minimum collision occurrence time for each large model is less than or equal to 0, the collision time score for each large model is determined as the minimum score value; when the minimum collision occurrence time for each large model is less than or equal to the safety time threshold and the minimum collision occurrence time for each large model is greater than 0, the collision time score for each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time for each large model, and the second value range from the minimum score value to the maximum score value.

[0232] In this embodiment, the second value range is the range of values ​​for the collision time score. For example, a safe time threshold of 2 seconds, a maximum score of 100, and a minimum score of 0 are used. For each large model, the electronic device can use the following formulas (9) and (10) to determine the collision time score of that large model.

[0233]

[0234] In formulas (9) and (10), TTC is the collision time score of the large model, and T min This represents the minimum collision time corresponding to the large model; n represents the number of second vehicles. This indicates the time when the collision between the first vehicle and the second vehicle i occurs at time step t.

[0235] In this embodiment, if the collision time between the first vehicle and a second vehicle is greater than or equal to 0, it indicates that a collision between the first vehicle and the second vehicle is possible; if the collision time is less than 0, it indicates that a collision between the first vehicle and the second vehicle is impossible. The electronic device can determine the minimum collision time T corresponding to the large model from the collision times greater than or equal to 0 corresponding to the large model. min .

[0236] In formula (10), only (maximum score - 1 / T) min The expression ) represents the positive correlation between minimum collision time and safety score, but it is not limiting. For example, the positive correlation between minimum collision time and safety score can also be expressed as (T) min / Safety time threshold) × maximum score value, such as (T min / 2)×100.

[0237] In this embodiment, the electronic device controls the collision time score of each large model between the minimum and maximum score values, avoiding excessively large or small collision time scores for large models, thereby improving the accuracy of the safety index scores for large models and enhancing the accuracy of the evaluation of large models.

[0238] (2)SV.

[0239] When the safety index includes speed variance, that is, when the safety index score includes speed variance score, the step S801 above, which evaluates the safety index score when multiple large models are deployed on the first vehicle, can be: when executing the driving strategy of the first vehicle using each large model, evaluate the speed variance of the first vehicle corresponding to each large model; based on the negative correlation between speed variance and speed variance score, the speed variance corresponding to each large model, and the third value range from the minimum score value to the maximum score value, determine the speed variance score of each large model.

[0240] In this embodiment, the third value range is the value range of the velocity variance score. The third value range can be the same as the second value range described above, to unify the benchmarks for velocity variance score and collision time score, facilitating the management of evaluation criteria for large models. The third value range can also be different from the second value range described above. For example, the second and third value ranges can be divided according to the importance of the velocity variance score and collision time score benchmarks. For instance, if the importance of the velocity variance score is higher than that of the collision time score, the second value range is 0–100, and the third value range is 0–200, facilitating accurate evaluation of large models.

[0241] For each large model, when the large model is deployed on the first vehicle and the driving strategy of the first vehicle is executed using the large model, the electronic device can use the following formula (11) to determine the speed variance of the first vehicle corresponding to the large model.

[0242]

[0243] In formula (11), S0 represents the speed variance of the first vehicle corresponding to the large model, T represents the time required to complete the driving strategy of the first vehicle using the large model, t represents the time step before the driving strategy of the first vehicle using the large model is completed, v0 represents the longitudinal speed of the first vehicle, δ0 represents the average longitudinal speed of the first vehicle corresponding to the large model, and δ0 can be determined by the following formula (12).

[0244]

[0245] In formula (12), δ0 represents the average longitudinal speed of the first vehicle corresponding to the large model, T represents the time required to complete the driving strategy of the first vehicle using the large model, t represents the time step before the driving strategy of the first vehicle using the large model is completed, and v0 represents the longitudinal speed of the first vehicle.

[0246] After the driving strategy of the first vehicle is executed using the large model, the electronic device can determine the speed variance score of the large model based on the negative correlation between the speed variance and the speed variance score, the third value range, and the speed variance corresponding to the large model.

[0247] In one example, the electronic device can calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation, thereby obtaining the speed ratio corresponding to each large model; based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and the third range of values ​​from the minimum score to the maximum score, the speed variance score of each large model is determined.

[0248] In this embodiment of the application, a maximum score of 100, a minimum score of 0, and a third value range of 0 to 100 are used as examples. For each large model, the electronic device can use the following formula (13) to determine the velocity variance score of the large model.

[0249]

[0250] In formula (13), SV is the speed variance score of the large model, and S0 represents the speed variance of the first vehicle corresponding to the large model; S safe This indicates the preset safe speed deviation. The preset safe speed deviation can be set according to actual needs. In formula (13), only the maximum score value × (1-S0 / S) is used. safe The ) indicates the negative correlation between the speed ratio and the speed variance score, that is, the negative correlation between speed variance and speed variance score, and does not serve as a limitation.

[0251] For example, the negative correlation between velocity variance and velocity variance score can also be expressed as:

[0252]

[0253] At this point, taking a maximum score of 100 as an example, the velocity variance score of the large model can also be determined using the following formula (14).

[0254]

[0255] In formula (14), SV is the speed variance score of the large model, S0 represents the speed variance of the first vehicle, and S safeThis indicates the preset safe speed deviation.

[0256] In this embodiment, the electronic device controls the velocity variance score of each large model between the minimum and maximum score values, avoiding excessively large or small velocity variance scores for the large models, thereby improving the accuracy of the safety index scores for the large models and enhancing the accuracy of the evaluation of the large models.

[0257] (3)TE.

[0258] When the efficiency index includes time efficiency, that is, when the efficiency index score includes time efficiency score, the step of evaluating the efficiency index score when multiple large models are deployed on the first vehicle in step S801 above can be: when executing the driving strategy of the first vehicle using each large model, evaluate the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy; based on the negative correlation between the consumed time and the time efficiency score, the consumed time corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value, determine the time efficiency score of each large model.

[0259] In this embodiment, the fourth value range is the value range of the time efficiency score. The fourth value range can be the same as the second value range described above, to unify the benchmarks for time efficiency and collision time scores, facilitating the management of large model evaluation standards. The fourth value range can also be different from the second value range described above. For example, the second and fourth value ranges can be divided according to the importance of the time efficiency and collision time scores. For instance, if the time efficiency score is more important than the collision time score, the second value range is 0–100, and the fourth value range is 0–200, facilitating accurate evaluation of large models.

[0260] For each large model, the large model is deployed on the first vehicle, and the driving strategy of the first vehicle is executed using the large model. The time T consumed by the first vehicle to complete the operation corresponding to the driving strategy is the time T required to complete the execution of the driving strategy of the first vehicle using the large model.

[0261] After the driving strategy of the first vehicle is executed using the large model, the electronic device can determine the time efficiency score of the large model based on the negative correlation between the time consumed and the time efficiency score, the fourth value range, and the time consumed by the large model.

[0262] In one example, the electronic device can calculate the ratio of the time consumed by each large model to the preset time limit, thus obtaining the time ratio for each large model. Based on the negative correlation between the time ratio and the time efficiency score, the time ratio for each large model, and the fourth range of values ​​from the minimum score to the maximum score, the time efficiency score for each large model is determined.

[0263] In this embodiment of the application, a maximum score of 100, a minimum score of 0, and a fourth value range of 0 to 100 are used as examples. For each large model, the electronic device can use the following formula (15) to determine the time efficiency score of the large model.

[0264]

[0265] In formula (15), TE is the time efficiency score of the large model, and T represents the time consumed by the large model. limit This indicates a time limit. The time limit can be set according to actual needs. In formula (15), only the maximum score value × (1-T / T) is used. limit The ) indicates the negative correlation between the time ratio and the time efficiency score, that is, the negative correlation between the time consumed and the time efficiency score, and does not play a limiting role.

[0266] For example, the negative correlation between time consumed and time efficiency score can also be expressed as:

[0267]

[0268] At this point, taking a maximum score of 100 as an example, the time efficiency score of the large model can also be determined using the following formula (16).

[0269]

[0270] In formula (16), TE is the time efficiency score of the large model, T represents the speed variance of the first vehicle, and T limit This indicates a time limit for the reservation.

[0271] In this embodiment, the electronic device controls the time efficiency score of each large model between the minimum and maximum score values, avoiding excessively large or small time efficiency scores for large models, thereby improving the accuracy of the efficiency index scores for large models and enhancing the accuracy of the evaluation of large models.

[0272] In step S802 above, the electronic device pre-configures the weights of each indicator. Using these pre-configured weights, the electronic device performs a weighted average of the security and efficiency scores of each large model to obtain a comprehensive score for each large model. Specifically, the comprehensive score for each large model can be determined using the following formula (17).

[0273] Score = w TTC ×TTC+ w SV ×SV+w TE ×TE (17)

[0274] In formula (17), Score represents the overall score of the large model, w TTC w represents the weight of the collision time score. SV w represents the weight of the velocity variance score. TE The weights represent the time efficiency scores, TTC represents the collision time score for the large model, SV represents the velocity variance score for the large model, and TE represents the time efficiency score for the large model.

[0275] In step S83 above, after obtaining the comprehensive scores of each large model, the electronic device selects the large model with the highest comprehensive score and deploys the selected large model within the autonomous driving system of the first vehicle. Subsequently, the large model serves as the on-board large model of the first vehicle, and the autonomous driving system of the first vehicle can utilize this on-board large model to execute the aforementioned autonomous driving strategy generation method.

[0276] In this embodiment of the application, while reducing the amount of data input for large language models, a vehicle-mounted evaluation scheme for large language models is proposed. TTC, SV, and TE are proposed. Through objective experimental evaluation, each type of large model suitable for vehicle mounting is scored, and the large language model with the higher score is selected as the vehicle-mounted large model mounting scheme.

[0277] The main objective of the vehicle-mounted large model selection strategy is to evaluate the performance of an agent based on a large language model in interpreting human commands in driving scenarios, and to evaluate how the large language model uses the provided functional primitives to generate code for motion planning.

[0278] In this application embodiment, two rule-based baseline strategies are employed: an intelligent driver model and a total braking principle based on minimal lane change. The former describes a rule-based approach that updates vehicle acceleration based on the specific speed and relative speed of the vehicle and the object directly in front to avoid collisions. The latter is an extension of the former; if the anticipated new lane offers a more favorable driving scenario and the maneuver can be performed safely, a lane change is executed. These baseline strategies can be considered to operate randomly because they are independent of human instructions and follow only predefined rules.

[0279] Large language models can generate coherent solutions for various autonomous driving tasks without additional fine-tuning. However, in code generation (such as generating policy planners), especially for complex road scenarios, large language models may produce suboptimal results. The autoregressive nature of these large language models poses a significant challenge to autonomous driving because tags generated earlier in the historical interaction sequence cannot be modified in the same iteration. This limitation restricts the ability of large language models to optimize initial effects. Embodiments of this application propose an in-vehicle large model selection strategy (such as...) Figure 8As shown in the figure, this can effectively improve some uncertainties in the process of mounting large vehicle models and improve the performance of large vehicle models in autonomous driving testing tasks.

[0280] In the technical solution provided in this application, the onboard large model plays the role of the decision-making "brain" in the autonomous driving system, while the perception or localization module plays the role of the vehicle's "eyes." The onboard large model does not directly affect the perception or localization module; the information collected by the perception or localization module serves as a reference to guide the higher-level decision-making process. By receiving processed data from these perception or localization modules, the onboard large model can enhance its intelligent decision-making capabilities, thereby significantly improving the performance of the autonomous vehicle. Downstream, the vehicle's control module (i.e., actuator) plays the role of the "hands" in the autonomous driving system, executing the driving strategy obtained from the decision-making process based on the onboard large model. Combined with this architecture, the autonomous driving system can execute... Figure 1 The autonomous driving strategy generation method shown (i.e., the pruning strategy based on layer-by-layer anomaly distribution) is used to generate autonomous driving strategies.

[0281] The inventors conducted experimental evaluations on the LLaMA7B and LLaMA13B large-scale models using the technical solutions provided in the embodiments of this application. They found that the large language model using the pruning strategy based on layer-by-layer anomaly distribution reduced the amount of data input by 0.4 times compared to the original large language model without the pruning strategy based on layer-by-layer anomaly distribution, while achieving 3 times the end-to-end inference speed in the inference engine.

[0282] Corresponding to the above-described autonomous driving strategy generation method, this application provides an autonomous driving strategy generation apparatus, such as... Figure 9 As shown, this device is applied to an autonomous driving system, where an in-vehicle large model is deployed within the system. The in-vehicle large model comprises multiple network layers. The device includes:

[0283] The first determining module 901 is used to determine the abnormal weight ratio of each network layer using calibration vehicle data;

[0284] The second determining module 902 is used to determine the sparsity of each network layer based on the abnormal weight ratio of multiple network layers, and the sparsity of each network layer is negatively correlated with the abnormal weight ratio.

[0285] The inference module 903 is used to process the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy. The ratio of the target weights to all weights included in each network layer is the sparsity of each network layer.

[0286] In some embodiments, the first determining module 901 is specifically configured to: use calibrated vehicle data to determine the abnormal score of each weight included in each network layer, and the abnormal average value of multiple weights included in each network layer; and determine the abnormal weight ratio of each network layer based on the abnormal score and abnormal average value of the multiple weights included in each network layer.

[0287] In some embodiments, the calibration vehicle data includes sub-data linked to each weight included in each network layer;

[0288] The first determining module 901 is specifically used to: multiply the absolute value of each weight by the second norm of the sub-data linked to that weight to obtain the anomaly score of each weight included in each network layer; calculate the average of the anomaly scores of multiple weights included in each network layer to obtain the anomaly average of multiple weights included in each network layer.

[0289] In some embodiments, the first determining module 901 is specifically used to: determine the weights of each network layer that meet preset conditions as abnormal weights, wherein the preset condition is that the abnormal score of the weight is greater than the first hyperparameter multiple of the abnormal average value of the network layer to which the weight belongs; and determine the ratio of the number of abnormal weights included in each network layer to the number of weights included in the network layer, thereby obtaining the abnormal weight ratio of each network layer.

[0290] In some embodiments, the second determining module 902 is specifically used to: determine the sparsity of each network layer based on the negative correlation between sparsity and the proportion of outlier weights, the first range of sparsity values ​​and the average sparsity, and the proportion of outlier weights of multiple network layers.

[0291] In some embodiments, the minimum value of the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value of the first value range is the sum of the average sparsity and the second hyperparameter.

[0292] In some embodiments, the first determining module 901 is specifically used to: convert calibration vehicle data into calibration text data; and use the calibration text data to determine the abnormal weight ratio of each network layer.

[0293] The inference module is specifically used to: convert current vehicle data into current text data; and process the current text data using the target weights included in each network layer to obtain an autonomous driving strategy.

[0294] In some embodiments, the first determining module 901 is specifically used to: input calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and use the calibration embedded vector to determine the abnormal weight ratio of each network layer.

[0295] The inference module is specifically used to: input the current vehicle data into the vector encoder to obtain the current embedded vector; and process the current embedded vector using the target weights included in each network layer to obtain the autonomous driving strategy.

[0296] In some embodiments, an actuator is further deployed within the autonomous driving system, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning approach; the aforementioned autonomous driving strategy generation apparatus further includes:

[0297] The execution module is used to send the strategy planning program to the executor after obtaining the autonomous driving strategy, so that the executor can execute the strategy planning program; and to demonstrate the strategy reasoning process.

[0298] In some embodiments, the calibration vehicle data and current vehicle data include at least one of the following: human commands, driving strategy evaluations, system messages, driving scenario data, and historical interaction information.

[0299] In the technical solution provided in this application, the in-vehicle large model is deployed within the autonomous driving system on the vehicle. The autonomous driving system uses calibrated in-vehicle data to determine the proportion of outlier weights in each network layer of the in-vehicle large model, thereby determining the sparsity of each network layer, such that the sparsity of each network layer is negatively correlated with the proportion of outlier weights in each network layer. The outlier weights in the network layers are strongly correlated with the computational power required for the in-vehicle large model, while the outlier weights have a relatively small impact on the accuracy of the autonomous driving strategy. In this embodiment, the sparsity of each network layer is negatively correlated with the proportion of abnormal weights in each network layer. That is, the larger the proportion of abnormal weights in a network layer, the fewer normal weights (i.e., target weights) participate in the calculation in that network layer. The proportion of target weights and abnormal weights participating in the calculation in each network layer is matched. By using this non-uniform layer-by-layer sparsity method, abnormal weights in each network layer are eliminated, and normal weights in each network layer are used to process real-time vehicle data to obtain autonomous driving strategies. While ensuring the accuracy of autonomous driving strategies, the number of weights participating in the calculation in the large vehicle model is greatly reduced, the overall computing power requirement of the large vehicle model is reduced, and the deployment of large language models in autonomous driving systems is realized when the computing power of vehicle chips is insufficient.

[0300] Corresponding to the above-described method for deploying large vehicle-mounted models, this application also provides a device for deploying large vehicle-mounted models, such as... Figure 10 As shown, the device includes:

[0301] Evaluation module 1001 is used to evaluate the safety index score when multiple large models are deployed on the first vehicle, and to evaluate the efficiency index score when multiple large models are deployed on the first vehicle.

[0302] The weighting module 1002 is used to perform a weighted average of the safety index score and efficiency index score of each large model to obtain the comprehensive score of each large model.

[0303] Deployment module 1003 is used to deploy the large model with the highest comprehensive score within the autonomous driving system of the first vehicle. The autonomous driving system is used to execute any of the above-mentioned autonomous driving strategy generation methods.

[0304] In some embodiments, the safety metric score includes a collision time score;

[0305] The evaluation module 1001 is specifically used to: evaluate the collision occurrence time between the first vehicle and each second vehicle corresponding to each large model when executing the driving strategy of the first vehicle using each large model; and determine the collision time score of each large model based on the safe time threshold and the multiple collision occurrence times corresponding to each large model.

[0306] In some embodiments, the evaluation module 1001 is specifically used for:

[0307] Determine the minimum collision occurrence time for each large model from among the multiple collision occurrence times corresponding to each large model;

[0308] When the minimum collision occurrence time corresponding to each large model is greater than the safe time threshold, the collision time score of each large model is determined as the maximum score value.

[0309] When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined as the minimum score value.

[0310] When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

[0311] In some embodiments, the safety metric score includes a velocity variance score;

[0312] Evaluation module 1001 is specifically used to: evaluate the speed variance of the first vehicle corresponding to each large model when executing the driving strategy of the first vehicle using each large model; and determine the speed variance score of each large model based on the negative correlation between speed variance and speed variance score, the speed variance corresponding to each large model, and the third value range from the minimum score value to the maximum score value.

[0313] In some embodiments, the evaluation module 1001 is specifically used to: calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation, and obtain the speed ratio corresponding to each large model; and determine the speed variance score of each large model based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and the third value range from the minimum score value to the maximum score value.

[0314] In some embodiments, the efficiency index score includes a time efficiency score;

[0315] The evaluation module 1001 is specifically used to: evaluate the time consumed by the first vehicle to complete the operation corresponding to the driving strategy of each large model when executing the driving strategy of each large model; and determine the time efficiency score of each large model based on the negative correlation between the consumed time and the time efficiency score, the consumed time corresponding to each large model, and the fourth value range from the minimum score value to the maximum score value.

[0316] In some embodiments, the evaluation module 1001 is specifically used to: calculate the ratio of the time consumed by each large model to the preset time limit, and obtain the time ratio for each large model; and determine the time efficiency score for each large model based on the negative correlation between the time ratio and the time efficiency score, the time ratio for each large model, and the fourth value range from the minimum score to the maximum score.

[0317] In the technical solution provided in this application, each large model is evaluated using safety and efficiency indicators, and the large model with the highest score is selected for installation on the vehicle. This method of determining the onboard large model reduces the amount of data input required and effectively mitigates some uncertainties during the installation process, thereby improving the performance of the onboard large model in autonomous driving tasks.

[0318] Corresponding to the above-described method for generating autonomous driving strategies, this application also provides an autonomous driving system, such as... Figure 11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0319] Memory 1103 is used to store computer programs;

[0320] The processor 1101 is used to execute the program stored in the memory 1103 to implement any of the above-mentioned autonomous driving strategy generation methods.

[0321] Corresponding to the above-described method for deploying large vehicle-mounted models, this application also provides an electronic device, such as... Figure 12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204, wherein the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.

[0322] Memory 1203 is used to store computer programs;

[0323] The processor 1201 is used to execute the program stored in the memory 1203 to implement any of the above-mentioned vehicle-mounted large model deployment methods.

[0324] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0325] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0326] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0327] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0328] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements any of the above-described autonomous driving strategy generation methods or any of the above-described vehicle-mounted large model deployment methods.

[0329] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the above-described autonomous driving strategy generation methods or any of the above-described vehicle-mounted large model deployment methods.

[0330] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0331] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0332] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of devices, autonomous driving systems, electronic devices, storage media, and program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0333] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for generating an autonomous driving strategy, characterized in that, Applied to an autonomous driving system, wherein an in-vehicle large model is deployed within the autonomous driving system, the in-vehicle large model comprising multiple network layers, the method includes: By using the sub-data linked to each weight in each network layer from the calibration vehicle data, the absolute value of each weight is multiplied by the second norm of the sub-data linked to that weight to obtain the anomaly score of each weight in each network layer. Calculate the average of the outlier scores of the multiple weights included in each network layer to obtain the average outlier score of the multiple weights included in each network layer; The weights of each network layer that meet a preset condition are identified as abnormal weights. The preset condition is that the abnormal score of the weight is greater than the average abnormal score of the network layer to which the weight belongs by a first hyperparameter multiple. Determine the ratio of the number of outlier weights in each network layer to the total number of weights in that network layer to obtain the outlier weight ratio of each network layer; Based on the proportion of abnormal weights in the multiple network layers, the sparsity of each network layer is determined, and the sparsity of each network layer is negatively correlated with the proportion of abnormal weights. By utilizing the target weights included in each network layer, the current vehicle data is processed to obtain an autonomous driving strategy. The ratio of the target weights to all weights included in each network layer is the sparsity of each network layer.

2. The method according to claim 1, characterized in that, The step of determining the sparsity of each network layer based on the proportion of abnormal weights of the multiple network layers includes: The sparsity of each network layer is determined based on the negative correlation between sparsity and the proportion of outlier weights, the first range of sparsity values ​​and the average sparsity, and the proportion of outlier weights of the multiple network layers.

3. The method according to claim 2, characterized in that, The minimum value in the first value range is the difference between the average sparsity and the second hyperparameter, and the maximum value in the first value range is the sum of the average sparsity and the second hyperparameter.

4. The method according to claim 1, characterized in that, The step of obtaining the anomaly score of each weight in each network layer by multiplying the absolute value of each weight by the second norm of the sub-data linked to that weight in the calibration vehicle data includes: converting the calibration vehicle data into calibration text data; and obtaining the anomaly score of each weight in each network layer by multiplying the absolute value of each weight by the second norm of the sub-data linked to that weight in the calibration text data. The step of processing the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy includes: converting the current vehicle data into current text data; and processing the current text data using the target weights included in each network layer to obtain an autonomous driving strategy.

5. The method according to claim 1, characterized in that, The step of obtaining the anomaly score of each weight in each network layer by multiplying the absolute value of each weight by the second norm of the sub-data linked to that weight in the calibration vehicle data includes: inputting the calibration vehicle data into a vector encoder to obtain a calibration embedded vector; and using the sub-data linked to each weight in each network layer in the calibration embedded vector, multiplying the absolute value of each weight by the second norm of the sub-data linked to that weight to obtain the anomaly score of each weight in each network layer. The step of processing the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy includes: inputting the current vehicle data into a vector encoder to obtain a current embedded vector; and processing the current embedded vector using the target weights included in each network layer to obtain an autonomous driving strategy.

6. The method according to claim 1, characterized in that, The autonomous driving system also deploys actuators, and the autonomous driving strategy includes a strategy planning program and a strategy reasoning approach; after obtaining the autonomous driving strategy, the method further includes: The strategy planning program is sent to the executor so that the executor executes the strategy planning program. The reasoning process of the strategy is demonstrated.

7. The method according to any one of claims 1-6, characterized in that, The calibration vehicle data and the current vehicle data include at least one of the following: human commands, driving strategy evaluation, system messages, driving scenario data, and historical interaction information.

8. A method for deploying a large vehicle-mounted model, characterized in that, The method includes: The safety metrics scores of multiple large models deployed on the first vehicle are evaluated, as are the efficiency metrics scores of multiple large models deployed on the first vehicle. The safety and efficiency scores of each large model are weighted and averaged to obtain the comprehensive score of each large model. The large model with the highest overall score is deployed within the autonomous driving system of the first vehicle, the autonomous driving system being used to perform the method according to any one of claims 1-7.

9. The method according to claim 8, characterized in that, The safety performance score includes a collision time score. The steps for evaluating safety metrics scores when multiple large models are deployed on a first vehicle include: When executing the driving strategy of the first vehicle using each large model, the collision occurrence time of the first vehicle and each second vehicle corresponding to each large model is evaluated; based on the safe time threshold and the multiple collision occurrence times corresponding to each large model, the collision time score of each large model is determined.

10. The method according to claim 9, characterized in that, The step of determining the collision time score for each large model based on a safety time threshold and multiple collision occurrence times corresponding to each large model includes: Determine the minimum collision occurrence time for each large model from among the multiple collision occurrence times corresponding to each large model; When the minimum collision occurrence time corresponding to each large model is greater than the safe time threshold, the collision time score of each large model is determined as the maximum score value. When the minimum collision occurrence time corresponding to each large model is less than or equal to 0, the collision time score of each large model is determined as the minimum score value. When the minimum collision occurrence time corresponding to each large model is less than or equal to the safety time threshold, and the minimum collision occurrence time corresponding to each large model is greater than 0, the collision time score of each large model is determined based on the positive correlation between the minimum collision occurrence time and the safety index score, the minimum collision occurrence time corresponding to each large model, and the second value range from the minimum score value to the maximum score value.

11. The method according to claim 8, characterized in that, The safety index score includes a velocity variance score; The steps for evaluating safety metrics scores when multiple large models are deployed on a first vehicle include: When executing the driving strategy of the first vehicle using each large model, evaluate the speed variance of the first vehicle corresponding to each large model; Based on the negative correlation between velocity variance and velocity variance score, the velocity variance of each large model, and the third range of values ​​from the minimum score to the maximum score, the velocity variance score of each large model is determined.

12. The method according to claim 11, characterized in that, The step of determining the velocity variance score for each large model based on the negative correlation between velocity variance and velocity variance score, the velocity variance corresponding to each large model, and the third value range from the minimum score to the maximum score includes: Calculate the ratio of the speed variance corresponding to each large model to the preset safe speed deviation to obtain the speed ratio corresponding to each large model; based on the negative correlation between the speed ratio and the speed variance score, the speed ratio corresponding to each large model, and the third range of values ​​from the minimum score to the maximum score, determine the speed variance score of each large model.

13. The method according to claim 8, characterized in that, The efficiency index scoring includes a time efficiency score; The step of evaluating the efficiency metric score when multiple large models are deployed on the first vehicle includes: When executing the driving strategy of the first vehicle using each large model, the time consumed by the first vehicle corresponding to each large model to complete the operation corresponding to the driving strategy is evaluated. Based on the negative correlation between time consumption and time efficiency score, the time consumption corresponding to each large model, and the fourth range of values ​​from the minimum score to the maximum score, the time efficiency score of each large model is determined.

14. The method according to claim 13, characterized in that, The step of determining the time efficiency score of each large model based on the negative correlation between time consumption and time efficiency score, the time consumption corresponding to each large model, and the fourth value range from the minimum score to the maximum score includes: Calculate the ratio of the time consumed by each large model to the preset time limit to obtain the time ratio for each large model; Based on the negative correlation between the time ratio and the time efficiency score, the time ratio corresponding to each large model, and the fourth range of values ​​from the minimum score to the maximum score, the time efficiency score of each large model is determined.

15. An autonomous driving strategy generation device, characterized in that, The device is applied to an autonomous driving system, wherein an in-vehicle large model is deployed within the autonomous driving system, the in-vehicle large model includes multiple network layers, and the device includes: The first determining module is used to utilize the sub-data linked to each weight in each network layer from the calibration vehicle data, multiply the absolute value of each weight by the second norm of the sub-data linked to that weight to obtain the abnormal score of each weight in each network layer; calculate the average of the abnormal scores of multiple weights in each network layer to obtain the abnormal average of multiple weights in each network layer; determine the weights in each network layer that meet a preset condition as abnormal weights, wherein the preset condition is that the abnormal score of the weight is greater than the abnormal average of the network layer to which the weight belongs by a first hyperparameter multiple; and determine the ratio of the number of abnormal weights in each network layer to the number of weights in the network layer to obtain the abnormal weight ratio of each network layer. The second determining module is used to determine the sparsity of each network layer based on the abnormal weight ratio of the multiple network layers, wherein the sparsity of each network layer is negatively correlated with the abnormal weight ratio. The inference module is used to process the current vehicle data using the target weights included in each network layer to obtain an autonomous driving strategy. The ratio of the target weights to all weights in each network layer is the sparsity of each network layer.

16. A vehicle-mounted large model deployment device, characterized in that, The device includes: The evaluation module is used to evaluate the safety index scores of multiple large models deployed on the first vehicle, and to evaluate the efficiency index scores of multiple large models deployed on the first vehicle. The weighting module is used to perform a weighted average of the safety and efficiency scores of each large model to obtain a comprehensive score for each large model. A deployment module is used to deploy the largest model with the highest overall score within the autonomous driving system of the first vehicle, the autonomous driving system being used to perform the method described in any one of claims 1-7.

Images