End-cloud collaborative vehicle-mounted interaction method for multiple task scenes

Abstract

The invention discloses an end-cloud cooperative vehicle-mounted interaction method for multiple task scenes. The method comprises the following steps: (1) training a fine tuning model in an edge executor of a vehicle-mounted end; (2) the vehicle-mounted terminal receives a user instruction and converts the user instruction into a task text instruction; (3) calculating the execution confidence coefficient of the task text instruction, if the execution confidence coefficient is greater than a threshold, entering step (5), and if the execution confidence coefficient is less than the threshold, entering step (4); (4) the cloud planner receives the original task, decomposes the original task into sub-tasks through a cloud large model, converts the sub-tasks into task text instructions and then sends the task text instructions to the edge executor; (5) the edge executor receives a task text instruction, sends the task text instruction into the fine tuning and enlarging model and outputs user instructions meeting different scenes; a scene-related vehicle end API interface is called, and a processing result is returned; and (6) summarizing the returned processing result by the edge executor, and making a final response to a user instruction. The problem that calling interfaces of the vehicle-mounted external tool are not uniform can be solved, and the efficiency of processing a multi-task scene is improved.

Description

Technical Field

[0001] The present invention belongs to the field of in-vehicle voice interaction, and in particular relates to an end-cloud collaborative in-vehicle interaction method for multiple task scenarios. Background Art

[0002] With the popularity of new energy vehicles and users' increasing demand for car interactive experience, how to make cars intelligent is becoming more and more important. However, due to design rules and model restrictions, traditional in-car assistants have limitations in data processing, semantic understanding and problem reasoning, making it difficult to deeply understand the driver's intention and give corresponding execution feedback.

[0003] The current large language model provides strong semantic understanding capabilities, and can have more natural and fluent conversations with drivers and passengers, and even provide more personalized services and recommendations. Using large language models as in-car assistants has shown great potential in practical applications.

[0004] For example, a Chinese patent document with publication number CN117198289A discloses a voice interaction method, apparatus, device, medium and product, and a specific implementation scheme is: obtaining voice information corresponding to a target user collected by a vehicle-mounted terminal, and obtaining text information corresponding to the voice information; determining a text expression type corresponding to the voice information based on the text information; using a target language model corresponding to the text expression type to determine understanding result information of the voice information; and outputting the understanding result information through the vehicle-mounted terminal.

[0005] However, large language models actually have the following problems:

[0006] 1. In the field of in-vehicle assistants, there are strong requirements for real-time information (real-time traffic conditions, weather information, etc.) and it involves interaction with in-vehicle terminal software. For different in-vehicle systems, the software interface standards are not unified. If the large language model is fine-tuned to enable it to obtain the ability to call the in-vehicle API interface, a large amount of data is required and the cost is high.

[0007] 2. The vehicle's on-board terminal has limited computing power, and the operation speed of complex large language models is slow. There is a high latency defect in processing complex tasks. However, deploying the model to the cloud is subject to the communication status between the vehicle and the cloud and the large amount of real-time data transmission of the entire vehicle. The processing efficiency of simple tasks is low.

[0008] 3. The existing large vehicle-mounted model cannot efficiently adapt to various mission scenarios, and it is necessary to fine-tune the large vehicle-mounted model, such as full parameter fine-tuning or LoRA efficient fine-tuning.

[0009] However, these methods usually require some operations on the structure or parameters of the large model, which still imposes a heavy burden on parameter storage and practical application. In addition, these methods are not convenient for switching the fine-tuned model between different scenarios, so they often only use data from the smart car field to fine-tune a model. The data used for fine-tuning is relatively mixed, making it difficult to provide car owners with more accurate and intelligent services for specific task scenarios.

[0010] To sum up, the existing deployment solutions have problems such as inconsistent calling interfaces for on-board external tools, difficulty for large on-board local models to handle complex tasks, high latency in data interaction between large cloud models and vehicle-side data, and inability to efficiently handle multi-task scenarios, which in turn affects the intelligent interaction experience between users and vehicles. Summary of the invention

[0011] The present invention provides a terminal-cloud collaborative vehicle-mounted interaction method for multiple task scenarios to solve the problem of inconsistent calling interfaces of on-board external tools, while reducing the delay of on-board local large models in processing complex tasks and the interaction between cloud large models and vehicle-side data, thereby improving the efficiency of processing multi-task scenarios.

[0012] A method for end-cloud collaborative vehicle interaction for multiple mission scenarios includes the following steps:

[0013] (1) constructing and training a fine-tuned large model in the edge executor on the vehicle side, wherein the fine-tuned large model includes a scene analysis module, a prompt selection module, an input processing module, and an output module;

[0014] Among them, the scenario analysis module uses the on-board lightweight large model to extract key information from the user command and determine the usage scenario of the current command; the prompt selection module selects the corresponding guiding text according to the usage scenario; the input processing module encodes the guiding text and the text corresponding to the user command and then splices them to obtain the spliced ​​prompt words; the output module inputs the spliced ​​prompt words into the decoder of the on-board lightweight large model and outputs the user command corresponding to the usage scenario;

[0015] (2) The vehicle terminal receives the original user command as well as the current vehicle status information and sensor data, and converts the original user command into a task text command in natural language;

[0016] (3) Calculate the execution confidence of the task text instruction by means of prompt words. If the execution confidence is greater than the set threshold, proceed to step (5) for local processing. If the execution confidence is less than the set threshold, proceed to step (4) for cloud processing.

[0017] (4) The cloud planner receives the task text instructions, analyzes the task requirements through the cloud's large language model, generates an execution plan, decomposes the execution plan into subtasks and converts them into task text instructions in natural language form, and then sends them to the edge executor through the communication protocol;

[0018] (5) The edge executor on the vehicle receives the task text instruction, sends the task text instruction as the user instruction to the fine-tuning large model, and outputs the user instruction that meets the needs of the car owner in different scenarios; and calls the vehicle-side API interface related to the scenario and returns the processing result;

[0019] (6) The edge executor summarizes the returned processing results and makes a final response to the user's instructions.

[0020] In step (1), the vehicle-mounted lightweight large model includes a text encoding Tokenizer, an Embedding layer, a decoder consisting of N decoder layers, and an MLP classifier.

[0021] In the process of training and fine-tuning the large model, data sets of different usage scenarios are used, and guiding texts are designed for different usage scenarios. The guiding texts are encoded by the Tokenizer and Embedding layers and used as the initial prompt words. The text corresponding to the user instructions is encoded by the Tokenizer and Embedding layers and then concatenated with the initial prompt words to obtain the concatenated prompt words. During the training process, the parameters of the decoder in the on-board lightweight large model are frozen, and the parameters of the Embedding layer are trained.

[0022] The guiding text is encoded as the initial prompt word, and the formula is:

[0023] E p =SoftEmbedding(Tokenizer(t))

[0024] Among them, E p represents the initial prompt word, and t represents the designed guiding text.

[0025] After encoding the text corresponding to the user instruction, concatenate it with the initialization prompt word to obtain the concatenated prompt word. The formula is:

[0026] E i =Concat(SoftEmbedding(Tokenizer(t i )), E p )

[0027] Among them, E i Indicates the final input of the splicing prompt word of the lightweight large model on board; t iRepresents the input text in the dataset sample, that is, the user instruction; Concat represents the concatenation operation.

[0028] In step (3), the formula for executing confidence is as follows:

[0029]

[0030] In the formula, Representing a large language model based on context Get the output probability of the next word; i represents the sequence number of the context after being encoded by the text encoding Tokenizer; x y and x n represents yes or no, corresponding to whether the answer is encoded by the text encoding Tokenizer; C represents the confidence level, C n Indicates the confidence of negation, C y Indicates positive confidence, that is, execution confidence.

[0031] In step (3), if the execution confidence is less than the set threshold, the vehicle terminal sends an HTTPS request to the cloud planner in JSON format, and the request content includes the original task description and parameters.

[0032] In step (4), the cloud planner includes a task parser, a plan generator, and a task decomposer, all of which are implemented through a large language model in the cloud.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The prompt fine-tuning capability enhancement method used in the present invention consumes resources equivalent to inputting a short sentence into the on-board lightweight large model, thereby improving the capabilities of the on-board lightweight large model in relevant scenarios in the field of smart cars with extremely small resource burden.

[0035] 2. The present invention can analyze and determine the usage scenario, and fine-tune the prompts according to the scenario switching, so that the lightweight large model on the vehicle can meet the needs of the car owner in a more targeted manner in multiple mission scenarios.

[0036] 3. The present invention uses a large language model on the cloud to perform task planning through end-cloud collaboration, and a small model on the vehicle edge to perform specific execution. By fine-tuning the small model to be compatible with the calling interface of various on-vehicle external tools, the problem of inconsistent interfaces and excessive fine-tuning costs can be solved.

[0037] 4. The present invention reduces the data interaction delay between the cloud and the vehicle by deploying a lightweight large model on the vehicle side, and solves the planning and subtask decomposition problems of complex tasks through the cloud, thereby improving the system's response speed and user experience.

[0038] 5. The design of the present invention supports the rapid integration of new vehicle-mounted systems and added functions, and has high scalability and flexibility. New vehicle-mounted systems and added functions can be supported by only fine-tuning the lightweight large language model on board. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a method for end-cloud collaborative vehicle-mounted interaction for multiple task scenarios of the present invention;

[0040] Figure 2 A training flow chart for fine-tuning a large model provided in an embodiment of the present invention;

[0041] Figure 3 A flowchart for fine-tuning a large model in the edge actuator of the present invention;

[0042] Figure 4 It is the overall structure diagram of the fine-tuning large model in the edge actuator of the present invention;

[0043] Figure 5 The schematic diagram is a framework structure diagram of the method of the present invention deployed in a vehicle system. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0045] like Figure 1 As shown, a method for end-cloud collaborative vehicle interaction for multiple task scenarios includes the following steps:

[0046] Step 1: Construct and train a fine-tuned large model in the edge executor on the vehicle side. The fine-tuned large model includes a scene analysis module, a prompt selection module, an input processing module, and an output module.

[0047] like Figure 3 and Figure 4 As shown, the scenario analysis module uses the on-board lightweight large model to extract key information from user commands and determine the usage scenario of the current command.

[0048] Prompt selection module, select the corresponding guiding text according to the usage scenario.

[0049] The input processing module encodes the guiding text and the text corresponding to the user instruction and then splices them to obtain a spliced ​​prompt word.

[0050] The output module inputs the spliced ​​prompt words into the decoder of the lightweight large model on board and outputs the user instructions corresponding to the usage scenario.

[0051] In step 1, when training and fine-tuning the large model, design guiding texts for different scenarios and use the corresponding encoded Tokens Embeddings as prompt (soft) Embeddings during initialization. Use data sets from different scenarios to freeze the parameters of the backbone network of the large vehicle model and train the prompt (soft) Embeddings parameters. Figure 2 As shown, the specific training process is as follows:

[0052] Step 1.1, establish a decoder consisting of N decoding layers as the backbone network of the vehicle-mounted large model.

[0053] In one embodiment, the decoding layer in the vehicle-mounted large model backbone network includes an attention layer, a position encoding layer, and an MLP.

[0054] Step 1.2, build Tokenizer.

[0055] Tokenizer consists of a Token id and a corresponding word, and is used to establish a correspondence between the Token id and the word in the vocabulary.

[0056] Step 1.3, build the (soft) Embedding layer.

[0057] The (soft) Embedding layer is used to map the Token ID to a high-dimensional vector that can be understood by the vehicle-mounted large model. The overall structure of the vehicle-mounted large model includes the steps of the (soft) Embedding layer, the decoder, and the MLP classifier for predicting the next output Token ID.

[0058] Step 1.4, collect data and pre-train the large vehicle model.

[0059] Pre-training data is obtained through web crawlers, using massive data to train the basic capabilities of large in-vehicle models.

[0060] Step 1.5, freeze the backbone network parameters of the large vehicle model.

[0061] Step 1.6, obtain M downstream task scenario data sets in the field of smart cars, such as vehicle manual data sets, vehicle maintenance data sets, path planning data sets, and driver psychology data sets.

[0062] The dataset is obtained by collecting and building intelligent car usage data from the Internet. Each dataset contains task scenarios and specific examples, where a specific example consists of an input from a car owner and a real answer.

[0063] Step 1.7, design M texts with positive guidance corresponding to the dataset scenes as the text content for prompt initialization.

[0064] The prompt initialization text is "Please think step by step" to stimulate the reasoning ability of the on-board large model.

[0065] Step 1.8, convert the prompt text into (soft) Embeddings through the Tokenizer and (soft) Embedding layers.

[0066] Step 1.9, set the hint (soft) Embeddings as trainable parameters.

[0067] Step 1.10, set the hyperparameters required for fine-tuning, such as learning rate and number of learning rounds.

[0068] Step 1.11, for each input of each dataset, when inputting the vehicle-mounted large model, the corresponding prompt (soft) Embeddings are concatenated after the dataset input (soft) Embeddings.

[0069] Process the input content of each input to the vehicle-mounted large model. At each input, the prompt embeddings corresponding to the used dataset are spliced ​​to the (soft) embeddings obtained by the input text in the dataset after passing through the Tokenizer and (soft) Embedding layers. The process formula is as follows:

[0070] E i =Concat(softEmbedding(Tokenizer(t i )), E p )

[0071] Among them, E i represents the (soft) Embeddings of the final input vehicle-mounted large model, t i represents the input text in the dataset sample, E p Indicates the hint (soft) Embeddings corresponding to the dataset.

[0072] Step 1.12: Input the concatenated (soft) Embeddings obtained in step 1.11 into the large vehicle model, predict the next Token, and obtain the original unstandardized score tensor logits output by the model.

[0073] E iAfter inputting the large vehicle model, it passes through the decoder and MLP classifier to obtain the original unstandardized score tensor logits, l0, for predicting the next token. l0 is the prediction of the large vehicle model for the next token, which is the original score of the next token in all token classes. Then, in step 1.12, the first token of the real output in the dataset sample is taken out, and after passing through the Tokenizer and (soft) Embedding layer, e0 is obtained and spliced ​​to E i After that, we get E ie Step 1.12: E ie As new input embeddings, it is input into the on-board large model, and the next token is predicted again through the decoder and classifier. After obtaining the prediction score l1, the (soft) embedding corresponding to the second token in the real output of the dataset sample, e1, is concatenated into E ie After that, we get the new E ie After that, step 1.12 is repeated to input E into the vehicle-mounted large model. ie , get the predicted score tensor, take out the Token and splice the corresponding (soft) Embedding to E ie The process continues until the real output of the dataset sample has no next Token. Finally, step 1.12 adds l0, l1, ..., l n After splicing, the output logits tensor of the on-board large model corresponding to the real output in the dataset is obtained.

[0074] Step 1.13, calculate the loss value based on the true output of the dataset and the output logits of the on-board large model in step 1.12.

[0075] Specifically, the logits tensor obtained in step 1.12 and the Token ids corresponding to the actual output of the data set are input into the cross entropy loss function to obtain the loss value.

[0076] Step 1.14, perform backpropagation based on the loss value, train and optimize all the prompt (soft) Embeddings, and obtain multiple fine-tuned prompts.

[0077] Specifically, back propagation is performed according to the loss value calculated in step 1.13, and the prompt (soft) Embeddings of different data sets are fine-tuned respectively to obtain prompt (soft) Embeddings for different downstream task scenarios.

[0078] Step 2: According to the current task, select the corresponding prompt (soft) Embeddings and splice them into the content input by the car owner, so that the fine-tuned large model on the vehicle can output content that meets the needs of the car owner in different scenarios.

[0079] In one embodiment, step 2 selects the corresponding prompts (soft) Embeddings for different downstream task scenarios obtained in step 1 and splices them to the owner's input content according to the current task, so that the in-vehicle large model can output content that meets the owner's needs in different scenarios, so that it has the ability to call the in-vehicle API, commonly used vehicle-side software such as weather query, map positioning, real-time news, etc. Then the lightweight large language model is deployed to the vehicle terminal as an edge executor, responsible for the specific execution of the subtasks.

[0080] The specific steps of step 2 are as follows:

[0081] Step 2.1: Use the vehicle-mounted big model to analyze the owner’s input and determine the usage scenario.

[0082] Step 2.2, according to the judgment result, select the prompt (soft) Embeddings corresponding to the scene in the prompt (soft) Embeddings library.

[0083] Step 2.3, after splicing the prompt (soft) Embeddings to the (soft) Embeddings of the car owner's input content, input them together into the vehicle model.

[0084] In step 2.4, use the large vehicle model to process the input Embeddings and obtain output that meets the needs of the scene.

[0085] Step 2.4 After the vehicle-mounted large model processes the (soft) Embeddings input in step 2.3, the vehicle-mounted large model obtains the probability prediction result of the next Token. Then, step 2.4 takes out the Token id of the Token class corresponding to the highest probability in the prediction result, and vectorizes this predicted Token id into a (soft) Embedding that can be understood and processed by the vehicle-mounted large model through the (soft) Embedding layer, represented as E o , after being concatenated to the input (soft) Embeddings, we get E io Then, the large model on the vehicle is io As input, predict the next Token id again and concatenate its corresponding (soft) Embedding to E io After that, we get the new E ioStep 2.4 repeats the prediction and concatenation steps until the id of the next token predicted is the id of the special token at the end of the text. Finally, step 2.4 converts all the predicted token ids into corresponding text tokens through the Tokenizer to form a coherent content as the output of the large vehicle model.

[0086] Step 3: The vehicle terminal receives the original user command and the current vehicle status information and sensor data, and converts the original command (which may be a voice signal) into a natural language text command form.

[0087] Step 4, calculate the execution confidence of the text instruction by means of prompt words, and choose to process locally to enter step 6 or send the text instruction to the cloud to enter step 5.

[0088] The method for calculating the confidence of text instructions specifically includes: realizing the execution confidence by means of prompt words, the prompt words are as follows: "Can you answer or respond to the following: [text instruction], please answer yes or no." Since the existing large language model is a decoder architecture, the model is based on the context. Get the output probability of the next word i represents the sequence number of the context after being encoded by the text encoding Tokenizer, x y and x n Represents the result of "yes" and "no" after being encoded by the text encoding Tokenizer. The calculation method of the execution confidence Cy is as follows:

[0089]

[0090] Get the execution confidence C y After that, a threshold α is set, and whether the result needs to be uploaded to the cloud is determined based on whether it is greater than the threshold. If it is greater than the threshold, the local execution proceeds to step 6; if it is less than the threshold, the vehicle terminal sends an HTTPS request to the cloud planner in JSON format, and the request content includes the original task description, parameters, and other necessary information, and then proceeds to step 5.

[0091] In step 5, the cloud planner receives the original task, analyzes the task requirements through the large language model, generates an execution plan, decomposes the plan into subtasks, and sends it to the edge executor through the communication protocol in the form of natural language.

[0092] The cloud planner consists of the following three components: task parser, plan generator, and task decomposer, all of which are implemented through a large language model.

[0093] The main responsibility of the task parser is to receive and parse the original task. It extracts key information from the task, such as task type, expected goal, and priority. It extracts structured information from the parsed results, such as task name, description, parameters, expected completion time, etc. The prompt template used is as follows: "You are a task parser, and your goal is to receive and parse the following original task. Please carefully analyze the text content and extract the following structured information from it: task type, expected goal, priority, task name, task description, parameters, and expected completion time. \nOriginal task: [Original task text] Task type: \nExpected goal: \nPriority: \nTask name: \nTask description: \nParameters: \nExpected completion time: \nNow please start parsing and output structured information." Among them, \n represents a line break, the text of the entire task is a string, and \n is used to distinguish different fields as a truncation.

[0094] The plan generator generates a high-level execution plan based on the information provided by the task parser. This generated plan outlines the steps, resources, and possible dependencies required to complete the task, thereby determining the order of task execution, required resources, etc. The prompt template used is as follows: "You are a plan generator, and you will generate a detailed execution plan based on the following information provided by the task parser. Based on the extracted task information, please develop a plan that covers all the steps required to complete the task, the required resources, and the possible dependencies between the steps. Determine the order of task execution and the specific resources required.\n[Task Parser Output]\nPlease generate an execution plan in the following format: Execution Plan: Step 1: [Describe the first step]\nRequired Resources: [List the required resources]\nDependencies: [Describe the prerequisites of this step]\nStep 2: [Describe the second step]\nRequired Resources: [List the required resources]\nDependencies: [Describe the prerequisites of this step]\n...Please make sure your plan is detailed and generate an execution plan."

[0095] The task decomposer is responsible for decomposing the high-level execution plan generated by the plan generator into smaller subtasks that can be executed by the edge. Use the large language model to decompose each step in the plan into specific subtasks, which are independently executable units. And define a clear interface for each subtask, including input parameters, expected outputs, and execution conditions. The prompt template used is as follows: "You are a task decomposer. Receive the following high-level execution plan generated by the plan generator and decompose it into smaller subtasks that can be executed by the edge. For each step in the plan, please create specific, independently executable subtasks. For each subtask, please define a clear interface, including input parameters, expected outputs, and execution conditions. Execution plan: [Plan generator output] \nPlease decompose each step in the following format: \nSubtask: \nSubtask: \n... \nEnsure that each subtask is clear and executable, and that all subtasks together can complete the entire execution plan. Please decompose the plan and define the subtask interface."

[0096] In step 6, after the vehicle-mounted edge executor receives the task, it uses a lightweight large language model to make local decisions and execute tasks, and calls the relevant vehicle-side API interface (trained using the method of fine-tuning prompted in step 1), and returns the processing result.

[0097] After receiving the subtask, the edge executor needs to parse the task content, including the task ID, task description, parameters, expected output, and execution conditions. Verify the integrity and validity of the subtask to ensure that the task can be executed correctly. Secondly, check the vehicle-side resources, such as CPU, memory, storage, and sensor status, to ensure that there are sufficient resources to execute the task. The edge executor calls a lightweight large language model, passes in task information described in the form of natural language text, and requests the model to provide decision or action guidance. After receiving the model response, parse the decision result, determine the specific operations to be performed, and call the relevant vehicle-side API interface (trained using the prompt fine-tuning method in step 1) if necessary to perform specific operations, such as adjusting vehicle settings, starting vehicle functions, and planning routes.

[0098] During the task execution process, the edge executor needs to monitor the execution status and vehicle status to ensure that the task is carried out as expected. If an exception or error is encountered during the execution process, the executor needs to capture the exception and record the error information, and decide whether to retry the task or report it to the cloud planner based on the error type.

[0099] In step 7, the edge executor summarizes the results obtained by calling the vehicle-side API and makes a final response to the user's instructions.

[0100] like Figure 5As shown, the framework structure of deploying the method of the present invention in the vehicle system includes a data conversion unit, a vehicle communication unit, an on-board API control unit, a cloud planner, and an edge executor.

[0101] The data conversion unit is used to receive various types of data from the vehicle, such as sensor data, vehicle status information, etc., as well as receive user instructions and convert voice data into natural language text form.

[0102] The vehicle communication unit is used to establish and maintain the communication connection between the vehicle and the cloud server, transmit vehicle status information, sensor data and received commands to the cloud server, receive control commands and feedback information sent by the cloud server, and pass them to the on-board API control unit.

[0103] The cloud planner is used to receive the original task, analyze the task requirements through a large language model, generate an execution plan, decompose the plan into subtasks, and send it to the edge executor in the form of natural language through a communication protocol.

[0104] The edge executor is used to receive subtasks and instructions from the cloud planner. Based on the received tasks, it calls the corresponding vehicle API to perform specific operations and controls and gives a final response.

[0105] The vehicle API control unit is used to manage and execute vehicle API calls initiated by the edge actuator. According to the API instructions, it controls vehicle hardware such as air conditioning, wipers, seats, etc. and vehicle software such as navigation, multimedia, and driving assistance.

[0106] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for end-cloud collaborative vehicle interaction for multiple mission scenarios, characterized in that: The following steps are involved: (1) constructing and training a fine-tuning large model in the edge executor on the vehicle side, wherein the fine-tuning large model includes a scene analysis module, a prompt selection module, an input processing module, and an output module; Among them, the scenario analysis module uses the on-board lightweight large model to extract key information from the user command and determine the usage scenario of the current command; the prompt selection module selects the corresponding guiding text according to the usage scenario; the input processing module encodes the guiding text and the text corresponding to the user command and then splices them to obtain the spliced ​​prompt words; the output module inputs the spliced ​​prompt words into the decoder of the on-board lightweight large model and outputs the user command corresponding to the usage scenario; (2) The vehicle terminal receives the original user command as well as the current vehicle status information and sensor data, and converts the original user command into a task text command in natural language; (3) Calculate the execution confidence of the task text instruction by means of prompt words. If the execution confidence is greater than the set threshold, proceed to step (5) for local processing. If the execution confidence is less than the set threshold, proceed to step (4) for cloud processing. (4) The cloud planner receives the task text instructions, analyzes the task requirements through the cloud's large language model, generates an execution plan, decomposes the execution plan into subtasks and converts them into task text instructions in natural language form, and then sends them to the edge executor through the communication protocol; (5) The edge executor on the vehicle receives the task text instruction, sends the task text instruction as the user instruction to the fine-tuning large model, and outputs the user instruction that meets the needs of the car owner in different scenarios; and calls the vehicle-side API interface related to the scenario and returns the processing result; (6) The edge executor summarizes the returned processing results and makes a final response to the user's instructions.

2. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 1 is characterized in that: In step (1), the vehicle-mounted lightweight large model includes a text encoding Tokenizer, an Embedding layer, a decoder consisting of N decoder layers, and an MLP classifier.

3. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 2 is characterized in that: In the process of training and fine-tuning the large model, data sets of different usage scenarios are used, and guiding texts are designed for different usage scenarios. The guiding texts are encoded by the Tokenizer and Embedding layers and used as the initial prompt words. The text corresponding to the user instructions is encoded by the Tokenizer and Embedding layers and then concatenated with the initial prompt words to obtain the concatenated prompt words. During the training process, the parameters of the decoder in the on-board lightweight large model are frozen, and the parameters of the Embedding layer are trained.

4. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 3 is characterized in that: The guiding text is encoded as the initial prompt word, and the formula is: E p =SoftEmbedding(Tokenizer(t)) Among them, E p represents the initial prompt word, and t represents the designed guiding text.

5. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 4 is characterized in that: After encoding the text corresponding to the user instruction, concatenate it with the initialization prompt word to obtain the concatenated prompt word. The formula is: E i =Concat(SoftEmbedding(Tokenizer(t i )),E p ) Among them, E i Indicates the final input of the splicing prompt word of the lightweight large model on board; t i Represents the input text in the dataset sample, that is, the user instruction; Concat represents the concatenation operation.

6. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 1 is characterized in that: In step (3), the formula for executing confidence is as follows: In the formula, Representing a large language model based on context Get the output probability of the next word; i represents the sequence number of the context after being encoded by the text encoding Tokenizer; x u and x n represents yes or no, corresponding to whether the answer is encoded by the text encoding Tokenizer; C represents the confidence level, C n Indicates the confidence of negation, C y Indicates positive confidence, that is, execution confidence.

7. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 1 is characterized in that: In step (3), if the execution confidence is less than the set threshold, the vehicle terminal sends an HTTPS request to the cloud planner in JSON format, and the request content includes the original task description and parameters.

8. The end-cloud collaborative vehicle interaction method for multiple task scenarios according to claim 1 is characterized in that: In step (4), the cloud planner includes a task parser, a plan generator, and a task decomposer, all of which are implemented through a large language model in the cloud.

Images