Task execution method and device for vehicle-mounted environment, equipment and storage medium

Abstract

The invention discloses a task execution method and device for a vehicle-mounted environment, equipment and a storage medium, and relates to the technical field of vehicle-mounted interaction, and the method comprises the steps: obtaining a current task target and current screen state information for controlling a vehicle machine screen; based on a pre-constructed candidate action space, the current task target and the current screen state information, constructing verification prompt information conforming to a preset template; inputting the verification prompt information into a target verification model for action verification to obtain an action verification score; determining a target action from the candidate action space according to the action verification score; and executing the target action to complete the current task target, reducing the delay of the task by changing the large language model from generating a long sequence to outputting an action score, and selecting an optimal action by evaluating all possible actions, thereby reducing illusion and errors when the large language model is directly generated, and improving the success rate of the task.

Description

Technical Field

[0001] The present application relates to the field of in-vehicle interaction technology, and in particular to a task execution method, apparatus, device, and storage medium in an in-vehicle environment. Background Art

[0002] Existing in-car voice systems struggle to handle complex commands for non-deeply customized applications (e.g., apps and mini-programs downloaded from the app store), especially those involving multi-step, cross-application task chains (e.g., searching for information followed by a purchase or navigation). This lack of deep understanding of user intent leads to a low execution success rate.

[0003] In addition, many current solutions are based on large language models (LLMs) to directly generate operation sequences, which usually requires a long inference time. In addition, directly generating operations may produce hallucinations or incorrect steps, resulting in longer task execution time or errors in task execution. Summary of the Invention

[0004] The main purpose of this application is to provide a task execution method, device, equipment and storage medium in a vehicle-mounted environment, aiming to solve the current technical problems of low success rate and efficiency of vehicle-mounted task execution.

[0005] To achieve the above objectives, the present application proposes a method for executing tasks in a vehicle-mounted environment, the method comprising:

[0006] Get the current task target and current screen status information for controlling the vehicle screen;

[0007] Constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information;

[0008] Inputting the verification prompt information into a target verification model to perform action verification and obtain an action verification score;

[0009] determining a target action from the candidate action space according to the action verification score;

[0010] Execute the target action to complete the current task goal.

[0011] In one embodiment, the top layer of the target verification model includes a feedforward neural network. Construction of the target verification model includes:

[0012] Acquire vehicle-mounted task execution trajectory data, and construct four-tuple sample data according to the vehicle-mounted task execution trajectory data;

[0013] Labeling the four-tuple sample data to obtain labeled sample data;

[0014] Obtaining a self-correction training data set based on the labeled sample data;

[0015] Training the initial large language model using the self-correcting training data set and the four-tuple sample data to obtain a pre-trained large language model;

[0016] A feedforward neural network is added to the top layer of the pre-trained large language model, and the vector value output by the pre-trained large language model is mapped to an action verification score to obtain a target verification model.

[0017] In one embodiment, the step of constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information includes:

[0018] Obtain multiple candidate actions based on a pre-constructed candidate action space;

[0019] Get the architecture information of the preset template;

[0020] Verification prompt information that conforms to the architecture information is constructed using the plurality of candidate actions, the current task target, and the current screen state information.

[0021] In one embodiment, the step of determining a target action from the candidate action space according to the action verification score includes:

[0022] sorting the candidate actions in the candidate action space according to the action verification score to obtain a sorting result;

[0023] Taking the candidate action with the highest action verification score in the sorting result as the reference action;

[0024] Performing a security policy check on the reference action;

[0025] When the security policy check of the reference action passes, the reference action is used as the target action.

[0026] In one embodiment, the step of performing a security policy check on the reference action includes:

[0027] Obtain real-time vehicle dynamic data, environmental perception information, and driver status information;

[0028] matching a corresponding safety rule from a predefined safety rule library according to at least one of the vehicle real-time dynamic data, the environmental perception information, and the driver status information;

[0029] A security policy check is performed on the reference action using the security rule.

[0030] In one embodiment, after the step of executing the target action to complete the current task goal, the method further includes:

[0031] When the current task target is completed, obtaining the execution action track and updating the screen status information;

[0032] Summarizing and structuring the target action and the updated screen state to obtain a structured interface information summary;

[0033] Updating working memory through the execution action trajectory and the structured interface information summary;

[0034] Accordingly, the step of constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information includes:

[0035] Verification prompt information that conforms to a preset template is constructed based on the pre-constructed candidate action space, the working memory, the current task goal, and the current screen state information.

[0036] In one embodiment, constructing the candidate action space includes:

[0037] Extract all discrete interactive actions from the car screen status;

[0038] Get non-explicit vehicle control actions;

[0039] A candidate action space is constructed according to the interactive action and the non-explicit vehicle control action.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a task execution device in a vehicle-mounted environment, wherein the task execution device in the vehicle-mounted environment comprises:

[0041] The acquisition module is used to obtain the current task target and current screen status information for controlling the vehicle screen;

[0042] A construction module, configured to construct verification prompt information that conforms to a preset template based on a pre-constructed candidate action space, the current task goal, and the current screen state information;

[0043] A verification module, configured to input the verification prompt information into a target verification model to perform action verification and obtain an action verification score;

[0044] a determination module, configured to determine a target action from the candidate action space according to the action verification score;

[0045] The execution module is used to execute the target action to complete the current task goal.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a task execution device for a vehicle-mounted environment, which includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, and the computer program is configured to implement the steps of the task execution method for the vehicle-mounted environment as described above.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the task execution method in the vehicle environment as described above are implemented.

[0048] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the task execution method in the vehicle environment as described above.

[0049] One or more technical solutions proposed in this application obtain the current task goal and current screen status information for controlling the vehicle screen; construct verification prompt information that conforms to a preset template based on a pre-constructed candidate action space, the current task goal and the current screen status information; input the verification prompt information into a target verification model for action verification to obtain an action verification score; determine a target action from the candidate action space based on the action verification score; execute the target action to complete the current task goal. By changing the large language model from generating long sequences to outputting action scores, the task delay is reduced, and the optimal action is selected by evaluating all possible actions, thereby reducing illusions and errors during direct generation of the large language model and improving the task success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 A flowchart illustrating a first embodiment of the method for executing a task in a vehicle environment of the present application;

[0053] Figure 2A flowchart illustrating a second embodiment of the method for executing a task in a vehicle environment of the present application;

[0054] Figure 3 A flowchart illustrating a third embodiment of the method for executing a task in a vehicle environment of the present application;

[0055] Figure 4 A schematic diagram of the interactive process of completing a task execution process in an in-vehicle environment by interaction between the cloud and the vehicle side, provided in an embodiment of the in-vehicle task execution method of the present application;

[0056] Figure 5 A brief flowchart of an embodiment of a method for executing a task in a vehicle environment of the present application is provided;

[0057] Figure 6 This is a schematic diagram of the module structure of the task execution device in the vehicle environment of an embodiment of the present application;

[0058] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the task execution method in the vehicle environment in the embodiment of the present application.

[0059] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0061] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] The main solution of the embodiment of the present application is: obtaining the current task goal and current screen status information for controlling the vehicle screen; constructing verification prompt information that conforms to a preset template based on a pre-constructed candidate action space, the current task goal and the current screen status information; inputting the verification prompt information into a target verification model for action verification to obtain an action verification score; determining a target action from the candidate action space based on the action verification score; and executing the target action to complete the current task goal.

[0063] Existing technologies, such as SeeAct and AutoDroid, use large models as generators. Their core is to receive user instructions and the current screen status (such as screenshots, GUI structure information, and XML information). Leveraging the understanding and reasoning capabilities of LLMs, they directly generate the next action (such as the thought process combined with a specific action description, such as "click coordinates (x, y)" or "enter text abc"). Prompt engineering technologies such as Chain of Thought (CoT) and ReACT are often used to enhance reasoning capabilities. The drawback is that the autoregressive generation process is time-consuming and error-prone. Alternatively, visual models such as Ferret UI or multimodal models are used to understand the car screen content, locate elements, and assist the large model in making decisions. This improves screen comprehension, but large language models are still used to generate operation sequence patterns, leading to latency and reliability issues.

[0064] This application provides a solution, the core idea of ​​which is to change the role of LLM from action generator to action verifier to solve the delay and reliability problems of existing technologies, especially in non-customized application scenarios in vehicles.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions, or a task execution device in an in-vehicle environment, such as an in-vehicle controller. This embodiment and the following embodiments will be described below using a task execution device in an in-vehicle environment as an example.

[0066] Based on this, the embodiment of the present application provides a method for executing a task in a vehicle environment, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the task execution method in the vehicle environment of the present application.

[0067] In this embodiment, the task execution method in the vehicle environment includes steps S10 to S50:

[0068] Step S10: Acquire the current task target for controlling the vehicle screen and the current screen status information.

[0069] It should be noted that the current task target is a task issued by the user in the vehicle environment and is used to control the vehicle screen, specifically the program on the vehicle screen or the device on the vehicle. The current task target can be obtained by collecting user voice data.

[0070] The current screen state information is a simplified user interface (UI) representation of the current screen, such as a simplified HTML-like structure or list of key elements. This information is not a full screenshot, thus reducing the number of tokens (words or characters) and instructions. The current screen state information can be obtained using the Accessibility Tree or XML layout information.

[0071] Step S20: constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information.

[0072] It should be noted that the candidate action space is all actions related to operating the vehicle screen that are constructed in advance. The candidate action space brings together all possible, discrete interaction actions and default actions that are critical to completing tasks and controlling equipment.

[0073] The preset template is a pre-defined template, through which the candidate action space, the current task goal and the current screen state information can be converted into structural information that meets the verification requirements of the target verification model.

[0074] The verification prompt is the verification prompt for the input target verification model. For example, the verification prompt is: the task is "navigate to the nearest charging station", the current screen is in the map application, and the candidate action is "click the search button". Is this action helpful in completing the task? (Yes / No).

[0075] In a feasible implementation, the step of constructing the action space may include: extracting all discretized interactive actions from the vehicle screen state; obtaining non-explicit vehicle control actions; and constructing a candidate action space based on the interactive actions and the non-explicit vehicle control actions.

[0076] It is understandable that at each given interaction step t, all possible, discrete interaction actions can be extracted from the current vehicle screen state, such as clicking button A, clicking list item B, long pressing icon C, text entry in input box D, scrolling in area E, etc. To enhance the robustness of GUIs with non-standard or special rendering characteristics (such as applets and game interfaces), a lightweight visual model can be introduced to assist in element detection and attribute recognition.

[0077] At any given step t, the goal is to extract a set of action spaces A k (t), k represents the task represented by the prompt proposed by the user this time, and these actions represent the UI state S related to the current step t. k (t) potential interactions. To identify UI elements, directly analyze the XML representation of the UI state Sk (t) When the keyboard is displayed on the screen, a single key corresponds to a UI element, which significantly increases the size of the action space. Therefore, when using the verification-driven GUI Agent, you can disable the standard keyboard and directly input text through the Android Debug Bridge (ADB) command.

[0078] It should be noted that in addition to UI-related operations, there are also some non-explicit in-vehicle control actions, that is, default actions that are crucial to completing tasks and device control, such as: opening a specified APP, navigating home / to the company, returning to the previous level, returning to the home screen, waiting for the device to respond, informing that the task is completed, and answering the user's questions about the task (under the premise of driving safety). Therefore, these actions can be supplemented to the action space. Through interactive actions and the non-explicit in-vehicle control actions, a finite and enumerable candidate action set space can be generated. This discretized action space is the basis for the verification-driven model to work efficiently.

[0079] In a feasible implementation, step S20 may include steps A11 to A13:

[0080] Step A11: Obtain multiple candidate actions based on the pre-constructed candidate action space.

[0081] It should be noted that, since there are multiple candidate actions in the candidate action space, multiple candidate actions can be obtained based on the pre-constructed candidate action space.

[0082] Step A12: Obtain the architecture information of the preset template.

[0083] It should be noted that the architecture information of the preset template contains key context information, which is specifically represented as follows:

[0084] Role definition: Agent's role and basic operating principles;

[0085] Task objective: the user instruction that needs to be completed at present;

[0086] Working memory: Contains historical action trajectories, key intermediate states, and structured UI information summaries extracted from previous steps (generated by LLM or rules);

[0087] UI state description: A streamlined representation of the current screen (e.g., an HTML-like structure or list of key elements for simplified HTML) rather than a raw screenshot to reduce the number of tokens. Instructions:

[0088] Instructions: Explicitly request the verifier LLM to output a judgment in a specific format (for example, a strict "yes" or "no" response);

[0089] Question to be verified: Is this action useful in completing the task?

[0090] The structure in the preset template includes role definition, task objectives, working memory, vehicle screen status description, instructions, and questions to be verified.

[0091] Step A13: Construct verification prompt information that conforms to the architecture information through the plurality of candidate actions, the current task target, and the current screen status information.

[0092] In the specific implementation, each candidate action can be connected to the current task target and the current screen state information to form a verification prompt information that conforms to the architecture information in the preset template. k (t), each action α∈A k (t) will be formatted into a predefined prompt template P to construct a corresponding validation prompt All subsequent formatted verification prompts, each set of prompt words corresponds to the action space A k Candidate actions α within (t).

[0093] Step S30: inputting the verification prompt information into the target verification model to perform action verification and obtain an action verification score.

[0094] It should be noted that the target verification model is a specially trained Verifier-based LLM. It can be a small language model optimized on the device or a model trained through device-cloud collaboration. The verification prompts for all generated candidate actions are fed into the target verification model for parallel (batch) evaluation, outputting an action verification score for each candidate action.

[0095] When performing action verification, prefix caching can be used: Automatic Prefix Caching, which is already available in libraries such as vLLM, can accelerate verification optimization. Since most of the content of all verification prompts in the same step (task objectives, historical records (if any), UI status) is the same, only the candidate actions are different. Therefore, KV Cache technology is used to reduce repeated calculations by caching intermediate calculation results, significantly accelerating the reasoning process of batch verification. At the same time, the verifier LLM only needs to perform a single forward propagation and obtain the probability distribution of the first token, eliminating the need for time-consuming autoregressive decoding, further improving efficiency.

[0096] The output of the Verifier LLM (Verified Large Language Model) is designed to be very simple. The output only needs to generate a very small number of tokens (usually 1) to express the judgment. By analyzing the probability or logits of the first generated token (such as the logits value of the "yes" token), a scalar verification score is mapped to each action a. The action verification score reflects the expected contribution of this action to completing the task.

[0097] Step S40: determining a target action from the candidate action space according to the action verification score.

[0098] It should be noted that a target action can be selected from the candidate action set in the candidate action space based on the action verification score of each candidate action. The target action is the action with the highest action verification score and passes the security policy check.

[0099] Step S50: Execute the target action to complete the current task goal.

[0100] It should be noted that after the target action is determined, the target action can be executed, and the selected and completed action is executed by simulating clicks, scrolling and other corresponding operations, thereby completing the current task goal.

[0101] In specific implementations, when executing the target action, there are also actions that require additional parameters, such as "opening other applications", "entering text", "answering user queries for tasks", etc., which may require calling a large language model. This large language model generates specific content for the lightweight small model LLM, namely the name of the application to be opened, the text to be entered, and the content of the answer. This step is separated from verification and only occurs when necessary.

[0102] It should be noted that, in order to improve the efficiency and accuracy of task completion, the vehicle screen status may be updated after the target action is completed. In a feasible implementation, after step S50, steps S51 to S53 are further included:

[0103] Step S51: When the current task target is completed, the execution action track is obtained and the screen status information is updated.

[0104] It is understandable that when the current task target is completed, the execution result will generate a new UI state S(t+1), so the execution action trajectory during the execution of the target action can be obtained and the screen status information can be updated.

[0105] Step S52: Summarize and structure the target action and the updated screen state to obtain a structured interface information summary.

[0106] It should be noted that after executing an action, the working memory can be updated by capturing the new UI state. Rather than just recording the action, the LLM is used to summarize and structure the executed action and the resulting state changes, resulting in a structured interface information summary.

[0107] Step S53: updating the working memory through the execution action trajectory and the structured interface information summary.

[0108] It should be noted that work can be updated through the execution action trajectory and structured interface information summary. If it is the first time to perform an action, the execution action trajectory and structured interface information summary are used to create working memory. For example, record "After clicking the search button, a list of search results appears." Working memory should include the task trajectory, key extracted information (such as search result items, selected products), and important in-vehicle context (such as current time, location, remaining battery / fuel level, navigation route information).

[0109] Furthermore, step S20 may include: constructing verification prompt information that complies with a preset template based on the pre-constructed candidate action space, the working memory, the current task goal, and the current screen state information.

[0110] It should be noted that if working memory exists or is updated, verification prompts that conform to the schema information in the preset template can be constructed using the candidate action space, working memory, current task objectives, and current screen state information. Working memory includes historical action trajectories, key information states, and structured UI information summaries extracted from previous steps (generated by LLM or rules), thereby improving the efficiency of constructing verification prompts.

[0111] This embodiment provides a method for executing tasks in an in-vehicle environment, which comprises obtaining the current task goal and current screen status information for controlling the vehicle screen; constructing verification prompt information that conforms to a preset template based on a pre-constructed candidate action space, the current task goal, and the current screen status information; inputting the verification prompt information into a target verification model for action verification to obtain an action verification score; determining a target action from the candidate action space based on the action verification score; and executing the target action to complete the current task goal. By changing the large language model from generating long sequences to outputting action scores, task latency is reduced, and the optimal action is selected by evaluating all possible actions, thereby reducing illusions and errors during direct generation of the large language model and improving the task success rate.

[0112] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2Before step S30, the task execution method in the vehicle environment further includes steps S21 to S25:

[0113] In this embodiment, the top layer of the target verification model includes a feed-forward neural network.

[0114] It should be noted that in the architecture of the target verification model, a lightweight trainable MLP head, i.e., a feedforward neural network, can be added to the top layer of the target verification model to map the token logits output by the LLM to the final scalar action score. Parameter fine-tuning techniques such as Q-LoRa can be used for training.

[0115] Step S21: Obtain vehicle-borne task execution trajectory data, and construct four-tuple sample data according to the vehicle-borne task execution trajectory data.

[0116] It should be noted that based on the collected vehicle-mounted task execution trajectories, including success and failure paths, a large amount of quadruple sample data is constructed.

[0117] The four-tuple sample data includes: state, context, successful action, and failed action, namely State(t), Context(t), PositiveAction(t), and NegativeAction(t).

[0118] State(t) represents the state of the current user interface (UI) directly perceived by the agent at step t. It mainly provides information about "what is on the screen now." The format is the HTML-like format mentioned in the proposal, or a list of extracted key interactive elements and their attributes (for example: [Button(text='navigation',clickable=True),InputField(id='search_box',editable=True),List(scrollable=True,items=...)]). Visual features are descriptors of key visual areas or elements extracted by the lightweight visual model. State(t) provides the direct object and environment for performing actions. The agent needs to analyze State(t) to identify which buttons can be clicked, which text boxes can be input, etc. PositiveAction(t) and NegativeAction(t) usually act directly on the elements represented in State(t).

[0119] Context(t) represents all the background and environmental information relevant to making the correct decision at step t, in addition to the current screen state, State(t). It primarily encompasses information about "what we are doing," "where we are coming from," and "what the surrounding environment is like." This includes task objectives, working memory, a history of actions, and vehicle-specific context (such as vehicle state, location, time, and user information), providing the necessary constraints and context for executing actions.

[0120] PositiveAction(t) and NegativeAction(t) represent the execution of correct actions and actions other than correct actions. VerifierLLM requires receiving both State(t) and Context(t) to accurately determine whether PositiveAction(t) is a better choice than NegativeAction(t) in order to achieve the task objectives and meet various constraints under the current interface state (State_t).

[0121] Step S22: labeling the four-tuple sample data to obtain labeled sample data.

[0122] It should be noted that when labeling quadruple sample data, in order to improve the accuracy of labeling and reduce the cost of labeling, labeling can be performed through human-machine collaboration, specifically by using the uncertainty of the verifier LLM itself to guide manual labeling.

[0123] In a feasible implementation, step S22 may include steps B11 to B13:

[0124] Step B11: Perform task verification using the initial large language model to obtain a historical action verification score.

[0125] In specific implementation, the Verifier Agent in training can be allowed to try to perform the task first. The Verifier Agent in training is the initial large language model, so as to obtain the historical action verification score.

[0126] Step B12: Calculate the entropy value of the historical action verification score distribution.

[0127] It is understandable that the entropy value of the verification score distribution of each historical action can be calculated as follows:

[0128] ε κ (t) = -τ(α κ (t))log(τ(α κ (t)))|α κ (t)∈A κ (t)

[0129] Among them, ε is the entropy value, τ represents the estimated distribution score for a specific verification action, and α κ (t) is the action performed. k (t) is the action space. Entropy is used to measure the uncertainty of the score distribution.

[0130] Step B13: When the entropy value is greater than or equal to a preset threshold, the sample data in the four-tuple sample data is marked as manual review, so as to perform manual review on the sample data to obtain labeled sample data.

[0131] In practice, the preset threshold can be set in advance. To set the preset threshold, entropy data can be collected first. This dataset can come from: labeled data from the previous iteration, a validation set specifically used for model evaluation and threshold setting, or a small amount of data collected early in the current round. The median is then calculated, combining the entropy values ​​of all decision steps collected in the previous step to form a list of entropy values. The median of this entropy list is calculated. The median is the value in the middle of the sorted values ​​(if there is an even number of data points, the average of the two middle values ​​is usually taken). This threshold is not fixed. As the validator LLM improves its capabilities over multiple rounds of training, the overall entropy distribution may shift toward lower values. Therefore, before the start of each iteration, the median entropy threshold based on the latest data is recalculated and set.

[0132] When the entropy value exceeds a preset threshold, the data is automatically marked for manual labeling and review. Labelers only need to check and correct the labeled steps and select the correct actions, thereby reducing labeling costs, iteratively improving agent capabilities, and expanding the training dataset. The labeling environment must consider driving simulation or real-world road test scenarios. At the same time, the collected data is used for the next round of training to improve the capabilities of the validator LLM. As its capabilities increase, the entropy of decisions decreases, and the proportion of manual intervention required decreases, thereby reducing labor costs and achieving an increase in data and capabilities. The human-machine collaborative labeling mechanism significantly reduces the cost and cycle of collecting and labeling large-scale, high-quality training data for in-vehicle scenarios, enabling rapid iteration and generalization of agent capabilities.

[0133] Step S23: obtaining a self-correction training data set based on the labeled sample data.

[0134] It should be noted that states can be identified from labeled sample data, and corresponding reverse / corrective actions can be defined for these error states. For example, for incorrect input text, the corresponding reverse action is to clear the text or roll back. Self-correction training pairs are constructed: error state, context, correct action, and other actions, where other actions are any non-corrective actions in the error state. This results in a self-correction training set.

[0135] Step S24: training the initial large language model using the self-correcting training data set and the four-tuple sample data to obtain a pre-trained large language model.

[0136] It's important to note that the self-correction training set can be mixed into the total training set quadruple sample data at a certain ratio, for example, 2.5%, to enhance the agent's robustness and ability to recover from errors. By leveraging examples in the annotated data that recover from an error state to the correct trajectory (for example, executing a "return" action after an erroneous click), the training model can select corrective actions such as "return" or "cancel" when it deviates from the task. This employs a contrastive learning approach to train the initial large language model to distinguish between "good" and "bad" actions, rather than directly predicting an absolute score. This ultimately results in a well-trained pre-trained large language model.

[0137] Step S25: Add a feedforward neural network on the top layer of the pre-trained large language model, map the vector value output by the pre-trained large language model into an action verification score, and obtain a target verification model.

[0138] In the specific implementation, a lightweight trainable MLP head is added to the top layer of a pre-trained LLM such as Qwen 2.5VL-7B, so as to map the token logits (vector values) output by the LLM to the final scalar action score to obtain the target verification model.

[0139] τ represents the estimated distribution score for a specific verification action, specifically a trainable MLP layer, V is the pre-trained verifier LLM, p α is an input prompt, which is as follows:

[0140] τ(α)=MLP·V(p α )

[0141] In the specific implementation, in order to improve the model training effect, a loss function can also be set. The loss function is designed as follows:

[0142]

[0143] The given task is k, in the action space A k In (t), the correct action of the mark is The remaining action sets constitute the negative samples A ′ k (t), where

[0144] This embodiment obtains on-board task execution trajectory data and constructs four-tuple sample data based on the on-board task execution trajectory data; labels the four-tuple sample data to obtain labeled sample data; obtains a self-correcting training dataset based on the labeled sample data; trains an initial large language model using the self-correcting training dataset and the four-tuple sample data to obtain a pre-trained large language model; adds a feedforward neural network on top of the pre-trained large language model, and maps the vector values ​​output by the pre-trained large language model into action verification scores to obtain a target verification model. The generated target verification model evaluates all possible actions and selects the optimal one, reducing illusions and errors during direct LLM generation. When constructing the target verification model, combined with paired training and self-correcting training for on-board tasks, it can complete more complex cross-application automation tasks, especially when processing non-customized applications. By changing the role of the large language model from an action generator to an action verifier and continuously training it, a target verification model capable of action verification is generated, thereby improving the latency and reliability of task execution.

[0145] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 , step S40 includes steps S401 to S404:

[0146] Step S401: sorting the candidate actions in the candidate action space according to the action verification score to obtain a sorting result.

[0147] It should be noted that after obtaining the action verification score, each candidate action in the candidate action space may be sorted according to the action verification score, thereby obtaining a sorting result of each candidate action according to the action verification score.

[0148] Step S402: taking the candidate action with the highest action verification score in the sorting result as a reference action.

[0149] In a specific implementation, the candidate action with the highest action verification score in the sorting results can be used as the reference action, that is, the action with the highest score is preliminarily selected.

[0150] Verification prompts are batched and fed into the verifier LLM V, which is a fine-tuned version of the pre-trained LLM. After evaluating all candidate actions in the action space, the action selected for execution is the one with the highest score, at step t:

[0151]

[0152] Step S403: Perform a security policy check on the reference action.

[0153] It is understandable that before the action is finally executed, the selected action must pass the security policy check. Therefore, the security policy check can be performed on the reference action. In a feasible implementation, the step of performing the security policy check on the reference action includes:

[0154] Obtain real-time vehicle dynamic data, environmental perception information, and driver status information;

[0155] matching a corresponding safety rule from a predefined safety rule library according to at least one of the vehicle real-time dynamic data, the environmental perception information, and the driver status information;

[0156] A security policy check is performed on the reference action using the security rule.

[0157] It should be noted that safety strategy checks are based on one or more of the following: real-time vehicle dynamics data, environmental perception information, and driver status information. Real-time vehicle dynamics data includes speed, gear, driving mode, turn signal status, etc. Environmental perception information (if available) includes obstacle and traffic light information from the Advanced Driver Assistance Systems (ADAS). Driver status monitoring (if available) includes vision, fatigue, etc.

[0158] In specific implementations, a predefined security rule library can be constructed in advance. For example, actions requiring text input are prohibited while the vehicle is in motion, actions that may interrupt navigation or emit disruptive sounds are restricted, and pop-up window operations that cover the main screen area require caution. Corresponding security rules can be matched from the predefined security rule library based on at least one of the vehicle's real-time dynamic data, environmental perception information, and driver status information. Thus, a security policy check is performed on the reference action through security comparison. For example, if the current reference action is to perform text input, but the current security rule prohibits the execution of text input, the security policy check of the reference action will fail.

[0159] If the security policy check of the reference action fails, the reference action will be treated as a high-risk action. The high-risk action can be directly rejected or fed back to the user for confirmation whether to execute it.

[0160] Step S404: When the security policy check of the reference action passes, the reference action is used as the target action.

[0161] It should be understood that if the security policy check of the reference action passes, the reference action can be used as the target action that can be finally executed.

[0162] It should be noted that the overall task execution process in the vehicle environment can be deployed entirely on the cloud or on the vehicle side, or partially on the vehicle side and the other part on the cloud side. For example, the verifier LLM model, action space construction module and security policy module are deployed on the vehicle side to ensure low-latency response, maintain short-term working memory and execute actions. At the same time, the cloud can undertake the initial planning and disassembly of complex tasks using a large-parameter LLM, periodic training and fine-tuning updates of the verifier LLM model, storage and query of large-scale knowledge, and verification of some high-computing power requirements (if the computing power on the end side is insufficient). The end and cloud communicate through a defined protocol, taking into account both efficiency and privacy, such as Figure 4 As shown, Figure 4 This is a diagram of the interaction process between the cloud and the vehicle to complete the task execution process in the vehicle environment. Users can interact with the vehicle through voice or touch screen. The vehicle is deployed with the GUI Agent end-side core component and the vehicle operating system HMI (displaying the user interface and receiving user input). It is parsed and the action space is constructed (including GUI actions and vehicle default actions) through the GUI parser (Android Accsessibility / lightweight visual model). The verifier LLM, that is, the target verification model, performs batch verification + prefix caching to select actions and obtain reference actions. The driving safety policy model is used to check the safety policy. After passing the check, the action is executed. At the same time, the vehicle data / sensor interface receives additional action parameters to complete the execution of the entire action. The vehicle computer also stores working memory, including (task status, historical actions, context). The end-cloud communication module can also transmit actions to the cloud. The cloud can perform complex task planning (processing long chains / fuzzy instructions) and knowledge base management (user preferences, etc.) and output them to the vehicle computer. The cloud includes a model training and management module and a data processing and labeling module. The model training and management module includes verifier LLM training and model version deployment, and deploys the model to the vehicle computer. The data processing and labeling module includes data storage and a human-computer collaborative labeling interface / platform, so that the model can be trained through training data.

[0163] This embodiment sorts the candidate actions in the candidate action space according to the action verification score to obtain a sorting result; takes the candidate action with the highest action verification score in the sorting result as a reference action; performs a security policy check on the reference action; and when the security policy check of the reference action passes, takes the reference action as the target action. By sorting the candidate actions in the candidate action space, the optimal action can be efficiently screened out. Using the action verification score as the basis for sorting can ensure that the best candidate action is selected, avoiding unnecessary repeated attempts or inefficient decisions. Adding a security policy check to the selection of the reference action can effectively prevent actions that do not meet security requirements from being executed, ensuring the safety and reliability of the system operation. By performing verification scoring and security checks on each candidate action, the occurrence of erroneous actions can be reduced, the stability of the system during execution can be improved, and potential risks or losses can be reduced.

[0164] For example, in order to help understand the implementation process of the task execution method of the vehicle environment obtained by combining this embodiment with the above embodiment 1, please refer to Figure 5 , Figure 5 A simplified flowchart of a task execution method for an in-vehicle environment is provided. Specifically, the following steps are performed: ① A candidate action space (button A, button B, button C, and button D) is constructed by extracting actions from the vehicle-mounted system and supplementing them with default actions. ② A verification prompt template is constructed for each candidate action using the candidate action space, UI extraction (XML, HTML, etc.), and working memory. The verification prompt template includes a role definition, a task objective, working memory, a UI described in XML, and specific instructions (requiring only yes or no answers and whether action X is useful for completing the task). ③ The verification prompt template is input into an AI target verification model, and batch-scored using a validator with a prefix cache. The highest-scoring action is initially selected. After the highest-scoring action passes the safety policy of a safe driving policy module, the action is completed and executed. ⑤ After the action is completed, the working memory is updated to obtain an updated vehicle-mounted system screen state, and steps ①-⑤ are iteratively executed.

[0165] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the task execution method of the vehicle-mounted environment of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0166] This application also provides a task execution device for a vehicle-mounted environment, please refer to Figure 6 , the task execution device of the vehicle environment includes:

[0167] The acquisition module 10 is used to acquire the current task target for controlling the vehicle screen and the current screen status information.

[0168] The construction module 20 is used to construct verification prompt information that complies with a preset template based on the pre-constructed candidate action space, the current task goal and the current screen state information.

[0169] The verification module 30 is used to input the verification prompt information into the target verification model to perform action verification and obtain an action verification score.

[0170] The determination module 40 is configured to determine a target action from the candidate action space according to the action verification score.

[0171] The execution module 50 is used to execute the target action to complete the current task goal.

[0172] The in-vehicle task execution device provided by this application, which utilizes the in-vehicle task execution method described in the aforementioned embodiments, can address the current technical issues of low success rate and efficiency in in-vehicle task execution. Compared to the prior art, the in-vehicle task execution device provided by this application offers the same beneficial effects as the in-vehicle task execution method described in the aforementioned embodiments. Other technical features of the in-vehicle task execution device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0173] In one embodiment, the top layer of the target verification model includes a feedforward neural network, and the construction module 20 is further used to obtain vehicle-mounted task execution trajectory data and construct four-tuple sample data based on the vehicle-mounted task execution trajectory data; label the four-tuple sample data to obtain labeled sample data; obtain a self-correcting training data set based on the labeled sample data; train the initial large language model through the self-correcting training data set and the four-tuple sample data to obtain a pre-trained large language model; add a feedforward neural network to the top layer of the pre-trained large language model, map the vector value output by the pre-trained large language model to an action verification score, and obtain a target verification model.

[0174] In one embodiment, the construction module 20 is also used to obtain multiple candidate actions based on a pre-constructed candidate action space; obtain the architecture information of a preset template; and construct verification prompt information that conforms to the architecture information through multiple candidate actions, the current task objectives and the current screen status information.

[0175] In one embodiment, the determination module 40 is further used to sort the candidate actions in the candidate action space according to the action verification score to obtain a sorting result; use the candidate action with the highest action verification score in the sorting result as a reference action; perform a security policy check on the reference action; and when the security policy check of the reference action passes, use the reference action as the target action.

[0176] In one embodiment, the determination module 40 is also used to obtain real-time dynamic data of the vehicle, environmental perception information and driver status information; match corresponding safety rules from a predefined safety rule library based on at least one of the real-time dynamic data of the vehicle, the environmental perception information and the driver status information; and perform a safety policy check on the reference action through the safety rules.

[0177] In one embodiment, the task execution device in the vehicle environment also includes an update module, which is used to obtain the execution action trajectory and update the screen status information when the current task target is completed; summarize and structure the target action and the updated screen status to obtain a structured interface information summary; update the working memory through the execution action trajectory and the structured interface information summary; and construct verification prompt information that conforms to a preset template based on a pre-constructed candidate action space, the working memory, the current task target, and the current screen status information.

[0178] In one embodiment, the construction module 20 is further used to extract all discrete interactive actions from the vehicle screen state; obtain non-explicit vehicle control actions; and construct a candidate action space based on the interactive actions and the non-explicit vehicle control actions.

[0179] The present application provides a task execution device for a vehicle-mounted environment, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the task execution method for the vehicle-mounted environment in the above-mentioned embodiment one.

[0180] Reference below Figure 7, which shows a schematic diagram of the structure of a task execution device suitable for implementing an in-vehicle environment in an embodiment of the present application. The task execution device in the in-vehicle environment in an embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The task execution device in the vehicle environment shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0181] like Figure 7 As shown, the task execution device of the in-vehicle environment may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a ROM (Read Only Memory) 1002 or a program loaded from a storage device 1003 into a RAM (Random Access Memory) 1004. Various programs and data required for the operation of the task execution device of the in-vehicle environment are also stored in the RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, an LCD (Liquid Crystal Display), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. The communication devices 1009 can allow the task execution equipment in the vehicle environment to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows the task execution equipment in the vehicle environment with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have instead.

[0182] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0183] The in-vehicle task execution device provided by this application, employing the in-vehicle task execution method described in the aforementioned embodiments, can address the current technical issues of low success rate and efficiency in in-vehicle task execution. Compared to the prior art, the in-vehicle task execution device provided by this application achieves the same beneficial effects as the in-vehicle task execution method described in the aforementioned embodiments. Other technical features of this in-vehicle task execution device are the same as those disclosed in the aforementioned embodiments and are not further elaborated upon here.

[0184] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0185] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0186] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the task execution method in the vehicle-mounted environment in the above-mentioned embodiment.

[0187] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash memory), optical fiber, CD-ROM (CD-Read Only Memory, portable compact disk read-only memory), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0188] The computer-readable storage medium may be included in the task execution device in the vehicle-mounted environment; or may exist independently without being installed in the task execution device in the vehicle-mounted environment.

[0189] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the task execution device of the vehicle environment, the task execution device of the vehicle environment: obtains the current task target and current screen status information for controlling the vehicle screen; constructs verification prompt information that conforms to the preset template based on the pre-constructed candidate action space, the current task target and the current screen status information; inputs the verification prompt information into the target verification model for action verification to obtain an action verification score; determines the target action from the candidate action space according to the action verification score; and executes the target action to complete the current task target.

[0190] The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0191] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0192] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0193] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned in-vehicle task execution method. This computer-readable storage medium can address the current technical issues of low success rate and efficiency in in-vehicle task execution. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the in-vehicle task execution method provided in the aforementioned embodiments, and are not further elaborated here.

[0194] The present application also provides a computer program product, including a computer program, which implements the steps of the task execution method in the vehicle environment as described above when the computer program is executed by a processor.

[0195] The computer program product provided in this application can address the current technical issues of low success rate and efficiency in executing in-vehicle tasks. Compared to the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the in-vehicle task execution method provided in the above-mentioned embodiments, and will not be elaborated here.

[0196] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A task execution method in a vehicle environment, characterized in that: The task execution method of the vehicle-mounted environment includes: Get the current task target and current screen status information for controlling the vehicle screen; Constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information; Inputting the verification prompt information into a target verification model to perform action verification and obtain an action verification score; determining a target action from the candidate action space according to the action verification score; Execute the target action to complete the current task goal.

2. The method according to claim 1, wherein The top layer of the target verification model includes a feed-forward neural network. The construction of the target verification model includes: Acquire vehicle-mounted task execution trajectory data, and construct four-tuple sample data according to the vehicle-mounted task execution trajectory data; Labeling the four-tuple sample data to obtain labeled sample data; Obtaining a self-correction training data set based on the labeled sample data; Training the initial large language model using the self-correcting training data set and the four-tuple sample data to obtain a pre-trained large language model; A feedforward neural network is added to the top layer of the pre-trained large language model, and the vector value output by the pre-trained large language model is mapped to an action verification score to obtain a target verification model.

3. The method according to claim 1, wherein The step of constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information includes: Obtain multiple candidate actions based on a pre-constructed candidate action space; Get the architecture information of the preset template; Verification prompt information that conforms to the architecture information is constructed using the plurality of candidate actions, the current task target, and the current screen state information.

4. The method according to claim 1, wherein The step of determining a target action from the candidate action space according to the action verification score comprises: sorting the candidate actions in the candidate action space according to the action verification score to obtain a sorting result; Taking the candidate action with the highest action verification score in the sorting result as the reference action; Performing a security policy check on the reference action; When the security policy check of the reference action passes, the reference action is used as the target action.

5. The method according to claim 4, wherein The step of performing security policy checking on the reference action comprises: Obtain real-time vehicle dynamic data, environmental perception information, and driver status information; matching a corresponding safety rule from a predefined safety rule library according to at least one of the vehicle real-time dynamic data, the environmental perception information, and the driver status information; A security policy check is performed on the reference action using the security rule.

6. The method according to claim 1, wherein After the step of executing the target action to complete the current task goal, the method further includes: When the current task target is completed, obtaining the execution action track and updating the screen status information; Summarizing and structuring the target action and the updated screen state to obtain a structured interface information summary; Updating working memory through the execution action trajectory and the structured interface information summary; Accordingly, the step of constructing verification prompt information that conforms to a preset template based on the pre-constructed candidate action space, the current task goal, and the current screen state information includes: Verification prompt information that conforms to a preset template is constructed based on the pre-constructed candidate action space, the working memory, the current task goal, and the current screen state information.

7. The method according to any one of claims 1 to 6, characterized in that The construction of the candidate action space includes: Extract all discrete interactive actions from the car screen status; Get non-explicit vehicle control actions; A candidate action space is constructed according to the interactive action and the non-explicit vehicle control action.

8. A task execution device in a vehicle environment, characterized in that: The device comprises: The acquisition module is used to obtain the current task target and current screen status information for controlling the vehicle screen; A construction module, configured to construct verification prompt information that conforms to a preset template based on a pre-constructed candidate action space, the current task goal, and the current screen state information; A verification module, configured to input the verification prompt information into a target verification model to perform action verification and obtain an action verification score; a determination module, configured to determine a target action from the candidate action space according to the action verification score; The execution module is used to execute the target action to complete the current task goal.

9. A task execution device in a vehicle environment, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the task execution method for an in-vehicle environment according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the task execution method of the in-vehicle environment according to any one of claims 1 to 7 are implemented.

Images