A method, device, vehicle, medium, and product for vehicle-mounted applications.

Abstract

This invention relates to the field of vehicle software control technology, and discloses an in-vehicle application control method, device, vehicle, medium, and product. The method includes: acquiring user demand information; calling a large model to plan operation tasks based on the demand information and installed in-vehicle applications, wherein the operation tasks include several operation steps, each operation step describing a human-like operation on the in-vehicle application; and executing the operation tasks through an action execution module. This invention improves the control accuracy of in-vehicle applications.

Description

Technical Field

[0001] This invention relates to the field of vehicle software control technology, specifically to an in-vehicle application control method, device, vehicle, medium, and product. Background Technology

[0002] As the capabilities of large-scale models in the multimodal domain continue to improve, human-computer interaction has evolved from the most primitive input methods such as mice and keyboards to more natural forms of interaction, such as handwriting recognition and voice recognition. Consequently, vehicle intelligent control technology is developing rapidly, with more and more applications such as audio-visual entertainment, shopping, and food delivery being installed in in-vehicle systems, transforming the car from a mere means of transportation into a more comprehensive system. Improving the interactive experience between drivers and passengers and in-vehicle applications is increasingly worthy of in-depth consideration. In recent years, some technologies have used large-scale models to recognize user voice commands or text, and then used intelligent agents to simulate user clicks, inputs, drags, and other operations to trigger applications on the in-vehicle screen, thus replacing manual screen operations without intruding into the application's background or calling the application's API. However, the intelligence level of these technologies is still not high enough; large-scale models often need to consult the user at every step of the operation, resulting in a poor user experience and an inability to address user needs in a single step. Therefore, how to improve the control accuracy of in-vehicle applications is a problem worthy of in-depth research. Summary of the Invention

[0003] In view of this, the present invention provides a vehicle application control method, device, vehicle, medium and product to solve the problem of low control accuracy in vehicle applications.

[0004] In a first aspect, the present invention provides a vehicle application control method, the method comprising: acquiring user demand information; calling a large model to plan an operation task based on the demand information and the installed vehicle application, the operation task comprising several operation steps, each operation step being used to describe a human-like operation on the vehicle application; and executing the operation task through an action execution module.

[0005] Based on the aforementioned technical means, after acquiring user needs, a large model is invoked to automatically plan and execute step-by-step tasks that include human-like operations. This solves the problems of frequent user queries and low intelligence in existing in-vehicle application control technologies. It eliminates the need for manual user operation and API calls. Through the human-like operation planning of in-vehicle applications using a large model, user needs can be met in one step, reducing user intervention and significantly improving the interaction efficiency between drivers and passengers and in-vehicle applications. It is compatible with various in-vehicle applications, making in-vehicle interaction more convenient and intelligent.

[0006] In some optional implementations, the step of executing the operation task through the action execution module includes: calling the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen display; executing the operation action through the action execution module, and determining whether the current operation step was successfully executed based on the changed vehicle screen display through the large model; if the current operation step was successfully executed, then calling the large model to predict the operation action for the next operation step; if the current operation step failed, then returning to the step of calling the large model to predict the operation action based on the current operation step and the current vehicle screen display, and re-predicting.

[0007] Based on the aforementioned technical methods, a closed-loop process of "predicting operation actions, executing, judging success or failure, retrying, and continuing" solves the problem of task interruption due to operational errors in existing technologies. The large-scale model judges the success or failure of an operation in real time based on the vehicle's screen, and re-predicts actions when it fails, forming a dynamic adjustment mechanism. This avoids task failure caused by single operational errors, ensuring operational continuity. Simultaneously, by using the large-scale model's autonomous judgment to replace manual intervention, user involvement is further reduced, improving the reliability and smoothness of in-vehicle application control, and ensuring the stable completion of complex tasks.

[0008] In some optional implementations, the step of calling the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen image includes: defining the role of the large model; acquiring historical images and the thinking information of the previous operation step, wherein the thinking information of the previous operation step is the analysis process information of the large model when analyzing the operation action corresponding to the previous operation step, and the historical images are screen images obtained from the previous few operation steps; inputting the current operation step, the current vehicle screen image, the thinking information of the previous operation step, and the historical images into the large model so that the large model can perform analysis and output the thinking information of the current operation step; extracting action functions from a preset action function library through the thinking information of the current operation step, and obtaining the operation action by combining the action functions.

[0009] Based on the aforementioned technical methods, by defining large model roles and inputting historical images and previous thought information, the problems of insufficient contextual understanding and large deviations in action prediction in existing technologies are solved. The multi-role setting clarifies the model's positioning, and historical images and thought information provide the model with a complete operational context, enabling it to combine the current interface state and historical logic to deduce actions. This significantly improves the accuracy of action prediction, making human-like operations (such as clicking and input) more in line with user habits, reducing misoperations caused by contextual fragmentation, and enhancing the naturalness and accuracy of vehicle-machine interaction.

[0010] In some optional implementations, if the current operation step is successfully executed, the large model is invoked to predict the operation action for the next operation step, including: updating the historical image based on the changed vehicle screen image; inputting the historical image, the thinking information of the current operation step, the next operation step, and the changed vehicle screen image into the large model so that the large model can analyze and output the thinking information of the next operation step; extracting action functions from a preset action function library through the thinking information of the next operation step, and obtaining the operation action of the next operation step by combining the action functions.

[0011] Based on the aforementioned technical methods, by updating historical images and inputting them into a large model to predict the next action, the problems of disjointed operation steps and easy deviation from the current interface state in existing technologies are solved. It updates historical information based on the real-time screen image after the operation, ensuring that the large model always infers the next action based on the latest interface state, avoiding invalid predictions based on outdated information. These technical methods make operation steps more seamlessly connected, especially suitable for dynamic interface changes such as page transitions in shopping applications, improving the smoothness and logic of task execution and further reducing operation interruptions. Inputting the current thought information into the large model solves the problems of insufficient model understanding of context and large deviations in action prediction in existing technologies. The multi-role setting clarifies the model's positioning, and historical images and thought information provide the model with a complete operational context, enabling it to infer actions by combining the current interface state and historical logic. This significantly improves the accuracy of action prediction, making human-like operations (such as clicking and input) more in line with user habits, reducing misoperations caused by context fragmentation, and enhancing the naturalness and accuracy of vehicle-machine interaction.

[0012] In some optional implementations, the thinking information includes action thinking information and planning thinking information. The action thinking information is used to represent the large-scale model thinking process when predicting operational actions, and the planning thinking information is used to represent the large-scale model thinking process when determining the position of operational steps in the operational task.

[0013] Based on the aforementioned technical means, the thinking information is clearly defined to include action thinking and planning thinking, thus solving the problem of the disconnect between model action prediction and overall task planning in existing technologies. Action thinking ensures that the reasoning behind a single-step operation, such as why a certain location is clicked, is reasonable, while planning thinking ensures that the operation does not deviate from the overall task. The combination of the two allows the large model to accurately predict single-step actions while anchoring to the overall goal, reducing the error of "focusing only on single steps and deviating from the task," improving the global consistency of vehicle application control, and making the execution of complex tasks more efficient.

[0014] In some optional implementations, the step of returning to the step of calling the large model to predict the operation action based on the current operation step and the current vehicle screen image if the current operation step fails, and then re-predicting, includes: if the current operation step fails, obtaining the changed vehicle screen image; controlling the vehicle screen to jump to the current vehicle screen image before the current operation step was executed; and re-inputting the changed vehicle screen image, the current operation step, and the current vehicle screen image into the large model so that the large model re-predicts the operation action.

[0015] Based on the aforementioned technical methods, by redirecting back to the original interface after a failure and re-predicting the action, the problems of difficulty in recovering from operation failures and easy task termination in existing technologies are solved. When an operation fails, the system not only obtains the failure screen information but also reverts to the interface state before execution, providing a basis for "error comparison" for the large model, enabling it to specifically correct the prediction logic. These technical methods enhance the system's fault tolerance, making it particularly suitable for scenarios such as vehicle network fluctuations and interface loading delays, avoiding task failures due to temporary errors and ensuring the final fulfillment of user commands.

[0016] In some optional implementations, the training steps of the large model include: obtaining a first large model and a second large model, wherein the second large model has more parameters than the first large model; obtaining a training dataset; inputting the current training data in the training dataset into the first large model and outputting multiple prediction results through the first large model; scoring the multiple prediction results through the second large model to obtain a target prediction result with a score ranking in the top preset position; combining the target prediction result and the current training data to form new training data, and using the new training data to fine-tune the first large model; replacing the first large model before fine-tuning the first large model with the fine-tuned first large model, and returning to the step of inputting the current training data in the training dataset into the first large model and outputting multiple prediction results through the first large model for iteration; and ending the training when the number of iterations exceeds a preset threshold.

[0017] Based on the aforementioned technical methods, this reinforcement learning process—"generating results with a small-scale model, scoring with a large-scale model, and iterative fine-tuning"—solves the problems of large-scale models having large parameter counts and insufficient in-vehicle deployment resources, or small-scale models having low accuracy, common in existing technologies. The small-scale model, by fitting its own high-scoring results, improves inference accuracy while reducing hardware resource consumption, and each inference step takes less time, adapting to the real-time requirements of in-vehicle scenarios. Compared to directly fitting a fixed standard answer, this method requires less training data, can quickly adapt to changing in-vehicle applications, and balances model performance with deployment costs.

[0018] In some optional implementations, obtaining the training dataset includes: generating simulated user demand information; executing the simulated demand information through the second large model, and recording the in-vehicle screen screenshots and model thinking information output by the second large model at each step during the execution; and generating the training dataset based on the simulated demand information, the recorded in-vehicle screen screenshots, and the model thinking information.

[0019] Based on the aforementioned technical methods, by generating simulated data using a large-scale model and recording screenshots and thought processes, the problems of difficult data collection and high manual costs in in-vehicle scenarios are solved in existing technologies. Utilizing a large-scale model to simulate the user's command execution process can automatically generate a large amount of realistic operational data. Combined with manual filtering to remove invalid information, the workload of manual annotation is significantly reduced. This type of data contains a complete "command-interface-action-thought" chain, providing high-quality samples for fine-tuning small models, ensuring that model training is more closely aligned with real-world interaction scenarios in in-vehicle applications, and improving generalization capabilities.

[0020] In some optional implementations, scoring the multiple prediction results using the second major model includes: scoring each prediction result for accuracy and format using multiple second major models.

[0021] Based on the aforementioned technical methods, multiple large models are used to score the prediction results for accuracy and format, solving the problems of subjective single scoring criteria and low-quality data selection in existing technologies. Multi-model scoring, similar to a "multi-judge mechanism," reduces the scoring bias of a single model and ensures the accuracy of the results. Format scoring uses regular expression matching to check the completeness of thought information and the standardization of action format, ensuring that the data meets training requirements. The high-value data selected through dual scoring effectively improves the fine-tuning effect of small models, making the model output more accurate and standardized, and further enhancing the reliability of vehicle application control.

[0022] Secondly, the present invention provides an in-vehicle application control device, the device comprising: a demand acquisition module for acquiring user demand information; a task planning module for calling a large model to plan operation tasks based on the demand information and the installed in-vehicle applications, the operation tasks including several operation steps, each operation step describing a human-like operation on the in-vehicle application; and an execution module for executing the operation tasks through the action execution module.

[0023] Thirdly, the present invention provides a vehicle comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0024] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0025] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0026] The technical solution provided by this invention has the following advantages:

[0027] (1) Based on the above technical means, after obtaining user needs, the large model is called to automatically plan and execute step-by-step tasks that include human-like operations, which solves the problem of frequent user inquiries and low intelligence in the control of in-vehicle applications in the prior art. It does not require manual operation by the user, nor does it require calling the application API interface. Through the human-like operation planning of the in-vehicle application by the large model, it can meet the user's needs in one step, reduce user intervention, significantly improve the interaction efficiency between drivers and passengers and in-vehicle applications, adapt to a variety of in-vehicle applications, and make in-vehicle interaction more convenient and intelligent.

[0028] (2) Based on the above technical means, the problem of task interruption after operation error in the prior art is solved by the closed-loop process of "predicting operation action, execution, judging success or failure, retrying, and continuing". The large model judges whether the operation is successful in real time based on the screen of the vehicle. If it fails, it re-predicts the action, forming a dynamic adjustment mechanism. This avoids task failure caused by a single operation error and ensures the continuity of operation. At the same time, the large model's autonomous judgment replaces manual intervention, further reducing user participation, improving the reliability and smoothness of vehicle application control, and ensuring that complex tasks can be completed stably.

[0029] (3) Based on the above technical means, by defining large model roles, inputting historical images and previous thinking information, the problems of insufficient understanding of context and large deviation in action prediction in existing technologies are solved. The multi-role setting clarifies the model's positioning, and historical images and thinking information provide the model with a complete operational context, enabling it to combine the current interface state and historical logic to deduce actions. This significantly improves the accuracy of action prediction, makes human-like operations (such as clicking and input) more in line with user habits, reduces misoperations caused by context fragmentation, and enhances the naturalness and accuracy of vehicle-machine interaction.

[0030] (4) Based on the above technical means, by updating historical images and inputting them into the large model to predict the next action, the problems of poor connection between operation steps and easy departure from the current interface state in the prior art are solved. It updates historical information based on the real-time screen image after the operation, ensuring that the large model always infers the next action based on the latest interface state, avoiding invalid predictions based on outdated information. It makes the operation steps more closely connected, especially suitable for dynamic interface change scenarios such as page jumps in shopping applications, improving the smoothness and logic of task execution, and further reducing operation interruptions. Inputting the current thinking information into the large model solves the problems of insufficient understanding of context and large deviation in action prediction in the prior art. The multi-role setting clarifies the model's positioning, and historical images and thinking information provide the model with a complete operation context, enabling it to combine the current interface state and historical logic to infer actions. This significantly improves the accuracy of action prediction, makes human-like operations (such as clicking and input) more in line with user habits, reduces misoperations caused by context fragmentation, and enhances the naturalness and accuracy of vehicle-machine interaction.

[0031] (5) Based on the above technical means, it is clear that the thinking information includes action thinking and planning thinking, which solves the problem of the disconnect between model action prediction and overall task planning in the existing technology. Action thinking ensures that the reasoning of a single-step operation, such as why a certain position is clicked, is reasonable, while planning thinking ensures that the operation does not deviate from the overall task. The combination of the two allows the large model to accurately predict single-step actions and anchor the overall goal, reducing the error of "only focusing on single steps and deviating from the task", improving the global consistency of vehicle application control, and making the execution of complex tasks more efficient.

[0032] (6) Based on the above technical means, by jumping back to the original interface after failure and re-predicting the action, the problems of difficulty in recovery and easy termination of tasks after operation failure in the prior art are solved. When the operation fails, the system not only obtains the failure screen information, but also traces back to the interface state before execution, providing a basis for "error comparison" for the large model, so that it can make targeted corrections to the prediction logic. This enhances the fault tolerance of the system, and is especially suitable for scenarios such as vehicle network fluctuations and interface loading delays, avoiding task failure due to temporary errors and ensuring the final achievement of user instructions.

[0033] (7) Based on the above technical means, the reinforcement learning process of "generating results with a small-scale model, scoring with a large-scale model, and iterative fine-tuning" solves the problems of large-scale models having a large number of parameters and insufficient vehicle deployment resources, or small-scale models having low accuracy in existing technologies. The small-scale model improves inference accuracy while reducing hardware resource consumption by fitting its own high-scoring results, and the inference time for each step is shorter, which is suitable for the real-time requirements of vehicle scenarios. Compared with directly fitting a fixed standard answer, this method requires less training data, can quickly adapt to the changing vehicle applications, and balances model performance and deployment cost.

[0034] (8) Based on the above technical means, simulation data is generated by using a large-scale model and screenshots and thought information are recorded, which solves the problems of difficult data collection and high manual costs in the existing technology for the thought chain in vehicle scenarios. By using a large-scale model to simulate the user's command execution process, a large amount of operation data that fits reality can be automatically generated. Combined with manual screening to remove invalid information, the workload of manual annotation is greatly reduced. This type of data contains a complete "command-interface-action-thought" chain, providing high-quality samples for fine-tuning of small models, ensuring that model training is more in line with the real interaction scenarios of vehicle applications, and improving generalization ability.

[0035] (9) Based on the above technical means, the prediction results are scored for accuracy and format by multiple large models, which solves the problems of subjective single scoring criteria and low quality of selected data in the existing technology. Multi-model scoring is similar to a "multi-judge mechanism", which reduces the scoring bias of a single model and ensures the accuracy of the results; format scoring checks the completeness of thinking information and the standardization of action format through regular matching, ensuring that the data meets the training requirements. The high-value data selected by dual scoring can effectively improve the fine-tuning effect of small models, making the model output more accurate and standardized, and further enhancing the reliability of vehicle application control. Attached Figure Description

[0036] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating an in-vehicle application control method according to an embodiment of the present invention;

[0038] Figure 2 This is another schematic flowchart of an in-vehicle application control method according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the large model training process according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of an in-vehicle application control device according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of the hardware structure of a vehicle according to an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] According to an embodiment of the present invention, an embodiment of an in-vehicle application control method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] This embodiment provides a vehicle application control method. Figure 1 This is a flowchart of an in-vehicle application control method according to an embodiment of the present invention, the process including the following steps:

[0045] Step S101: Obtain user's needs information;

[0046] Step S102: Call the large model to plan operation tasks based on the demand information and the installed vehicle applications. The operation tasks include several operation steps, and each operation step is used to describe a human-like operation on the vehicle application.

[0047] Step S103: The operation task is executed through the action execution module.

[0048] Specifically, as people's perception of vehicles has shifted from mere transportation tools to a secondary space for users, more and more in-vehicle applications such as audio-visual entertainment, shopping, and food delivery have been integrated into vehicle infotainment systems. Cars are no longer limited to their function as transportation tools, and the vehicle space is transforming from a traditional driving space into a more intelligent and diverse space. Therefore, improving the convenient, fast, and safe interactive experience between drivers and passengers and in-vehicle applications is increasingly worthy of in-depth consideration. In recent years, the increasing maturity of multimodal large-scale model technology has led to its application in more and more fields, providing a good solution for enhancing the safe and convenient experience of human-computer interaction. This invention applies multimodal large-scale model technology to the interaction between drivers and passengers and in-vehicle infotainment systems.

[0049] Specifically, the vehicle-mounted infotainment system in this embodiment of the invention collects "user demand information" in real time. "User demand information" refers to the requests issued by drivers and passengers to the system via voice, text, etc., such as "Buy me a KFC family bucket" or "Play a trailer for a recently released movie," covering various scenarios including shopping, audio-visual entertainment, and service booking. The acquisition method primarily relies on the vehicle-mounted voice system (i.e., the voice recognition module integrated in the vehicle-mounted system, which can convert user voice commands into text information). If the user inputs text commands via touch, these can also be directly received by the system as demand information. The core of this step is to accurately capture user intent, providing a foundation for subsequent task planning.

[0050] Subsequently, the vehicle's voice system converts the driver's and passengers' voice commands into text. This embodiment of the invention also deploys a task distribution system, which determines whether the driver's or passengers' commands are distributed to the in-vehicle application intelligent control system. For example, the command "turn on the air conditioning" is used to control vehicle hardware, so the command directly interfaces with the vehicle system's air conditioning control software interface, thus not requiring access to the in-vehicle application intelligent control system. However, "order a beef burger" requires calling a third-party in-vehicle application installed on the vehicle. These in-vehicle applications often do not provide relevant software interfaces and cannot complete the user's command through interface calls, thus requiring access to the in-vehicle application intelligent control system.

[0051] The in-vehicle intelligent control system primarily handles user requests for shopping, food delivery, and audio-visual content. Its main function is to call upon a large model to predict human-like actions based on images on the vehicle's infotainment screen, and then execute the predicted actions until the user's command is fulfilled.

[0052] Therefore, after obtaining the user's demand information, the vehicle application intelligent control system can call up a large model to plan operation tasks based on the demand information and the installed vehicle applications. The operation tasks include several operation steps, each of which describes a human-like operation on the vehicle application.

[0053] The "large model" specifically refers to a multimodal large model, which has the ability to process multimodal data such as text and images, understand user needs, and reason about operational logic based on visual information. "Installed in-vehicle applications" includes various applications and mini-programs pre-installed on the in-vehicle system or installed by the user (such as food delivery apps, shopping apps, video platforms, etc.). The large model first analyzes the demand information (e.g., "buying KFC" requires associating with a food delivery app), and then, combined with the list of applications already installed on the in-vehicle system, plans a series of coherent operational tasks. For example, for the demand of "buying a KFC family bucket," the operational tasks can be broken down as follows: 1. Open the food delivery app; 2. Enter "KFC family bucket" in the search box; 3. Select the target store; 4. Click "add to cart"; 5. Submit the order and pay. "Human-like operations" in each step refer to actions that mimic human behavior (such as clicking screen coordinates, swiping pages, and typing text). These actions are consistent with the logic of a human manually operating the in-vehicle system, do not require calling internal application interfaces, and are non-intrusive interactions.

[0054] The action execution module is a functional module in the vehicle infotainment system (such as an Android system) responsible for translating the operation instructions output by the large model into actual screen operations. After the large model plans the operation steps, the action execution module will receive the anthropomorphic operation instructions corresponding to each step in sequence (such as "Tap(379,750)" means clicking the food delivery app icon at screen coordinates (379,750), and trigger the corresponding operation (such as simulating finger clicks, text input, etc.) through the underlying interface of the vehicle infotainment system until the entire operation task is completed.

[0055] During execution, the module interacts with the vehicle's infotainment system in real time to ensure precise action responses and avoid operational failures caused by coordinate deviations or system delays. By autonomously planning operational tasks through a large model, it replaces the traditional step-by-step user confirmation interaction mode, reducing user intervention during driving and improving driving safety. The non-intrusive interaction is compatible with all in-vehicle applications, eliminating the need for application developers to adapt interfaces, lowering the technical implementation threshold, and enabling niche in-vehicle applications or mini-programs to enjoy intelligent control. The human-like operation steps conform to user habits, avoiding the stiff experience of mechanical commands and making in-vehicle interaction more natural. The technical solution provided by this invention covers multiple scenario needs, upgrading the in-vehicle infotainment system from a simple control terminal to an "intelligent service center," strengthening the vehicle's attribute as a "second space," and comprehensively improving the user experience for drivers and passengers.

[0056] In some optional implementations, step S103 above includes:

[0057] Step a1: Call the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen display;

[0058] Step a2: The operation action is executed through the action execution module, and the large model determines whether the current operation step has been successfully executed based on the changed vehicle screen image.

[0059] Step a3: If the current operation step is executed successfully, the large model is invoked to predict the operation action for the next operation step.

[0060] Step a4: If the current operation step fails, return to the step of the large model to predict the operation action based on the current operation step and the current vehicle screen image, and re-predict.

[0061] After planning the operation task, the action execution module needs to execute the operation steps in the operation task one by one. In this embodiment of the invention, the "current operation step" refers to the single step being executed in the operation task (such as "opening the food delivery app" or "entering the product name"). The "current vehicle screen image" is the real-time interface image before the execution of this step (such as the main screen of the vehicle system or the search page of the in-vehicle application). It is obtained through the vehicle system's screenshot function and input into the large model in the form of image data. Based on the visual understanding of the current interface (such as recognizing the icon position and input box area) and the task objective of the current step, the large model infers the specific operation action. For example, if the current step is "opening the food delivery app" and the screen displays the main screen icon, the large model will predict the "Tap(x,y)" action, where x and y are the coordinates of the food delivery app icon on the screen. The action parameters are accurate to the pixel level to ensure consistency with human clicking habits.

[0062] After predicting the operation action of the current operation step, the operation action is executed through the action execution module, and the large model reflects on whether the current operation step was successfully executed based on the changed vehicle screen image.

[0063] After receiving the operation action instructions output by the large model, the action execution module executes the actual operation through the input simulation interface of the vehicle system. For example, a click action triggers the corresponding screen coordinates, a swipe action generates continuous touch events along a specified path, and an input action calls the text injection interface to complete text input. After the operation is completed, the vehicle screen will change (e.g., jumping from the main screen to the food delivery app homepage). The system automatically captures the changed screen and inputs it into the large model. The large model compares the screen before and after the operation (e.g., determining whether the target app was successfully opened and whether the input content is displayed correctly) to determine whether the jump from the previous page to the current page meets expectations, and outputs a judgment result of "execution successful" or "execution failed". For example, if the screen remains on the main screen after the operation, it is determined that "the step of opening the food delivery app failed". At the same time, the large model will output a reflective thought process. For example, the multimodal large model outputs: "According to the two screenshots provided, the first one shows the phone's main screen, and the second one shows the homepage of the 'food delivery' app. This indicates that the latest operation (opening the 'food delivery' app) has been successfully executed, and the user has entered the 'food delivery' app's homepage." The corresponding thought information is Reflect Thought: Based on the execution process of the input, reflect on whether the previous step was executed successfully as expected. If an error occurs, the error reason will be output. This thought information can ensure that each operation is in line with expectations.

[0064] Once the large model determines that the current step is successful, the system automatically moves on to the next stage of the task, repeating the logic of step a1. The large model combines the new screen display (such as the homepage of a food delivery app) and the next operation step (such as "searching for KFC") to predict the new action (such as clicking the coordinates "Tap(a,b)" of the search box). This process forms a coherent link of "completing one step → moving to the next step," requiring no user intervention and ensuring that the task progresses in an orderly manner according to plan.

[0065] If an operation fails (e.g., a misalignment of the click coordinates prevents the food delivery app from opening), the system does not interrupt the task. Instead, it feeds back the failure information (e.g., the changed screen remains the main screen) to the larger model, triggering a re-prediction process. The larger model analyzes the cause of the failure (e.g., coordinate recognition error), adjusts the action parameters (e.g., corrects the click coordinates), or changes the operation method (e.g., if the first click does not respond, predicts the "Wait()" action to wait for the page to load before clicking again), until the current step succeeds.

[0066] like Figure 2As shown, this embodiment of the invention, through a closed-loop mechanism of "prediction-execution-reflection-retry," fully reflects on the operation of each step predicted by the large model, solving the problem of task interruption upon operation failure in traditional in-vehicle interaction. Even in the event of occasional errors (such as screen loading delays or coordinate recognition deviations), the system can autonomously correct itself, ensuring the robustness of the task. For example, if there is no response when clicking "submit order" for the first time during the food delivery order process, the system will automatically retry, avoiding manual intervention by the user. Secondly, the large model dynamically judges the operation result based on the real-time screen image, replacing the traditional fixed logic verification method, adapting to the interface differences of different applications (such as the different button positions in different food delivery applications), and enhancing the universality of the solution. Thirdly, no user intervention is required to judge the operation, and there is no need to re-interact with the user via voice to verify the correctness of the command, reducing distraction during driving and improving driving safety. Fourthly, the coherent step-by-step progression mechanism reduces the average time of complex tasks (such as from opening the food delivery application to completing payment) by more than 30%, significantly improving the efficiency and smoothness of in-vehicle interaction, allowing drivers and passengers to focus more on driving while easily fulfilling diverse needs.

[0067] In some alternative implementations, step a1 above includes:

[0068] Step b1, define the roles of the large model;

[0069] Step b2: Obtain historical images and the thinking information of the previous operation step. The thinking information of the previous operation step is the analysis process information when the large model analyzes the corresponding operation action of the previous operation step. The historical images are the screen images obtained from the previous few operation steps.

[0070] Step b3: Input the current operation step, the current vehicle screen image, the thinking information of the previous operation step, and the historical image into the large model so that the large model can analyze and output the thinking information of the current operation step.

[0071] Step b4: Extract action functions from the preset action function library based on the thinking information of the current operation step, and obtain the operation action by combining the action functions.

[0072] Specifically, in the actual use of vehicle application control methods, in addition to the problem of the system forgetting the operation steps, there is often the problem of operation error. Therefore, the technical means to improve the accuracy of vehicle application control methods based on large models still have room for further optimization.

[0073] In the embodiments of this invention, multiple modules use multimodal large models. The output accuracy of the large model is crucial. In order to stimulate the capabilities of multimodal large models, the most common method in traditional technology is to adjust the prompt words of the model input and add more useful information to the prompt words, which can make the model have better output results.

[0074] The embodiments of the present invention employ an improved technical approach, which increases the input of each step in the execution of the large model through multi-role dialogue and adds the thinking process of the model output, thereby achieving the goal of improving the accuracy of the model.

[0075] Because this system employs a multi-loop, chain-of-thought reasoning logic, adding historical user requests and model feedback actions to each step of the large model's input results in a qualitative leap in the quality of the model's output. This embodiment of the invention first defines three roles: System, User, and Assistant. Then, based on these three roles, information is input into the large model through dialogue.

[0076] The term "System" refers to the pre-assigned role to a large model, allowing it to understand its function. In this embodiment of the invention, "System" is used to assign an assistant-like role to the large model. For example, a prompt might tell the large model, "You are an assistant operating the vehicle's infotainment screen." This role definition provides the large model with clear task boundaries, avoiding reasoning biases caused by role ambiguity (such as confusing the processing logic of navigation and shopping instructions), and ensuring that the model's output always revolves around the core objective of "in-vehicle application operation."

[0077] The "User" refers to the user role that inputs commands. During the initial call to the large model in a dialogue interaction, the User is used to input the desired information, such as, "I want to order a KFC burger." In subsequent calls to the large model, the User is used to input the image that caused the screen to change due to the previous action.

[0078] An assistant is a thinking role that outputs the thought process of the large model during the execution of user instructions, and then feeds it back to the large model as input, so as to facilitate the accurate execution of subsequent steps by the large model.

[0079] After defining the roles, each step of the large model's output includes thought information and the action performed in the current step, while each step of the large model's input includes historical images and thought information from the previous step. For example, the message format used to input messages into the large model at each step is as follows:

[0080] [{"role": "system",

[0081] "content": "You are a helpful assistant.",},

[0082] { "role": "user",

[0083] "content": "You are a GUI agent. You are given a task and your actionhistory, with screenshots.

[0084] ### User Instruction ###

[0085] Buy me a KFC beef burger

[0086] ### Atomic Actions ###

[0087] The atomic action functions are listed in the format of `name(arguments): description` as follows:

[0088] - Tap(x, y): Tap the position (x, y) in current screen.

[0089] - Swipe(x1, y1, x2, y2): Swipe from position (x1, y1) to position(x2, y2).

[0090] - Type(text): Type the "text" in an input box.

[0091] - Enter(): Press the Enter key after typing (useful for searching).

[0092] - finished(): Completed user requirements and exited GUI agentservice.",},

[0093] {"role": "assistant",

[0094] "content": "{"Thought": Multiple stores were found. Select the store that is conveniently located. Action:Tap(66,86)}",},

[0095] {"role": "user",

[0096] "content": "<product list page image(history_image_base64)>",},

[0097] {"role": "assistant",

[0098] "content": "{"Thought": I saw several beef burgers, and based on the positive reviews, I chose one and added it to my cart. Action: Tap(66,5)}",},

[0099] {"role": "user",

[0100] "content": "<current_image_base64> ",}]

[0101] In the above example of the message format for inputting the large model, first, the System content is added, defining the large model as an assistant; second, the User text content is added, mainly describing the user's needs and the atomic actions to be output; then, when the model executes each subsequent step, the history of the overall process is added, and the addition method is to alternate between the User and the Assistant in the order of the operation process, where the information input by the User into the large model is the historical image, and the information input by the Assistant into the large model is the historical action prediction results and thought process; finally, the screenshot information of the current vehicle screen is added.

[0102] The output of the large model is shown in the following example:

[0103] {"reflect_thought": "The first image shows the home screen with multiple app icons, the second image shows the Meituan app open, and the third image shows the results of searching for 'roses' within the Meituan app. This indicates that the user has successfully entered Meituan's flower ordering section."}

[0104] "plan_thought": "The current task is to order a bouquet of roses through the Meituan App. The next step is to select a specific bouquet to continue the order process."

[0105] "action_thought": "To continue the task, I need to choose one from the listed rose bouquets. Choosing a popular or highlighted option is a reasonable next step. The top-ranked bouquet, '

Pence Love Letter

[0106] "action_object_str": {

[0107] "name": "Tap",

[0108] "arguments": {

[0109] "x": 143,

[0110] "y": 738}}}

[0111] Based on the example of the output of the large model above, in order to reduce the number of times the large model is called and the execution time of each step, the thinking content and the prediction action are output together, so as to reduce the time consumption of each loop while ensuring the model effect.

[0112] The "historical images" refer to screenshots of the vehicle's infotainment screen corresponding to the previous few steps (to reduce the amount of input information, this embodiment only includes the three most recent historical images). These images are extracted in real-time from the vehicle's cache, stored in base64 encoded format, and input into the large model to reconstruct the context of the operation (such as the jump from the main screen to the in-vehicle application homepage and then to the search page). The "thinking information of the previous operation step" is a record of the reasoning process generated by the large model when processing the previous step (such as "because the search box is located at the top center of the screen, the Tap(360,200) action is selected"), which includes an analysis of the operation intent, interface element recognition, and the rationality of the action. These two types of information together constitute the "temporal logic chain" of the operation, solving the problem of isolated judgment caused by traditional models relying only on a single frame.

[0113] For predicting the action in the current step, the current operation step, the current in-vehicle screen display, the thought process information from the previous operation step, and historical images are input into the large model for analysis. Upon receiving the input, the large model first analyzes the current in-vehicle screen display using a vision module (identifying the position and function of elements such as icons, buttons, and input boxes). Then, it reconstructs the interface transition logic by combining historical images, while also considering the thought process information from the previous step to understand the continuity of the operation (e.g., "The previous step opened the food delivery app; now a search needs to be performed, so the search box should be located first"). Based on this, the model generates new thought process information for the current operation step (e.g., "Search for KFC"), mainly including three parts: judgment of the current interface state (e.g., "The search box is active; text can be entered directly"), reasoning about the operation logic (e.g., "Entering 'KFC' will trigger the display of a list of related stores"), and verification of the action's rationality (e.g., "Choose the Type action instead of the Tap action because text needs to be entered, not clicked").

[0114] Subsequently, in this embodiment of the invention, action functions are extracted from a preset action function library based on the thinking information of the current operation step, and operation actions are obtained by combining the action functions.

[0115] The preset action function library is a predefined collection of atomic functions for in-vehicle operations, covering actions such as Tap (click), Swipe (swipe), and Type (input). Each function has a clear parameter format (e.g., Tap requires x and y coordinates, and Type requires text content), as shown in the table below.

[0116] Table 1. Action Function Diagram

[0117]

[0118] The system analyzes the operational intent in the thought information (e.g., the Type function corresponding to the required input text), matches the corresponding function from the library, and fills in the parameters based on the details in the thought information (e.g., the input content is KFC), ultimately generating an executable action (e.g., Type(KFC)). If the operation involves multiple consecutive actions (e.g., swiping the page and then clicking), it is implemented through function combination to ensure the continuity of the action sequence.

[0119] The technical solution provided by the embodiments of the present invention defines the role as a clear task boundary in the large model, reduces interference from irrelevant information, and makes the reasoning more focused on the in-vehicle application operation scenario.

[0120] Secondly, the introduction of historical images and information from the previous step provides the model with a complete operational context, avoiding misjudgments caused by insufficient information in a single frame, and making action prediction more consistent with the overall logic.

[0121] Thirdly, this invention considers the mechanisms of information generation and parsing to drive interpretable reasoning in the model, rather than black-box output. Compared to traditional methods that use plain text as historical records, this embodiment employs a dialogue format, incorporating image information into the prompts during the model's reasoning process. It also constructs a thought chain process with a sequential order, allowing the multimodal large model to more fully understand the overall execution process. This improves the accuracy of action prediction and facilitates subsequent error tracing and model optimization. Furthermore, allowing the model to fully consider its actions during the output phase has a greater impact on its reasoning output.

[0122] Fourth, the pre-defined action function library ensures the standardization and executability of operation actions, avoids the generation of invalid or non-compliant action instructions by the model, and further enhances the stability and smoothness of the interaction.

[0123] In some alternative implementations, step a3 above includes:

[0124] Step c1: Update the historical image based on the changed vehicle infotainment screen image;

[0125] Step c2: Input the historical images, the thought information of the current operation step, the next operation step, and the changed vehicle screen image into the large model so that the large model can analyze and output the thought information of the next operation step.

[0126] Step c3: Extract action functions from the preset action function library based on the thinking information for the next operation step, and obtain the operation action for the next operation step by combining the action functions.

[0127] Specifically, steps c1 to c3 describe how, when the current operation step is successfully executed, the large model needs to re-input the thinking information of the current operation step and the changed vehicle screen image into the large model to prepare for the action prediction of the next operation step. The technical principle is the same as that of steps b1 to b4 mentioned above. You can refer to the explanation of the principle of steps b1 to b4 mentioned above, and it will not be repeated here.

[0128] In some optional implementations, the large model outputs not only Reflect Thought (determining whether the current operation step was successfully executed) but also Action Thought and Plan Thought. Action Thought represents the large model's thought process when predicting operation actions, outputting the rationale for the predicted action, thus increasing the accuracy of action prediction. Plan Thought represents the large model's thought process for determining the position of the operation step within the task, considering the overall plan, such as whether the current step was successfully executed and whether it's possible to jump to the next step. This thought process ensures that the execution of user instructions does not deviate from the initial plan.

[0129] By employing the aforementioned technical means, the thinking information is clearly defined to include action thinking and planning thinking, thus resolving the problem of the disconnect between model action prediction and overall task planning in existing technologies. Action thinking ensures that the reasoning behind a single-step operation, such as why a certain location is clicked, is reasonable, while planning thinking ensures that the operation does not deviate from the overall task. The combination of the two allows the large model to accurately predict single-step actions while remaining anchored to the overall goal, reducing errors of "focusing only on single steps and deviating from the task," improving the global consistency of vehicle application control, and making the execution of complex tasks more efficient.

[0130] In some alternative implementations, step a4 above includes:

[0131] Step d1: If the current operation fails, obtain the changed vehicle screen image;

[0132] Step d2: Control the vehicle's infotainment screen to jump to the current screen image before the current operation step was executed;

[0133] Step d3: Re-input the changed vehicle screen image, the current operation steps, and the current vehicle screen image into the large model so that the large model can predict the operation actions again.

[0134] When the current operation fails, this embodiment of the invention first acquires the changed vehicle infotainment screen. The changed vehicle infotainment screen refers to the actual interface displayed on the vehicle after the operation is performed (such as an error page redirected after clicking an error button, or an interface that does not respond after inputting invalid content). When the large model determines that the operation has failed (such as the target in-vehicle application not opening as expected, or the input content not being displayed), the system automatically captures the screen through the vehicle infotainment screenshot module and stores it in base64 encoded format. This screen contains key information about the operation failure (such as the error page prompt text, the state of untriggered buttons), providing visual evidence for subsequent analysis of the cause of the failure.

[0135] Then, the system controls the vehicle's infotainment screen to jump back to the screen state before the current operation was executed. That is, the interface state before the operation started. The system uses the vehicle's underlying interface to call the cached data of that screen and force the screen to revert to the state before the operation. For example, if clicking the "vehicle application icon" does not redirect to the application's home page (failure), the screen will return to the home screen, restoring the initial state where the vehicle application icon is clickable. This process is transparent to the user, requires no manual operation, and ensures that the retry environment is consistent with the initial operation.

[0136] The system then inputs the "changed in-vehicle screen image" (failure result), the "current operation step" (e.g., "opening the food delivery app"), and the "image before operation" (main screen) into the multimodal large model. The large model compares the differences between the images before and after the operation, and combines this with abnormal information in the failure image (e.g., unresponsive icon status), to analyze the cause of the failure (e.g., click coordinate deviation, icon recognition error, page loading delay, etc.), and then adjusts the inference logic. For example, if the error is determined to be a coordinate deviation, the model will re-identify the accurate coordinates of the Meituan icon on the main screen and output a corrected Tap action to ensure the validity of the operation.

[0137] Based on the aforementioned technical means, by acquiring the changed failure screen and reverting to the state before the operation, the large model is provided with failure samples and retry benchmarks, enabling it to specifically correct actions and avoid repeating the same errors. Compared with traditional blind retry mechanisms, the error correction efficiency is significantly improved. The multimodal large model, based on the joint analysis of "pre-operation screen, failure screen, and current step," achieves interpretable reasoning for the cause of failure, rather than randomly adjusting actions, significantly improving the accuracy of retry actions and ensuring that the task can continue efficiently after error correction. The technical solution provided by this invention is fully automated, requiring no manual return or re-initiation of commands by the user. Especially in driving scenarios, this reduces attention distraction and improves driving safety. Simultaneously, it is adaptable to the interface differences of various in-vehicle applications, ensuring that complex tasks can still be completed stably after occasional errors, enhancing the reliability of vehicle-machine interaction and the smoothness of the user experience.

[0138] In some optional implementations, the large model provided in this embodiment of the invention is trained in the following manner:

[0139] Step e1: Obtain the first and second largest models, with the second largest model having more parameters than the first largest model;

[0140] Step e2: Obtain the training dataset;

[0141] Step e3: Input the current training data in the training dataset into the first large model, and output multiple prediction results through the first large model;

[0142] Step e4: The second model scores the multiple prediction results separately to obtain the target prediction result with the score ranking in the top preset position;

[0143] Step e5: Combine the target prediction results with the current training data to form new training data, and use the new training data to fine-tune the first model.

[0144] Step e6: Replace the first large model before fine-tuning with the first large model after fine-tuning training, and return to the step of inputting the current training data in the training dataset into the first large model and outputting multiple prediction results through the first large model.

[0145] Step e7: When the number of iterations exceeds the preset threshold, training ends.

[0146] Specifically, currently, open-source multimodal large models are divided into several different versions based on varying numbers of parameters, such as the Qwen series of open-source multimodal models: Qwen2.5-VL-72B, Qwen2.5-VL-32B, Qwen2.5-VL-7B, and Qwen2.5-VL-3B. Generally speaking, the larger the number of parameters, the stronger the inference output capability. For example, Qwen2.5-VL-72B generates far more accurate results than the Qwen2.5-VL-7B model with fewer parameters. However, this does not mean that Qwen2.5-VL-7B cannot generate accurate results; it simply means that its probability of generating such results is lower. Increasing the number of parameters in a model firstly requires more GPU resources for deployment, and secondly, it increases the computation time for inference, thus severely impacting the user experience for drivers and passengers.

[0147] To deploy a larger, more accurate model without expanding current hardware resources, and to ensure that the inference time per step is less than the time set by the designers, this embodiment of the invention selects a first-largest model with fewer parameters (e.g., Qwen2.5-VL-7B). Furthermore, extensive testing has shown that a second-largest model with more parameters (e.g., Qwen2.5-VL-72B) among currently available open-source models can generally meet the accuracy requirements of this system. This embodiment of the invention provides a reinforcement learning mechanism of "small-scale model generation - large-scale model scoring - iterative fine-tuning" to achieve efficient adaptation of the first-largest model in automotive scenarios, such as... Figure 3 As shown, the specific process is as follows.

[0148] First, we obtain the first and second largest models, with the second model having a larger number of parameters than the first. The first largest model refers to a small-parameter multimodal model suitable for in-vehicle deployment (such as Qwen2.5-VL-7B), which has a small number of parameters, fast inference speed, and is well-suited to the limited hardware resources of in-vehicle systems. The second largest model refers to a multimodal model with a larger number of parameters and stronger capabilities (such as Qwen2.5-VL-72B), used to provide high-quality scoring and data generation capabilities. Both are based on the same technical system to ensure model structure compatibility and facilitate knowledge transfer.

[0149] Next, a training dataset is acquired, and the current training data from this dataset is input into the primary model, which then outputs multiple prediction results. For example, for a single training data point (such as the scenario of "opening a food delivery app"), the primary model generates five differentiated prediction results by setting different random seeds (e.g., N=5), covering subtle differences in action parameters and thought processes. This increases the diversity of results, making it easier to select a better solution later.

[0150] Next, the second-largest model scores each prediction result, resulting in a target prediction result ranked among the top 20%. Specifically, the second-largest model acts as a judge, scoring each prediction result according to preset scoring criteria. Accuracy scoring determines whether the action conforms to the scene logic (e.g., whether the clicked location is the Meituan icon), and format scoring verifies whether the output conforms to the JSON format and includes a complete thought process. The system then selects the highest-scoring results ranked among the top 20% (e.g., the top 20%) as the target prediction result.

[0151] Next, the target prediction results and the current training data are combined to form new training data, which is then used to fine-tune the first main model. The new training data is constructed in pairs of "current training data + target prediction results," and LoRa (Low-Rank Adaptation) technique is used to fine-tune the first main model. Only the low-rank matrix parameters in the model weight matrix are updated, rather than all parameters, reducing computational resource consumption while preserving the model's original capabilities. The goal of fine-tuning is to enable the first main model to learn the logic for generating high-scoring results.

[0152] Then, the finely tuned first-class model is used to replace the original first-class model, and the process returns to step e3 for iteration. With each iteration, the performance of the first-class model gradually improves, and the proportion of high-scoring predictions increases. Through multiple iterations, the model continuously fits the high-quality results it generates. Training ends when the number of iterations exceeds a preset threshold. The preset threshold (e.g., 10 iterations) is set based on model performance convergence. When the proportion of high-scoring results increases by less than 5% in three consecutive iterations, the model performance can be considered stable, training is terminated, and the final finely tuned first-class model is output.

[0153] Through the aforementioned technical means, this invention addresses the contradiction between insufficient performance of small models and limited deployment of large models in in-vehicle scenarios by employing a mechanism of generating small-parameter models and scoring large-parameter models. The first large model, after fine-tuning, achieves inference accuracy close to that of the second large model, but with only 1 / 10 or less of its parameters, perfectly suited to the limited GPU resources and real-time requirements of in-vehicle systems. Iterative fine-tuning allows the small model to continuously fit its own generated high-quality results, rather than a fixed standard answer, reducing reliance on massive amounts of labeled data. This saves more data compared to traditional fine-tuning and is more adaptable to the dynamic updates of in-vehicle applications, exhibiting stronger generalization capabilities. LoRa fine-tuning technology reduces training costs, enabling automakers to optimize models without significant hardware investment, facilitating technology deployment. The scoring mechanism ensures that the selected training data is both accurate and standardized, avoiding interference from invalid data, making model fine-tuning more efficient. Ultimately, this achieves the triple advantages of low resource consumption, high accuracy, and fast response in in-vehicle interaction, allowing drivers and passengers to experience smooth and intelligent in-vehicle application control services.

[0154] In some alternative implementations, step e2 includes:

[0155] Step f1: Generate simulated user requirements information;

[0156] Step f2: Simulate the demand information through the second model, and record the screen screenshots and thinking information of the vehicle system output by the second model at each step during the execution process;

[0157] Step f3 generates a training dataset based on simulated demand information, recorded in-vehicle screen screenshots, and model thinking information.

[0158] Specifically, fine-tuning the first major model requires a large amount of data. Due to the variability and flexibility of in-vehicle applications and mini-programs, the thought chain data required in this embodiment of the invention is more difficult to collect than traditional data. Therefore, this embodiment of the invention provides a method to construct a dataset by using a second major model to be responsible for the main collection and manual assistance in screening.

[0159] In this embodiment, simulated user needs are first generated using a second model. These simulated user needs refer to a set of instructions designed based on common in-vehicle scenarios, covering multiple areas such as shopping, audio-visual entertainment, and service booking, for example, "Buy me a KFC chicken burger" or "Book two movie tickets." These simulated needs must cover mainstream in-vehicle applications already installed on the vehicle's infotainment system, and the instructions must closely resemble real user habits to ensure the dataset reflects actual interaction scenarios. During the generation process, the diversity and typicality of the needs must be considered, including both simple and complex multi-step instructions to comprehensively train the model's reasoning capabilities.

[0160] Next, in this embodiment of the invention, the simulated demand information is executed through a second large model, and the screen screenshots and model thinking information output by the second large model at each step are recorded during execution. The second large model has strong graphic and textual understanding and logical reasoning capabilities, and can simulate human operation processes to execute simulated demands. During execution, the system drives the second large model to complete the entire chain task from "receiving instructions" to "generating operation actions" according to the operation process provided in the aforementioned method embodiment. For example, for the demand to buy a KFC chicken burger, the model will sequentially execute steps such as "opening the food delivery app → searching for KFC → selecting products → submitting the order". During this process, the system captures key information of each step in real time through the log recording function, including screen screenshots of the vehicle, model thinking information (including ReflectThought, PlanThought, and ActionThought), and specific operation actions output by the model. A complete "input-thinking-output" chain record is formed. The model's thinking information can be seen in the following example:

[0161] [{"step": 1, "prompt": "\nYou are a GUI agent. You are given a taskand your action history, with screenshots. "

[0162] "action_thought": "To proceed with the task, I need to locate and tapon the Meituan app icon on the home screen."

[0163] "action_object_str": {"name": "Tap", "arguments": {"x": 274, "y":1189}},

[0164] "reflect_thought": "None",

[0165] "plan_thought": "The user wants to book three movie tickets for \"水饺皇后\" using the Meituan app. ",}

[0166] {"step": 2, "prompt": "\nYou are a GUI agent. You are given a taskand your action history, with screenshots."

[0167] "action_thought": "Given the current screen, I should look for anicon or text that indicates movie tickets or entertainment. "

[0168] "action_object_str": {"name": "Tap", "arguments": {"x": 640, "y":436}},

[0169] "reflect_thought": "The first image shows the home screen with various app icons, and the Meituan app has been successfully tapped. "

[0170] "plan_thought": "To book three movie tickets for \"水饺皇后\"(Dumpling Queen), I need to navigate to the movie ticket section within the Meituan app. "},

[0171] {"step": 3, "prompt": "\nYou are a GUI agent. You are given a task and your action history, with screenshots."

[0172] "action_thought": "Currently, I am on the \"电影演出\" page with \"水饺皇后\" visible. "

[0173] "action_object_str": {"name": "Tap", "arguments": {"x": 118, "y": 1011}},

[0174] "reflect_thought": "The first image shows the home screen with various app icons, and the Meituan app has been successfully opened. "

[0175] "plan_thought": "To complete the task of booking three tickets for \"水饺皇后,\" I need to:

[0176] 1. Tap on the \"Dumpling Queen\" movie poster to enter its detail page. 2.Select the desired showtime and number of tickets (three in this case).

[0177] 3. Proceed to the payment page without adding any paymentinformation."},].

[0178] Then, a training dataset is generated using the simulated demand information collected in the aforementioned steps, along with recorded in-vehicle screen screenshots and model thinking information. For the generated training dataset, this embodiment of the invention also requires manual screening of the original data to remove segments where the second major model failed to execute (such as operation interruptions caused by interface recognition errors) and logically contradictory thinking information (such as conflicts between reflection results and actual interface states), retaining coherent and valid complete task chains. Subsequently, the screened data is assembled into a structured dialogue format, organizing information in a multi-role dialogue format to form training samples. The final generated training dataset must cover different in-vehicle applications and tasks of varying complexity, and each sample must contain a complete temporal logic chain to meet the contextual understanding requirements during the fine-tuning of the first major model.

[0179] The training data generation method provided in this invention automatically generates simulated data using a second large model, solving the problem of difficult data collection for in-vehicle scenario thought chains. The second large model can efficiently execute massive simulation requirements, generating logically consistent operation chains, significantly reducing workload compared to purely manual annotation and substantially lowering data collection costs. The simulated requirement information closely resembles real-world scenarios. Combined with screenshots of the vehicle's infotainment system and model thought information, the dataset contains rich multimodal context, providing "immersive" training samples for fine-tuning the small model and significantly improving the model's understanding of the in-vehicle interface. Manual screening removes invalid data, ensuring dataset quality and preventing erroneous samples from interfering with model learning. The conversational data structure is consistent with the model input format, reducing the complexity of data preprocessing and allowing the small model to directly learn the reasoning logic of the large model, quickly adapting to in-vehicle interaction scenarios and laying a data foundation for high accuracy in low-parameter models.

[0180] In some alternative implementations, step e4 above includes:

[0181] Step g1 involves scoring multiple prediction results using the second major model, including:

[0182] Step g2 involves using multiple second-large models to score the accuracy and format of each prediction result.

[0183] Specifically, training reinforcement learning-based models requires a reward function as guidance. The main purpose of the reward function is to score the generated results. Due to the diversity and variability of the thought chain data in this embodiment, a more powerful multimodal large model is used to replace the traditional scoring system. This embodiment employs a multi-model scoring and dual-dimensional evaluation mechanism of the reward function to ensure objective screening of the prediction results. Multiple second-large models are used to score the prediction results for both accuracy and format. The combined use of multiple models reduces the subjective bias of a single model. For each prediction result generated by the first-large model, the scoring process is divided into two parts.

[0184] The first part is the accuracy score, which is determined independently by multiple secondary models based on preset prompt words to assess the reasonableness of the results. Examples of preset prompt words are shown below:

[0185] "You will be responsible for evaluating the results generated by the model. You will determine whether the results generated by the model are reasonable given the provided image and input. The criteria for this judgment are as follows:"

[0186] <model_input>

[0187] You are a GUI agent. You need to perform the next action to complete the task.

[0188] ### User Instruction ###

[0189] Buy a braised chicken using Meituan

[0190] ### Atomic Actions ###

[0191] The atomic action functions are listed in the format of `name(arguments): description` as follows:

[0192] - Tap(x, y): Tap the position (x, y) in current screen.

[0193] - Swipe(x1, y1, x2, y2): Swipe from position (x1, y1) to position(x2, y2).

[0194] - Type(text): Type the "text" in an input box.

[0195] - Enter(): Press the Enter key after typing (useful for searching).

[0196] - finished(): Completed user requirements and exited GUI agentservice.

[0197] ### Action

[0198] Choose only one action or shortcut from the options provided.

[0199] < / model_input>

[0200] The output of the model you need to evaluate is as follows, showing only the action part:

[0201] <model_output>

[0202] {"name": "Tap", "arguments": {"x": 360, "y": 1532}}

[0203] < / model_output>

[0204] The output components are scored using the following criteria: accept: indicates an acceptable answer; reject: indicates an unreasonable answer.

[0205] The reasonable standards are as follows:

[0206] 1) Given the existence of a very clear standard answer, have you provided relatively consistent answers?

[0207] 2) In problems with multiple possibilities, the given answer conforms to common sense and is an effective decision or action.

[0208] 3) The conditions given to the model were not violated.

[0209] The score provides a rating for the action.

[0210] After scoring based on the preset prompts, the scoring criteria include whether the predicted action conforms to the logic of the current scene and whether the action parameters are accurate. Each second-largest model outputs either "accept" (1 point) or "reject" (0 points), and the final accuracy score is the result of the majority of models' judgments.

[0211] The second part is the format scoring, which verifies the format compliance of the prediction results through regular expression matching. It includes two dimensions: 1. Whether it contains the complete three thought processes (ReflectThought, PlanThought, ActionThought); 2. Whether the action description conforms to the preset JSON format (such as whether the action name and parameter key-value pairs are complete). Each dimension that meets the requirements receives 1 point, and that which does not meet the requirements receives 0 points, for a total of 2 points.

[0212] When conducting the comprehensive evaluation, the accuracy score and the format score are added together. For example, a prediction result might receive 1 point for accurate action coordinates and 2 points for including a complete thought process and conforming to the correct format, for a total of 3 points. This result is considered high-value and is awarded as data, which will be used for model fine-tuning. Data with a total score below 3 points is excluded from the list of high-quality data.

[0213] The technical means provided in this invention allow for joint scoring of multiple second-largest models, avoiding the limitations of a single model, reducing subjective errors, and making accuracy judgments more objective. The dual scoring dimensions balance the validity of results and the standardization of output; the combined 3-point data selected from both dimensions is both practical and operable. This technique provides precise reward signals for reinforcement learning, enabling the first-largest model to focus on generating correct and standardized results during fine-tuning, significantly improving the quality of model output.

[0214] This embodiment also provides an in-vehicle application control device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0215] This embodiment provides an in-vehicle application control device, such as... Figure 4 As shown, it includes:

[0216] Module 901, which acquires user demand information;

[0217] The task planning module 902 is used to call the large model to plan operation tasks based on the demand information and the installed vehicle applications. The operation tasks include several operation steps, and each operation step is used to describe a human-like operation on the vehicle application.

[0218] Execution module 903 is used to perform operation tasks through the action execution module.

[0219] In some alternative implementations, execution module 903 includes:

[0220] The action prediction unit is used to call the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen image.

[0221] The action execution unit is used to execute operation actions through the action execution module, and to determine whether the current operation step has been successfully executed based on the changed vehicle screen image through the large model.

[0222] The step flow unit is used to call the large model to predict the operation action for the next operation step if the current operation step is executed successfully.

[0223] The step repetition unit is used to return to the large model to predict the operation action based on the current operation step and the current vehicle screen image if the current operation step fails, and then re-predict the action.

[0224] In some alternative implementations, the motion prediction unit includes:

[0225] The character definition unit is used to define characters in a large model.

[0226] The thinking information acquisition unit is used to acquire historical images and thinking information from the previous operation step. The thinking information from the previous operation step is the analysis process information when the large model analyzes the corresponding operation action of the previous operation step. The historical images are the screen images obtained from the previous few operation steps.

[0227] The first analysis unit is used to input the current operation step, the current vehicle screen image, the thinking information of the previous operation step, and historical images into the large model so that the large model can analyze and output the thinking information of the current operation step.

[0228] The first action determination unit is used to extract action functions from a preset action function library based on the thinking information of the current operation step, and obtain the operation action by combining the action functions.

[0229] In some optional implementations, the step transition unit includes:

[0230] The screen update unit is used to update historical images based on the changed in-vehicle screen display.

[0231] The second analysis unit is used to input historical images, current operation step thinking information, next operation step, and changed vehicle screen image into the large model so that the large model can analyze and output the thinking information for the next operation step.

[0232] The second action determination unit is used to extract action functions from a preset action function library based on the thinking information of the next operation step, and to obtain the operation action of the next operation step by combining the action functions.

[0233] In some alternative implementations, the thinking information includes action thinking information and planning thinking information. The action thinking information is used to represent the large-scale model thinking process when predicting operational actions, and the planning thinking information is used to represent the large-scale model thinking process when determining the position of operational steps in the operational task.

[0234] In some alternative implementations, the step repeating unit includes:

[0235] The failure screen unit is used to obtain the changed vehicle screen image if the current operation step fails.

[0236] The screen restoration unit is used to control the vehicle screen to jump back to the current vehicle screen screen before the current operation step was executed;

[0237] The re-prediction unit is used to re-input the changed vehicle screen image, the current operation step, and the current vehicle screen image into the large model so that the large model can re-predict the operation action.

[0238] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0239] In this embodiment, the vehicle application control device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0240] This invention also provides a computer device having the above-described features. Figure 4 The vehicle-mounted application control device shown.

[0241] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a vehicle provided in an optional embodiment of the present invention, such as... Figure 5As shown, the vehicle includes one or more processors 10, memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the vehicle, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple vehicles can be connected, with each device providing some of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.

[0242] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0243] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0244] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on vehicle usage. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0245] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0246] The vehicle also includes a communication interface 30 for communicating with other devices or communication networks.

[0247] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0248] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0249] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A vehicle-mounted application control method, characterized in that, The method includes: Obtain user needs information; The large model is invoked to plan operation tasks based on the required information and the installed in-vehicle applications. The operation tasks include several operation steps, each of which describes a human-like operation on the in-vehicle application. The operation task is executed by the action execution module; the execution of the operation task by the action execution module includes: calling the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen image; and executing the operation action by the action execution module to execute the current operation step. The process of calling the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen image includes: defining the role of the large model; acquiring historical images and the thinking information of the previous operation step, wherein the thinking information of the previous operation step is the analysis process information of the large model when analyzing the operation action corresponding to the previous operation step, and the historical images are the screen images obtained from the previous several operation steps; inputting the current operation step, the current vehicle screen image, the thinking information of the previous operation step, and the historical images into the large model so that the large model can perform analysis; when the large model receives the input, it parses the current vehicle screen image through the vision module, then restores the interface jump logic by combining the historical images, and at the same time refers to the thinking information of the previous step to understand the continuity of the operation, and then outputs the thinking information of the current operation step; extracting action functions from a preset action function library through the thinking information of the current operation step, and obtaining the operation action by combining the action functions; The large model is the first large model, which is fine-tuned and trained using the target prediction results and the current training data. The target prediction results are obtained by the second large model scoring the multiple prediction results from the output of the current training data after inputting into the first large model, and the second large model scores the results to a pre-set ranking. The second large model has more parameters than the first large model.

2. The method according to claim 1, characterized in that, The step of executing the operation task through the action execution module further includes: The large model determines whether the current operation step was successfully executed based on the changed vehicle screen display. If the current operation step is executed successfully, the large model is invoked to predict the operation action for the next operation step; If the current operation step fails, the process returns to the step of calling the large model to predict the operation action based on the current operation step and the current vehicle screen image, and then re-predicts.

3. The method according to claim 2, characterized in that, If the current operation step is executed successfully, the large model is invoked to predict the operation action for the next operation step, including: The historical image is updated based on the changed vehicle infotainment screen display. The historical images, the thought process of the current operation step, the next operation step, and the changed vehicle screen image are input into the large model so that the large model can analyze them and output the thought process of the next operation step. The action function is extracted from the preset action function library based on the thinking information of the next operation step, and the operation action of the next operation step is obtained by combining the action function.

4. The method according to claim 2 or 3, characterized in that, The thinking information includes action thinking information and planning thinking information. The action thinking information is used to represent the large-scale model thinking process when predicting operational actions, and the planning thinking information is used to represent the large-scale model thinking process when determining the position of operational steps in the operational task.

5. The method according to claim 2, characterized in that, If the current operation step fails, the process returns to the step of calling the large model to predict the operation action based on the current operation step and the current vehicle screen display, and performs a re-prediction, including: If the current operation step fails, obtain the changed vehicle screen image; Control the vehicle's infotainment screen to jump to the current screen image before the current operation step is executed; The changed vehicle screen image, the current operation steps, and the current vehicle screen image are re-input into the large model so that the large model can re-predict the operation action.

6. The method according to claim 2, characterized in that, The training steps for the large model include: Obtain the first and second largest models, where the second largest model has more parameters than the first largest model; Obtain the training dataset; The current training data in the training dataset is input into the first large model, and multiple prediction results are output through the first large model. The second major model scores the multiple prediction results respectively, and the target prediction result with the score ranking in the top preset position is obtained. The target prediction result and the current training data are combined to form new training data, and the first large model is fine-tuned using the new training data. The process involves replacing the first large model before fine-tuning with the first large model after fine-tuning, and then iterating through the steps of inputting the current training data from the training dataset into the first large model and outputting multiple prediction results through the first large model. Training ends when the number of iterations exceeds a preset threshold.

7. The method according to claim 6, characterized in that, The acquisition of the training dataset includes: Generate simulated user demand information; The simulation requirements are executed through the second major model, and the screen screenshots and model thinking information output by the second major model at each step are recorded during the execution process. The training dataset is generated based on the simulated demand information, recorded screenshots of the vehicle infotainment system, and model thinking information.

8. The method according to claim 6, characterized in that, The scoring of the multiple prediction results using the second major model includes: The accuracy and format scores of each prediction result are evaluated using multiple second-largest models.

9. A vehicle-mounted application control device, characterized in that, The device includes: The requirement elicitation module obtains user requirement information; The task planning module is used to call the large model to plan operation tasks based on the required information and the installed in-vehicle applications. The operation tasks include several operation steps, and each operation step is used to describe a human-like operation on the in-vehicle application. An execution module is used to execute the operation task through an action execution module; the execution of the operation task through the action execution module includes: calling the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen display; and executing the operation action through the action execution module to execute the current operation step; The process of calling the large model to predict the corresponding operation action based on the current operation step and the current vehicle screen image includes: defining the role of the large model; acquiring historical images and the thinking information of the previous operation step, wherein the thinking information of the previous operation step is the analysis process information of the large model when analyzing the operation action corresponding to the previous operation step, and the historical images are the screen images obtained from the previous several operation steps; inputting the current operation step, the current vehicle screen image, the thinking information of the previous operation step, and the historical images into the large model so that the large model can perform analysis; when the large model receives the input, it parses the current vehicle screen image through the vision module, then restores the interface jump logic by combining the historical images, and at the same time refers to the thinking information of the previous step to understand the continuity of the operation, and then outputs the thinking information of the current operation step; extracting action functions from a preset action function library through the thinking information of the current operation step, and obtaining the operation action by combining the action functions; The large model is the first large model, which is fine-tuned and trained using the target prediction results and the current training data. The target prediction results are obtained by the second large model scoring the multiple prediction results from the output of the current training data after inputting into the first large model, and the second large model scores the results to a pre-set ranking. The second large model has more parameters than the first large model.

10. A vehicle, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

Images