Automated execution of task objective method and apparatus, device, medium

Abstract

The application discloses an automatic task execution method and device, equipment and a medium. The method comprises the following steps: inferring an operation event corresponding to an inference chain based on operation data according to an initial VLM model; fine-tuning the initial VLM model based on the inference chain to obtain a target VLM model; performing path exploration on an agent platform based on a first random task by the target VLM model to obtain a successful sample corresponding to the first random task; converting a task target into an execution script based on the successful sample, and executing the execution script by the target VLM model in cooperation with RPA automation to obtain an execution result. The application can improve the effect of automatic task execution to meet the requirement of automatic execution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, natural language processing and enterprise process automation, and in particular to a method and device for automatically executing a task target, equipment and a medium. BACKGROUND

[0002] With the rapid development of Internet technology, an agent platform has become a main platform for obtaining information and interacting in daily life, and the requirement for automatically executing a task is higher and higher. In the prior art, RPA (Robotic Process Automation) is used alone to replace or assist manual operation of repeatedly executing a rule-based or process-based operation (such as logging into a system, filling in a form, exporting / merging a report, comparing data across systems, etc.) on a graphical interface of an agent platform. Specifically, first, a rule or a process is converted into an execution script, then RPA identifies interface elements of the agent platform by using a selector, performs element positioning, and then when an automated process is triggered, the RPA automated task is distributed to multiple robots, which can perform in parallel and be scheduled in a queue, and finally, based on element positioning, the execution script is automatically executed on the agent platform by the robots. However, the RPA based on the selector or the image template is highly sensitive to small changes in DOM, text, style or coordinates on the graphical interface of the agent platform, and is prone to execution failure, resulting in poor automated execution effect. If an initial VLM model (Visual-Language Models) is used alone to automatically execute a task target, it can process visual (picture, screenshot, video frame) and text information at the same time, link a screen picture with context semantics, output specific operation instructions (such as clicking, inputting, selecting), simulate or drive human interaction on a computer, so as to recognize text, icons and layout semantics, and achieve good generalization on cross-interface or unseen visual appearance (for example, the same button can still be recognized under different themes), which means that the computer operation capability based on the VLM has good adaptability to changes in interface elements. However, the general initial VLM model usually emphasizes visual-language alignment and natural language response for generalization, and is designed for a model that is universal for multiple visual tasks, and is not designed for precise and verifiable interface operation control, so the automated execution effect of serialization, low latency and strict correctness (element precise positioning, transaction rollback, conditional judgment) is poor, and cannot meet the requirement for automated execution. SUMMARY

[0003] The embodiments of the present application provide a method and device for automatically executing a task target, equipment and a medium, which can improve the effect of automatically executing a task to meet the requirement for automated execution.

[0004] In a first aspect, embodiments of the present application provide a method for automatically performing a task target, applied to an automation system of an agent platform, and the method comprises:

[0005] collecting operation data corresponding to an operation event of the agent platform;

[0006] performing reasoning based on the operation data according to an initial VLM model, to obtain a reasoning chain corresponding to the operation event, wherein the reasoning chain indicates a model thinking process;

[0007] performing fine-tuning training on the initial VLM model according to the reasoning chain, to obtain a target VLM model;

[0008] exploring a path of the agent platform based on a first random task automation through the target VLM model, to obtain a success example corresponding to the first random task;

[0009] converting a task target into an execution script based on the success example, and performing the execution script through the target VLM model in cooperation with RPA automation, to obtain an execution result.

[0010] In a second aspect, an apparatus for automatically performing a task target is provided, applied to an automation system of an agent platform, and the apparatus comprises:

[0011] a collection module configured to collect operation data corresponding to an operation event of the agent platform;

[0012] a first calculation module configured to perform reasoning based on the operation data according to an initial VLM model, to obtain a reasoning chain corresponding to the operation event, wherein the reasoning chain indicates a model thinking process;

[0013] a second calculation module configured to perform fine-tuning training on the initial VLM model according to the reasoning chain, to obtain a target VLM model;

[0014] a third calculation module configured to explore a path of the agent platform based on a first random task automation through the target VLM model, to obtain a success example corresponding to the first random task;

[0015] an automation module configured to convert a task target into an execution script based on the success example, and perform the execution script through the target VLM model in cooperation with RPA automation, to obtain an execution result.

[0016] In a third aspect, an electronic device is provided according to embodiments of the present application, and the electronic device comprises:

[0017] at least one processor;

[0018] at least one memory for storing at least one program;

[0019] The at least one program, when executed by the at least one processor, implements the method for automatically executing a task target according to any one of the first aspect.

[0020] In a fourth aspect, a computer-readable storage medium is provided, which stores computer executable instructions for implementing the method for automatically executing a task target according to any one of the first aspect.

[0021] To sum up, the method of the above embodiments of the present application comprises: collecting operation data corresponding to an operation event of an agent platform; performing reasoning based on the operation data according to an initial VLM model to obtain a reasoning chain corresponding to the operation event; fine-tuning the initial VLM model according to the reasoning chain to obtain a target VLM model; automatically exploring a path of the agent platform based on a first random task by the target VLM model to obtain a successful sample corresponding to the first random task; converting a task target into an execution script based on the successful sample, and executing the execution script by the target VLM model in cooperation with RPA automation to obtain an execution result. The present application first performs reasoning based on the operation data according to the initial VLM model to obtain a reasoning chain corresponding to the operation event, wherein the reasoning chain indicates the thinking process of the model, and the characteristic that the reasoning chain indicates the thinking process of the model can be obtained by reverse reasoning, thereby providing a better data basis for subsequent fine-tuning training based on the reasoning chain. Then, the initial VLM model is fine-tuned according to the reasoning chain to obtain a target VLM model. On the basis of reliable reasoning chain, the accuracy of the target VLM model obtained by fine-tuning the initial VLM model according to the reasoning chain is higher, so that the reliability of the target VLM model is higher. Next, the target VLM model is used to automatically explore a path of the agent platform based on a first random task to obtain a successful sample corresponding to the first random task. The target VLM model can automatically explore diversified paths to obtain a successful sample corresponding to a global optimal solution of the first random task. Compared with manually labeling and constructing successful data, the number of successful samples can be expanded while efficiency is taken into account. Finally, the task target is converted into an execution script based on the successful sample, which can convert a task target with a very high repetition rate into a reusable execution script to effectively improve the execution efficiency and stability of the execution script. The execution script is executed by the target VLM model in cooperation with RPA automation to obtain an execution result. The advantages of RPA in execution stability, transaction management and enterprise integration can be utilized, and the advantages of the target VLM model in visual understanding, fuzzy matching and abnormal reasoning can be combined to make up for the respective shortcomings, so as to realize stable and adaptive automation, thereby improving the effect of automated execution of tasks to meet the requirements of automated execution. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the steps of a method for automating the execution of a task objective, as provided in one embodiment of this application.

[0023] Figure 2 This is a flowchart illustrating a method for automating the execution of a task objective according to an embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating the data acquisition process in one embodiment of this application;

[0025] Figure 4 This is a schematic diagram of an annotation human-computer interaction data collection and annotation tool provided in one embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating a successful example of automated acquisition of a target VLM model, provided in one embodiment of this application.

[0027] Figure 6 This is a flowchart illustrating the steps for automatically acquiring a successful example of a target VLM model, as provided in one embodiment of this application.

[0028] Figure 7 This is a hardware schematic diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] It is understandable that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0031] With the rapid development of Internet technology, an agent platform has become a main platform for obtaining information and interacting in daily life, and the requirement for automatically performing tasks is higher and higher. In the prior art, RPA (Robotic Process Automation) is used alone to replace or assist manual operation of repeatedly performing rules-based or process-based operations (such as logging into a system, filling out a form, exporting / merging a report, cross-system data comparison, etc.) on a graphical interface of an agent platform. Specifically, first, rules or processes are converted into an execution script, then RPA identifies interface elements of the agent platform through a selector, performs element positioning, and then when an automated process is triggered, the RPA automated task is assigned to multiple robots, which can perform in parallel and be queued and scheduled, and finally, based on element positioning, the execution script is automatically executed on the agent platform by the robot. However, the RPA based on the selector or the image template is highly sensitive to small changes in the DOM, text, style or coordinates on the graphical interface of the agent platform, and is prone to execution failure, resulting in poor automated execution effect. If the initial VLM model (Visual-Language Models) is used alone to automatically perform a task target, it can process visual (picture, screenshot, video frame) and text information at the same time, link the screen picture with the context semantics, output specific operation instructions (such as clicking, inputting, selecting), simulate or drive human interaction on a computer, so as to recognize text, icons, layout semantics, and achieve good generalization on cross-interface or unseen visual appearance (for example, the same button can still be recognized under different themes), which means that the computer operation capability based on the VLM has good adaptability to changes in interface elements. However, the general initial VLM model usually emphasizes the generalization of visual-language alignment and natural language response, and is designed for a model that is universal for multiple visual tasks, and is not designed for precise and verifiable interface operation control, so the automated execution effect of serialization, low latency, strict correctness (element precise positioning, transactional rollback, conditional judgment) is poor, and cannot meet the automated execution requirement.

[0032] Based on this, the application provides an automated task execution method and device, equipment and medium, which can improve the effect of automated task execution to meet the requirement of automated execution.

[0033] Exemplarily, firstly, inference is performed based on the operation data according to the initial VLM model to obtain an inference chain corresponding to the operation event, wherein the inference chain indicates the model thinking process, and characteristics of the inference chain indicating the model thinking process can be obtained by reverse inference, thereby providing a better data basis for subsequent fine-tuning training based on the inference chain; then, the initial VLM model is fine-tuned based on the inference chain to obtain a target VLM model, so that the accuracy of the target VLM model obtained by fine-tuning the initial VLM model based on the inference chain is higher based on the reliable inference chain, thereby making the reliability of the target VLM model higher; then, the target VLM model is used to automatically explore the agent platform based on the first random task to obtain a successful example corresponding to the first random task, so that the target VLM model can automatically explore diversified paths, and then a successful example of a global optimal solution corresponding to the first random task is obtained, and compared with manually annotated successful data, the number of successful examples can be expanded while efficiency is taken into account; the task target is converted into an execution script based on the successful example, the execution efficiency and stability of the execution script can be effectively improved by converting the task target with a very high repetition rate into a reusable execution script, and the execution script is executed by the target VLM model in cooperation with RPA automation to obtain an execution result, the advantages of RPA in execution stability, transaction management and enterprise integration can be utilized, and the advantages of the target VLM model in visual understanding, fuzzy matching and abnormal reasoning can be combined to make up for the respective shortcomings, thereby realizing stable and adaptive automation and improving the effect of automated task execution.

[0034] The automated task execution target method of the embodiments of the present application is applied to an automation system of an agent platform, as shown in Figure 1 and Figure 2 , the automated task execution target method includes but is not limited to the following steps:

[0035] It can be understood that the agent platform of the present application can be a webpage or an APP running on a mobile terminal or a computer terminal, and the embodiments of the present application do not limit the agent platform.

[0036] Step S100, operation data corresponding to an operation event of the agent platform is collected.

[0037] Exemplarily, the user captures the operation event in real time and synchronously stores the interface state in the execution process of real operation of the agent platform, thereby generating complete operation data. The operation data includes screen states and real actions, and the real actions and the screen states corresponding to the real actions before and after the real actions are arranged according to the action sequence of the real actions.

[0038] Specifically, referring to Figure 3 and Figure 4As shown, during the execution of the real operation of the user on the agent platform, the automatic system of the agent platform listens to the user operation in real time through the API of the underlying operating system or the browser debugging interface, captures operation events including mouse operation (click, double-click, move, scroll wheel), keyboard operation (key press / release, text input, shortcut key combination), etc. Each operation event records the target object and positioning method, screen or window coordinate position (x, y), and control identifier (DOM path, AccessibilityID, AutomationID, XPath, CSSSelector) and is attached with a timestamp (the exact time of event occurrence, used for sequence restoration), and the context information of system display parameters (screen resolution, zoom ratio) to ensure the reproducibility and sequence restoration capability of the event. Before and after each operation event, the system automatically intercepts the current screen or active window state. In this way, through the above synchronization mechanism, each operation event is strictly aligned with its front and back screen states, and the screen state corresponding to the first time before the real action starts is taken as the first state, forming a first state real action and a second state, and in this way, the causal chains corresponding to the continuous real actions are sequentially connected in the sequence order of the real actions, to obtain operation data.

[0039] In step S110, reasoning is performed based on the operation data according to the initial VLM model, to obtain a reasoning chain corresponding to the operation event, wherein the reasoning chain indicates the thinking process of the model.

[0040] It can be understood that, since the initial VLM model generates the thinking process first and then generates the specific execution action in the reasoning process, the quality of the thinking process will greatly affect the quality of the action generation. Therefore, generating good thinking process data to fine-tune the initial VLM model can improve the logical reasoning ability of "why do you operate like this" and thus improve the success rate of executing the task. In this way, the reasoning chain of the embodiments of the present application includes the screen state, the real action, and the thinking process, and the thinking process of the model is indicated by constructing the reasoning chain, to provide a reliable data basis for subsequent fine-tuning training.

[0041] For example, the reasoning process can refer to the CoT structure, and is based on the operation result of the operation event and the current screen state The initial VLM model first generates the thinking process and then generates the real action of specific execution . When the model makes a prediction, the history and are transmitted in the form of multiple rounds of dialogue, and the last N screen shots are attached to represent the screen state (N is set to 2 by default), for convenience of description, N is taken as 2 in the subsequent description, and the reasoning process is shown in the following expression:

[0042] ;

[0043] wherein, is the nth thinking process, is the nth real action, is the operation result of the operation event, is the Nth screen state, is the N screenshots.

[0044] Specifically, the real action is as shown in Table 1:

[0045] Table 1

[0046]

[0047] Step S120, fine-tuning training is performed on the initial VLM model according to the inference chain to obtain a target VLM model.

[0048] Exemplarily, the full parameters of the initial VLM model can be fine-tuned in the inference chain, which can improve the accuracy of the target VLM model; part of the parameters of the initial VLM model can also be fine-tuned in the inference chain, specifically, the part of the parameters of the initial VLM model can be fine-tuned through LoRA, AdapterTuning or PromptTuning, which can only fine-tune part of the parameters to improve the accuracy of the target VLM model and reduce the cost required for fine-tuning training; the initial VLM model can also be fine-tuned in the inference chain through multi-modal instruction data, which can fine-tune training for specific needs and improve the accuracy of the target VLM model. The specific way of fine-tuning training of the initial VLM model in the embodiments of the present application is not limited.

[0049] Exemplarily, since the initial VLM model often lacks in the accuracy of specific interface operations in order to cover multiple visual semantic tasks, the inference chain of the embodiments of the present application includes screenshots, corresponding operation sequences and natural language dialogues / annotations, so that fine-tuning is performed through the inference chain, which can significantly improve the ability of the target VLM model in interface action planning, element positioning and operation confirmation. In actual application, the target VLM model is continuously iteratively optimized, so that the accuracy of the target VLM model is continuously improved.

[0050] Exemplarily, in order to ensure the reliability of the target VLM model, the embodiments of the present application comprehensively evaluate the performance of the ComputerUse ability of the VLM in different application scenarios through a double evaluation strategy, and the evaluation strategy is as shown in Table 2:

[0051] Table 2

[0052]

[0053] Specifically, through the evaluation mode of ByTurnEvaluation, since each step completely relies on the prediction result of the target VLM model on the output of the previous round, it further examines the ability of the target VLM model to completely autonomously explore to complete the task goal. The evaluation mode of ByTaskEvaluation is to simulate the case where the correct guidance is given by artificial, and to evaluate whether the target VLM model can remain accurate, and to more consider the instruction compliance ability of the target VLM model itself. In this way, through the above two evaluation modes, the ability of the target VLM model under ideal conditions and the performance in the real environment can be well distinguished, and there is a relatively perfect evaluation system and index for the target VLM model in the use process, thereby improving the reliability of the target VLM model.

[0054] In step S130, the target VLM model automatically explores the path of the agent platform based on the first random task, and obtains the success example corresponding to the first random task.

[0055] Exemplarily, since the target VLM model has interface operation reasoning ability after fine-tuning. At this time, by randomly generating the first random task to give the target VLM model, setting the temperture (a commonly used parameter to control the randomness and creativity of large language models) of the target VLM model to a high value, the target VLM model can try low-probability but possibly more optimal solutions, thereby exploring more paths, and further obtaining the success example of the global optimal solution corresponding to the first random task. In addition, since high-quality "screenshot + operation sequence + natural language dialogue" success data is scarce, compared with artificial annotation of success data, automatic exploration of paths by the target VLM model can expand the number of success examples and improve efficiency.

[0056] Exemplarily, as shown in Figure 5 and Figure 6 , the target VLM model includes a visual encoder, a language encoder, a cross-modal alignment layer, and a language decoder. First, the visual encoder obtains the visual features corresponding to the front end of the agent platform; then the language encoder performs text analysis on the first random task to obtain text features; then the cross-modal alignment layer correlates the text features and the visual features to generate a to-be-executed command; finally, the language decoder decodes and executes the to-be-executed command, and in the case of triggering an end action within a limited number of steps, a potential success example is obtained. A batch of potential examples are formed by multiple potential success examples, and then imported into a path annotation tool for manual review or automatic review according to a preset rule to obtain the success example corresponding to the first random task, and otherwise discard the examples greater than the limited number of steps to trigger the end action.

[0057] Step S140, converting the task target into an execution script based on the success example, and then executing the execution script through the target VLM model to obtain an execution result.

[0058] For example, first, the target example with a similarity greater than a similarity threshold is screened from the success example, and then the characteristics of the target example are parsed by the rule engine of the automation system of the agent platform, and then the characteristics are matched with the predefined rules to obtain the target rule, and finally the target example is mapped to the execution script according to the mapping relationship of the target rule. The high-repetition task target can be converted into an execution script to effectively improve the execution efficiency and stability of the task target, and effectively avoid the problems of large calculation resource consumption and reduced execution efficiency of the initial VLM model in the prior art. The embodiments of the present application do not limit the way of obtaining the execution script.

[0059] For example, there is no target example with a similarity greater than a similarity threshold in the success example, and the target VLM model explores the path according to the task target and completes the task target to obtain an execution result, and the execution result is taken as a new success example.

[0060] For example, first, the RPA identifies the interface elements of the agent platform through the selector to locate the elements, and then when the automation process is triggered, the RPA automation task is distributed to multiple robots, and the multiple robots can execute in parallel and be queued. Finally, based on the element positioning, the robot executes the execution script in the agent platform. Because there are often interface element changes in real scenarios, the RPA execution script may still fail when facing dynamic interface changes. If an exception occurs during the execution of the RPA execution script, the target VLM model starts a passive takeover process to obtain an execution result; if it is predicted that the RPA cannot adapt to the execution of the task target, the target VLM model starts an active takeover process to obtain an execution result. In this way, the effect of the automation execution of the task target is improved.

[0061] Understandably, in real-world applications, it's undesirable to grant the target VLM model high degrees of generation freedom; rather, reducing them is preferable. Therefore, ensuring the target VLM model maintains correct execution logic at low temperatures requires further enhancement. This embodiment can further strengthen the determinism of the target VLM model when performing specific tasks by collecting execution results for fine-tuning training. Alternatively, reinforcement learning can be used to further enhance the determinism, enabling the finding of successful paths with fewer exploration attempts and more stable performance at low temperatures. Then, the fine-tuned target VLM model automatically explores paths on the agent platform based on a second random task, obtaining verification samples. These verified samples are used as new successful samples, adding samples for the next successful execution of the task objective, thus completing closed-loop learning.

[0062] In some embodiments, the inference chain includes a first data chain and a second data chain. The process of obtaining the inference chain corresponding to an operation event based on operational data using an initial VLM model includes: inferring the first data chain corresponding to the operation event from the initial VLM model using reverse inference based on operational data; fine-tuning and training the initial VLM model on operational data to obtain an intermediate VLM model, and using the intermediate VLM model to perform inference to obtain the inference data chain; and then generating the second data chain corresponding to the operation event by sampling the inference data chain multiple times based on the Bootstrap concept.

[0063] It is understandable that, since the initial VLM model inference process involves first generating a thought process and then generating specific execution actions, the quality of the thought process significantly affects the quality of the action generation. Therefore, generating good thought process data to fine-tune the initial VLM model can improve the ability to generate logical reasoning of "why this operation is performed," thereby increasing the success rate of task execution. Thus, this embodiment of the application constructs a first data chain to indicate the model's thought process, providing a data foundation for the reliability of obtaining the target VLM model through fine-tuning training of the initial VLM model based on the first data chain.

[0064] For example, the initial VLM model performs reverse reasoning based on the operation result corresponding to the operation event, combined with the screen state of the operation data and the actual action, until the user command is completed, and obtains the first data chain corresponding to the operation event. The first data chain indicates the thought process of completing the operation event.

[0065] Specifically, the first data chain can be represented by the following expression:

[0066] ;

[0067] in, The nth first thought process reasoned for the initial VLM model, The initial VLM model, The operation result corresponding to the operation event, The nth real action corresponding to the operation event, The nth-1 screen state corresponding to the operation event, The nth screen state corresponding to the operation event.

[0068] Exemplarily, the first data chain can be determined by the following steps:

[0069] (I) Scene construction:

[0070] Taking the operation result (i.e. user instruction) of the operation event as "opening the Chrome browser and searching for 'weather forecast'" as an example, the operation data corresponding to the operation event is shown in the following expression:

[0071] ;

[0072] Among them, The desktop interface, no browser is opened, The Chrome window has been opened but no page has been loaded, The Chrome address bar has been input with "weather forecast", Clicking the Chrome icon, Entering "weather forecast" in the address bar, Pressing the enter key.

[0073] (II) Reverse reasoning:

[0074] 1) First iteration: input query and ( , ) into the initial VLM model for reverse reasoning, and the initial VLM model outputs: = "clicking the Chrome icon to start the browser, which is a prerequisite for performing the search".

[0075] 2) Second iteration: input query, ( , ) and ( , ) into the initial VLM model for reverse reasoning, and the initial VLM model outputs: = "entering 'weather forecast' in the address bar, which is the search content explicitly required by the user".

[0076] 3) Third iteration: input query, ( ), ( , , ( , ) is input to the initial VLM model for back reasoning, and the initial VLM model outputs: = "Press Enter to submit the search because the address bar has entered the keyword". In this way, the first data chain composed of to , the screen state and the real action is obtained.

[0077] Exemplarily, in the case of scarce or high-cost labeled data, the intermediate VLM model is obtained by fine-tuning training the initial VLM model on the operation data, so that the inference data chain can be obtained according to the intermediate VLM model without completely relying on external labeled data, thereby expanding the data. In addition, the intermediate VLM model is iteratively updated in the subsequent steps, which can learn from the output of its own continuous improvement, realize iterative self-improvement, and form a virtuous cycle of "better model → higher quality data → better model".

[0078] It can be understood that, since the Bootstrap idea is multiple random sampling, the diversity of the generated second data chain can be increased, thereby expanding the coverage of the second data chain, and providing a data basis for the generalization of the target VLM model based on the second data chain for fine-tuning training.

[0079] Exemplarily, the second data chain includes a second thinking process, a screen state and a real action, and the second data chain can be determined by the following expression:

[0080] ;

[0081] Wherein, is the n-1th second thinking process obtained by the intermediate VLM model, is the intermediate VLM model, is the operation result corresponding to the operation event, is the nth real action corresponding to the operation event, is the n-1th screen state corresponding to the operation event, is the nth screen state corresponding to the operation event, is the predicted thinking in the candidate pair, is the target thinking in the target candidate pair, is the predicted action in the candidate pair, is the target action in the target candidate pair.

[0082] In some embodiments, the operation data includes a plurality of sequentially connected real actions, and generating the second data chain corresponding to the operation event based on the Bootstrap idea by sampling the inference data chain multiple times includes: randomly sampling the inference data chain to generate a candidate pair set, wherein the candidate pair set includes a plurality of candidate pairs, and each candidate pair includes a predicted action and a predicted thought; screening candidate pairs that meet a matching condition of the real action from the candidate pair set to obtain a target candidate pair corresponding to the real action, and adding a target thought of the target candidate pair to the second data chain, wherein the matching condition is that the predicted action of the candidate pair matches the real action; repeating the above steps until the target thought of the target candidate pair corresponding to each real action is added to the second data chain.

[0083] Exemplarily, the second data chain can be determined by the following steps:

[0084] (I) Scene construction:

[0085] Taking the operation result (i.e., user instruction) of the operation event as an example, the operation result is "open Chrome browser and search 'weather forecast'", and the operation data corresponding to the operation event is shown in the following expression:

[0086]

[0087] wherein, is a desktop interface without a browser being opened, is a Chrome window being opened but without a page being loaded, is a Chrome address bar having "weather forecast" being input, is clicking a Chrome icon, is inputting "weather forecast" in the address bar, is pressing the enter key.

[0088] (II) Initial sampling:

[0089] The candidate pair set is generated by randomly sampling the inference data chain, and the candidate pair set is composed of candidate pairs, and each candidate pair is composed of a predicted action and a predicted thought. Specifically, the candidate pairs are shown in Table 3:

[0090] Table 3

[0091]

[0092] (III) Screening matching actions:

[0093] Each predicted action is matched with the first real action "click the Chrome icon", and only the predicted action matching the real action is reserved, so that ​Target candidate pair .

[0094] (Fourth), iteration:

[0095] The target action in the target candidate pair is added to the second data chain. The first real action is fixed, and the second thinking process is continued except for the first real action . In this way, the initial sampling and matching action screening process are repeated for the second real action, and the complete second data chain is finally obtained.

[0096] In some embodiments, fine-tuning the initial VLM model according to the reasoning chain to obtain the target VLM model includes: freezing the initial VLM model; adding low-rank matrices to the Attention layer and the feedforward layer in the Transformer of the initial VLM model; based on the first data chain and the second data chain, fine-tuning the low-rank matrices by minimizing a preset loss function, thereby obtaining the target VLM model.

[0097] It can be understood that the first data chain represents the original ability of the initial VLM model before fine-tuning, providing a reliable basic truth reference and starting point for subsequent fine-tuning training, and ensuring that the basic cognitive ability of the model will not be lost in fine-tuning. The second data chain has improved data diversity compared to the first data chain, effectively expanding the size and diversity of the data, helping the fine-tuning training to learn more generalized patterns and avoiding overfitting to limited original data. In this way, the stability and basic ability provided by the first data chain and the diversity and generalization ability provided by the second data chain are combined to be used for fine-tuning training, making the fine-tuning training effect better.

[0098] Exemplarily, first, freeze all parameters of the initial VLM model; then select the Attention layer and the feedforward layer in the Transformer of the initial VLM model as the position to add low-rank matrices, which can further enhance the understanding of the context relationship of the task target on the basis of preserving the ability of the Attention layer of the initial VLM model, while more flexibly adjusting the processing method of the information flow. In addition, fine-tuning only through low-rank matrices can avoid the computational and storage overheads brought by full parameter update, thereby improving the efficiency of fine-tuning training; then based on the first data chain and the second data chain, fine-tune the low-rank matrices by minimizing a preset loss function, thereby obtaining the target VLM model. In this way, the extraction path of the target VLM model to the nonlinear features can be adjusted, and the representation learning ability and control effect of the task target can be enhanced.

[0099] Specifically, the Attention layer includes q_proj, k_proj, v_proj, and o_proj, and the feedforward layer includes gate_proj, up_proj, and down_proj. The Attention layer and the feedforward layer can be represented by Table 4:

[0100] Table 4

[0101]

[0102] Exemplarily, since the PaddingToken is a padding mask and the imageToken mask does not affect the training, the preset loss function of the embodiment of the present application does not include the PaddingToken mask loss and the imageToken mask loss. Since there are multiple rounds of dialogues in the training process, the preset loss function of the embodiment of the present application does not calculate the mask loss before the last <im_start> (that is, the last Assistant reply). In this way, the model only learns the part of the last round of Assistant reply in the multiple rounds of dialogues, and the content of the previous multiple rounds of dialogues does not participate in the loss calculation, so as to improve the efficiency of fine-tuning training.

[0103] In some embodiments, further executing the execution script through the target VLM model in cooperation with the RPA automation to obtain an execution result includes: automatically executing the execution script on the agent platform through the RPA, while automatically monitoring the state of the agent platform, to obtain a state result; and in a case where the state result is an abnormal state or a case where there is no change for a long time, automatically exploring a path on the agent platform according to the task target through the target VLM model to obtain the execution result.

[0104] In some embodiments, further executing the execution script through the target VLM model in cooperation with the RPA automation to obtain an execution result includes: comparing interface elements of a screenshot of the agent platform at the current time and interface elements of a screenshot of a successful example before automatically executing the execution script on the agent platform, to obtain a change value; and in a case where the change value is greater than a threshold value, automatically exploring a path on the agent platform according to the task target through the target VLM model to obtain the execution result.

[0105] Exemplarily, the execution result can be determined by the following steps:

[0106] (I) Before the execution script is automatically executed on the agent platform, the screenshots of the current time of the agent platform and the screenshots of the successful samples are executed as follows: first, the common GUI elements in the screenshots are identified by a target detection model, such as buttons, input boxes, icons, labels, and other interface elements, and the bounding boxes of the interface elements are located, and then the visual information is obtained, and then the underlying structure information corresponding to the agent platform (such as the UI hierarchy file of Android or the DOM tree of Web) is obtained, and then the platform metadata is obtained, and the platform metadata and the visual information are fused to obtain the feature value, and finally the feature values between the two are compared to obtain the change value; in the case that the change value is greater than the threshold value, the interface elements of the agent platform change greatly, and the RPA cannot normally complete the task target. At this time, the target VLM model is used to automatically explore the path of the agent platform according to the task target, and the execution result is obtained to improve the reliability of executing the task target.

[0107] (II) In the case that the change value is less than or equal to the threshold value, the interface elements of the agent platform change little, and the RPA can normally complete the task target. At this time, the RPA identifies the interface elements of the agent platform through the selector, locates the elements, then when the automation process is triggered, the RPA automation task is assigned to multiple robots, and the multiple robots can execute in parallel and be queued, and finally based on the element positioning, the execution script is automatically executed on the agent platform through the robot to obtain the execution result, thereby improving the reliability of executing the task target.

[0108] (III) In the process of RPA automatic execution of the execution script, the state of the agent platform is monitored through the interface of the agent platform, and the state result is obtained; in the case that the state result is an abnormal state or there is no change for a long time, it is indicated that the RPA has an abnormality in the process of executing the script and cannot normally complete the task target. At this time, the target VLM model is used to automatically explore the path of the agent platform according to the task target, and the execution result is obtained to improve the reliability of executing the task target.

[0109] In some embodiments, after the execution script is executed by the target VLM model in cooperation with the RPA automation, the execution result is obtained, and then the execution result is obtained as an execution trajectory sample, and the execution trajectory sample is subjected to a data cleaning operation to obtain an initial training data set; the target VLM model is fine-tuned based on the initial training data set, and the fine-tuned target VLM model is used to automatically explore the path of the agent platform based on a second random task to obtain a to-be-verified sample; the to-be-verified sample that passes the verification is used as a new successful sample.

[0110] It can be understood that, since the target VLM model does not expect to give the target VLM model a high generation freedom during the application process in the real scene, on the contrary, to reduce the temperature, whether the execution logic at low temperature can keep correct as much as possible needs to be further strengthened. Therefore, first of all, whether the execution result is successful or failed, the RPA success or failure execution track (including screen snapshot, execution step, final result) corresponding to the execution result is automatically converted into an execution track sample as a new training sample of the target VLM model. Then, the execution track sample is subjected to data cleaning operation to obtain an initial training data set, and the target VLM model is fine-tuned based on the initial training data set periodically or online. The target VLM model can be further strengthened in certainty when performing a specific task by collecting successful samples for fine-tuning training, and the successful path can be found in fewer exploration times, and the performance at low temperature is also more stable. Then, the agent platform is automatically path explored based on the second random task by the fine-tuned target VLM model, and the to-be-verified sample is obtained. The to-be-verified sample that passes the verification is used as a new successful sample for the successful sample of the next execution task target, so as to complete the closed-loop learning.

[0111] Exemplarily, first, the execution track sample is subjected to data cleaning operations such as deduplication and operation sequence alignment to obtain an initial training data set, which can improve the data quality of the initial training data set. The fine-tuning and path exploration are basically the same as in the above embodiment, and will not be repeated here. Then, the to-be-verified sample can be verified by a preset rule, or the to-be-verified sample can be verified by manual verification. The verification method is not limited in the embodiments of the present application, and the specific verification method can be selected according to actual needs.

[0112] In some embodiments, an automatic execution task target device is applied to an automatic system of an agent platform, and the automatic execution task target device comprises: a collection module, the collection module is used to collect operation data corresponding to an operation event of the agent platform; a first calculation module, the first calculation module is used to infer based on the operation data according to an initial VLM model to obtain an inference chain corresponding to the operation event, wherein the inference chain indicates a model thinking process; a second calculation module, the second calculation module is used to fine-tune train the initial VLM model according to the inference chain to obtain a target VLM model; a third calculation module, the third calculation module is used to automatically explore a path of the agent platform based on a first random task by the target VLM model to obtain a successful sample corresponding to the first random task; and an automation module, the automation module is used to convert a task target into an execution script based on the successful sample, and automatically execute the execution script by the target VLM model in cooperation with RPA to obtain an execution result.

[0113] It can be understood that the specific embodiments of the automatic task execution target device and the specific embodiments of the automatic task execution target method of the embodiments of the present application are basically the same, and will not be repeated here.

[0114] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the automatic task execution target method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0115] Please refer to Figure 7 , Figure 7 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0116] The processor 701 can be implemented in the form of a GPU (Graphics Processing Unit, graphics processing unit) in a private deployment (i.e. in an off-network state), a general CPU (Central Processing Unit, central processing unit) in a SaaS (Software as a Service) deployment (i.e. in an on-network state), a microprocessor, an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application;

[0117] The memory 702 can be implemented in the form of a ROM (ReadOnly Memory, read-only memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory, random access memory), etc. The memory 702 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 702 and called and executed by the processor 701 to implement the automatic task execution target method of the embodiments of the present application;

[0118] The input / output interface 703 is used to realize information input and output;

[0119] The communication interface 704 is used to realize the communication interaction between the device and other devices, which can realize communication through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0120] The bus 705 transmits information between various components (such as the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device;

[0121] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected with each other through the bus 705 to implement communication connection between devices inside.

[0122] In some embodiments, the application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned method for automatically performing a task target.

[0123] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The embodiments described in the application embodiments are used to more clearly illustrate the technical solutions of the application embodiments, and do not constitute a limitation on the technical solutions provided by the application embodiments. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the application embodiments are also applicable to similar technical problems.

[0125] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the application embodiments, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0126] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment scheme.

[0127] Those skilled in the art can understand that all or some steps in the above-mentioned method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0128] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but can be used for clarity, and merely establishes the order unless otherwise stated below.

[0129] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0130] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0132] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0133] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

Claims

1. A method for automating the execution of a task objective, the method comprising: An automation system applied to an agent platform, the automation execution task target method comprising: collecting operation data corresponding to an operation event of the agent platform; performing reasoning based on the operation data according to an initial VLM model to obtain a reasoning chain corresponding to the operation event, wherein the reasoning chain indicates a model thinking process; performing fine-tuning training on the initial VLM model according to the reasoning chain to obtain a target VLM model; exploring a path of the agent platform based on a first random task automation through the target VLM model to obtain a success sample corresponding to the first random task; converting a task target into an execution script based on the success sample, and executing the execution script through the target VLM model in cooperation with RPA automation to obtain an execution result; wherein the reasoning chain comprises a first data chain and a second data chain, and the performing reasoning based on the operation data according to the initial VLM model to obtain the reasoning chain corresponding to the operation event comprises: performing reverse reasoning based on the operation data according to the initial VLM model to obtain the first data chain corresponding to the operation event; performing fine-tuning training on the initial VLM model on the operation data to obtain an intermediate VLM model, and performing reasoning through the intermediate VLM model to obtain a reasoning data chain, and then generating the second data chain corresponding to the operation event based on Bootstrap idea by sampling the reasoning data chain multiple times; wherein the operation data comprises a plurality of sequentially connected real actions, and the generating the second data chain corresponding to the operation event based on Bootstrap idea by sampling the reasoning data chain multiple times further comprises: generating a candidate pair set by randomly sampling the reasoning data chain, wherein the candidate pair set comprises a plurality of candidate pairs, and each candidate pair comprises a predicted action and a predicted thinking; selecting a candidate pair satisfying a matching condition corresponding to the real action from the candidate pair set to obtain a target candidate pair corresponding to the real action, and adding a target thinking of the target candidate pair to the second data chain, wherein the matching condition is that the predicted action of the candidate pair matches the real action; until the target thinking of each target candidate pair corresponding to the real action is added to the second data chain.

2. The method of claim 1, wherein the automation of the execution of the task objective is based on a set of rules. The fine-tuning training on the initial VLM model according to the reasoning chain to obtain the target VLM model comprises: freezing the initial VLM model; adding a low-rank matrix to an Attention layer and a feedforward layer in a Transformer of the initial VLM model; based on the first data chain and the second data chain, fine-tuning training the low-rank matrix by minimizing a preset loss function, and then obtaining the target VLM model.

3. The method of claim 1, wherein the automated execution of the task objective is further based on a user profile of the user. The executing the execution script through the target VLM model in cooperation with RPA automation to obtain an execution result comprises: ​ automatically executing the execution script on the agent platform through the RPA, while monitoring the state of the automatic execution on the agent platform to obtain a state result; In the case that the state result is an abnormal state or a long time without change, path exploration is automatically performed on the agent platform according to the task target through the target VLM model, and the execution result is obtained.

4. The method of claim 3, wherein, The execution script is automatically executed on the agent platform through the target VLM model in cooperation with RPA, and the execution result is obtained. Before the execution script is automatically executed on the agent platform, the interface elements of the screenshot of the agent platform at the current time and the interface elements of the screenshot of the successful sample are compared, and a change value is obtained. In the case that the change value is greater than a threshold value, path exploration is automatically performed on the agent platform according to the task target through the target VLM model, and the execution result is obtained.

5. The method of claim 1, wherein the automated execution of the task objective is further based on a user profile. After the execution script is automatically executed on the agent platform through the target VLM model in cooperation with RPA, and the execution result is obtained, the following steps are further included: The execution result is taken as an execution trajectory sample, and a data cleaning operation is performed on the execution trajectory sample, and an initial training data set is obtained. The target VLM model is fine-tuned based on the initial training data set, and path exploration is automatically performed on the agent platform based on a second random task through the fine-tuned target VLM model, and a to-be-verified sample is obtained. The to-be-verified sample that passes verification is taken as a new successful sample.

6. An apparatus for automatically performing a task objective, characterized by An automatic system applied to an agent platform includes an automatic execution task target device. An acquisition module is configured to acquire operation data corresponding to an operation event of the agent platform. A first calculation module is configured to infer based on an initial VLM model to obtain an inference chain corresponding to the operation event, wherein the inference chain indicates a model thinking process. The inference chain includes a first data chain and a second data chain, and the inference based on the initial VLM model and the operation data to obtain the inference chain corresponding to the operation event includes: The initial VLM model is fine-tuned based on the operation data to obtain an intermediate VLM model, and inference data chain is obtained through the intermediate VLM model, and then the inference data chain is sampled multiple times based on a Bootstrap idea to generate the second data chain corresponding to the operation event. The operation data includes a plurality of real actions connected in sequence, and the second data chain corresponding to the operation event is generated based on the Bootstrap idea includes: A candidate pair set is generated by randomly sampling the inference data chain, wherein the candidate pair set includes a plurality of candidate pairs, and each candidate pair includes a predicted action and a predicted thought. ​ filtering the candidate pairs from the candidate pair set to obtain a target candidate pair corresponding to the real action, and adding a target thought of the target candidate pair to the second data chain, wherein the matching condition is that the predicted action of the candidate pair matches the real action; until the target thought of the target candidate pair corresponding to each real action is added to the second data chain; a second calculation module, configured to fine-tune train the initial VLM model according to the reasoning chain to obtain a target VLM model; a third calculation module, configured to perform path exploration on the agent platform based on a first random task by the target VLM model to obtain a success sample corresponding to the first random task; an automation module, configured to convert a task target into an execution script based on the success sample, and execute the execution script by the target VLM model in cooperation with RPA automation to obtain an execution result.

7. An electronic device, comprising: comprising: at least one processor; at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, the method for automatically executing a task target as claimed in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing computer-executable instructions, the computer-executable instructions comprising: The computer executable instructions are used to execute the method for automatically executing a task target as claimed in any one of claims 1 to 5. The computer executable instructions are used to execute the method for automatically executing a task target as claimed in any one of claims 1 to 5.

Images