Full-scene task autonomous execution method and system based on large model

Abstract

The invention discloses a full-scene task autonomous execution method and system based on a large model, belongs to the technical field of computer application and artificial intelligence, and aims to solve the technical problems that an existing RPA technology is lack of autonomous thinking ability, and a browser use technology is limited in application range, is greatly influenced by factors such as webpage change and the like, and is poor in reliability. The Computer use technology is inaccurate in operation, high in development difficulty and has security risks, and the technical scheme comprises the following steps: establishing an MCP service, and compiling a task autonomous execution tool set containing various basic operation instructions including software opening, browser starting, precise clicking and information filling; the MCP service is deployed in the Windows virtual machine environment, and stable operation of the service is ensured; constructing a task autonomous execution workflow agent with task understanding, planning and scheduling capabilities; the user inputs a task target through a natural language, and the agent receives and records the specific demand of the user as a target guide for subsequent task generation and execution.

Description

Technical Field

[0001] This invention relates to the fields of computer applications and artificial intelligence technology, specifically a method and system for autonomous execution of tasks across all scenarios based on a large model. Background Technology

[0002] With the development of modern society, business volume in various fields has surged and processes have become increasingly complex. A large number of repetitive and tedious tasks, such as data entry and document processing, consume significant human time and energy, and are prone to errors due to fatigue and negligence, leading to losses for enterprises. Simultaneously, complex business processes involve multi-departmental collaboration, making manual coordination inefficient and error-prone. The rapid development of technologies such as computers and artificial intelligence has provided strong support for automation, such as Robotic Process Automation (RPA). RPA is essentially an automation solution based on preset rules and fixed processes, its core pain point being the lack of autonomous "thinking" ability. Simply put, RPA is more like a diligent and efficient but inflexible "digital laborer," highly valuable in scenarios heavily reliant on stable rules, but unable to accurately handle complex businesses requiring real-time judgment and dynamic decision-making.

[0003] Currently, there are two main directions for automated task execution technologies across various industries: browser use (based on browser operations) and computer use (based on computer operations). Browser use utilizes browser automation technologies, such as tools like Selenium and Puppeteer, to simulate user actions like clicking, scrolling, and typing within the browser, thereby automating web page content interaction. It can also leverage visual recognition technology to handle complex web page structures and dynamic content, and is commonly used for automating web page tasks, such as automatically searching for products and completing order processes on e-commerce platforms, extracting key information from web pages to aid market research and analysis, and acting as intelligent customer service to answer frequently asked questions online. However, it has significant drawbacks: frequent changes to web pages, anti-scraping measures on websites, and unstable networks can all affect its effectiveness, and its functionality is limited to web scenarios, resulting in a less broad range of applications. Computer use, on the other hand, works by using AI agents to call the operating system's APIs, operating system-level interfaces, and tools to achieve comprehensive control over the computer, including launching applications, operating the file system, and interacting with desktop applications. It is suitable for automatically handling daily tasks such as email composition and meeting scheduling in enterprise office scenarios, assisting in opening software and writing code during software development, and quickly responding to customer inquiries in the customer service field. However, its development is difficult, as it needs to balance operating system compatibility, security and resource management. Directly operating the computer poses security risks, which can easily lead to malicious operations and data leaks. At the same time, it is slow when performing complex tasks, and the accuracy of operation needs to be improved. Summary of the Invention

[0004] The technical objective of this invention is to provide a method and system for autonomous execution of tasks across all scenarios based on a large model, in order to address the problems of existing RPA technology lacking autonomous thinking capabilities, browser use technology having limited applicability and being greatly affected by factors such as changes in web pages, and computer use technology being inaccurate in operation, difficult to develop, and posing security risks.

[0005] The technical objective of this invention is achieved as follows: a method for autonomous execution of tasks across all scenarios based on a large model, the specific method of which is as follows:

[0006] Build an MCP service and develop a set of task autonomous execution tools that cover various basic operation commands such as opening software, launching a browser, precise clicking, and information entry;

[0007] Deploy the MCP service in a Windows virtual machine environment to ensure stable service operation;

[0008] Construct an intelligent agent capable of autonomously executing task workflows, with the ability to understand, plan, and schedule tasks;

[0009] Specific requirements received from user input: The user inputs the task objective through natural language, and the intelligent agent receives and records the user's specific requirements as a target guide for subsequent task generation and execution.

[0010] By deeply analyzing the user's specific needs through intelligent agents, the user's specific needs are accurately parsed into a clear and actionable list of task steps;

[0011] Based on the parsed task step list and combined with the MCP service tool registration information, the system intelligently matches and obtains the required tool and its corresponding parameters. In other words, it calls the language model to intelligently select the most suitable tool and corresponding parameters (such as application name, coordinate points, text content, etc.) to ensure the accuracy of tool calls and the consistency of context.

[0012] Based on the determined tools and parameters, the corresponding tools are invoked to perform actions. For visually related task operations such as clicking and filling, efficient screen capture technology is used to obtain the current screen screenshot. Multimodal large model is used to analyze the task content and screenshot, accurately locate the coordinates of the target content, and accurately execute the corresponding actions according to the operation type.

[0013] As a preferred option, building an MCP service involves adopting a microservice architecture, using REST APIs to implement inter-service communication, and constructing a highly available MCP service with task scheduling, status management, and log tracking capabilities.

[0014] As a preferred option, the task autonomous execution toolset is written in Python and integrates the Selenium, Browser-Use and PyAutoGUI libraries, supporting core operations such as software startup, browser control, precise clicking and information entry.

[0015] Among them, the task autonomous execution toolset has the capabilities of parameterized configuration, exception handling, logging and task backtracking, ensuring that tasks are controllable, stable and efficient.

[0016] As a preferred option, the MCP service is deployed in a Windows virtual machine environment as follows:

[0017] The Windows virtual machine environment must have at least a 4-core CPU, 8GB of memory, sufficient storage space, and a Python 3.8+ environment set up and configured.

[0018] Use conda to create an independent virtual environment to isolate project dependencies and install the required Flask, Requests, and Selenium dependencies.

[0019] Register the MCP service as a system service to enable it to start automatically in the background;

[0020] By configuring firewall policies and port mapping, the network security and accessibility of services are ensured, providing reliable support for the subsequent execution of automated tasks.

[0021] As a preferred option, the following is a specific method for constructing a task autonomous execution workflow agent capable of understanding, planning, and scheduling tasks:

[0022] Knowledge base construction: Common tasks are embedded and represented based on the m3e-large model and stored in the Milvus vector database, supporting semantic matching and fast retrieval;

[0023] Task understanding and planning: Combining custom task prompt word templates, the large language model Qwen2.5-72B-Instruct-AWQ is invoked to perform semantic parsing of user intent and generate a clear task plan;

[0024] Scheduling and Control: Dynamically schedule MCP services according to the task logic chain, and provide feedback and adjustments on the execution status.

[0025] As a preferred approach, the intelligent agent deeply analyzes the user's specific needs and precisely parses these needs into a clear and actionable list of task steps, as follows:

[0026] The intelligent agent searches the knowledge base and attempts to match the task template that best matches the requirements.

[0027] If no task template closely matches the requirements, the Qwen2.5-72B large model and customized prompts are used to perform in-depth analysis of the requirements, extract the operational intent, target components and process structure, and output a standardized list of task steps.

[0028] More preferably, the analysis of task content and screenshots using a multimodal large model is as follows: image recognition and feature extraction are performed on the acquired screenshots, the task content is converted into text features, and fused with image features. Through training and inference of the Qwen2.5-VL-72B-Instruct-AWQ multimodal model, the coordinate position of the target content in the screenshot is determined.

[0029] A large-scale model-based autonomous task execution system for all scenarios, used to implement the large-scale model-based autonomous task execution method described above; the system includes:

[0030] The MCP service module is used to build MCP services and provide a set of tools for autonomous task execution.

[0031] The service deployment module is used to deploy the MCP service in a Windows virtual machine environment;

[0032] The intelligent agent module is used to build intelligent agents for autonomous task execution workflows, enabling task understanding, planning, and scheduling.

[0033] The requirement receiving module is used to receive specific requirements input by the user.

[0034] The requirement parsing module is used to deeply analyze user requirements using a large language model and parse them into an actionable task list.

[0035] The tool and parameter matching module is used to match the required tools and parameters based on the task content.

[0036] The task execution module is used to invoke tools to perform actions. For click and input operations, it combines screen capture technology and multimodal large model analysis to locate the target coordinates and execute the operation. The task execution module includes a screen capture submodule and a multimodal analysis submodule. The screen capture submodule is used to capture screenshots. The multimodal analysis submodule is used to analyze the task content and screenshots using the Qwen2.5-VL-72B-Instruct-AWQ multimodal model to locate the coordinates of the target content.

[0037] An electronic device includes: a memory and at least one processor;

[0038] The memory contains computer programs;

[0039] The at least one processor executes the computer program stored in the memory, causing the at least one processor to execute the autonomous execution method for full-scene tasks based on a large model as described above.

[0040] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the above-described method for autonomous execution of full-scenario tasks based on a large model.

[0041] The autonomous task execution method and system based on a large model in this invention have the following advantages:

[0042] (I) This invention constructs a closed-loop automated task execution chain of "perception-decision-execution" with multimodal large model and reasoning large model as the core, enabling intelligent agents to autonomously operate computer browsers and other software to achieve autonomous task execution, effectively improving task processing efficiency and quality, breaking through the scenario limitations and accuracy bottlenecks of traditional automation technology, and is suitable for automated processing of computer tasks.

[0043] (II) This invention adopts task autonomous execution technology to effectively improve efficiency, reduce error rate, and optimize business processes, meeting the urgent needs of enterprises for efficient and precise operation. It solves the problems of existing RPA technology lacking autonomous thinking ability, browser use technology having limited applicability and being greatly affected by factors such as webpage changes, and computer use technology having inaccurate operation, high development difficulty, and security risks. It realizes autonomous, efficient, and precise execution of computer tasks, improves task processing efficiency and quality, and optimizes business processes.

[0044] (III) For visual-related tasks such as clicking and filling, this invention uses screen capture technology to collect the current window image and combines it with the Qwen2.5-VL-72B-Instruct-AWQ multimodal large model to perform image and text joint analysis, accurately identify the position coordinates of the target element, and finally complete the corresponding action according to the operation type (click, input, etc.), so as to achieve efficient, accurate and autonomous execution of the task;

[0045] (iv) This invention utilizes a large language model to deeply analyze user needs and transform them into an operable task list, achieving a precise understanding and decomposition of user needs; the multimodal model, combined with efficient screen capture technology, accurately locates the coordinates of the operation target by analyzing the screenshot, breaking through the accuracy bottleneck of traditional operations and making the operation more accurate and reliable; the MCP service provides a toolset covering basic commands such as opening software and browser operations, and is deployed on a Windows virtual machine to ensure stable operation, providing rich tool support and a stable operating environment for task execution; the constructed task autonomous execution workflow intelligent agent realizes task understanding, planning and scheduling, making the entire task execution process more intelligent and orderly;

[0046] (V) This invention is driven by user input requirements, and after requirement analysis and tool parameter matching, it finally calls the tool to complete the task execution, breaking the limitations of the web scenario, realizing the autonomous, efficient and accurate execution of computer tasks in all scenarios, significantly improving task processing efficiency and quality, reducing enterprise labor costs, reducing human error, optimizing business processes, and meeting the urgent needs of enterprises for efficient and accurate operation. Attached Figure Description

[0047] The invention will be further described below with reference to the accompanying drawings.

[0048] Appendix Figure 1 This is a flowchart illustrating the autonomous execution method for full-scenario tasks based on a large model. Detailed Implementation

[0049] The following detailed description of the autonomous task execution method and system based on a large model for all scenarios of the present invention is provided with reference to the accompanying drawings and specific embodiments.

[0050] Example 1:

[0051] As attached Figure 1 As shown in the figure, this embodiment provides a method for autonomous execution of tasks across all scenarios based on a large model. The method is as follows:

[0052] S1. Build the MCP service and write a set of task autonomous execution tools that cover various basic operation instructions such as opening software, launching browser, precise clicking and information entry;

[0053] S2. Deploy the MCP service in a Windows virtual machine environment to ensure stable service operation;

[0054] S3. Construct a task autonomous execution workflow intelligent agent with the ability to understand, plan, and schedule tasks;

[0055] S4. Receive specific user input requirements: The user inputs the task objective through natural language. The intelligent agent receives and records the user's specific requirements as a target guide for subsequent task generation and execution.

[0056] S5. Through intelligent agents, deeply analyze the user's specific needs and accurately parse the user's specific needs into a clear and operable list of task steps.

[0057] S6. Based on the parsed task step list and combined with the MCP service tool registration information, intelligently match and obtain the required tool and its corresponding parameters. That is, call the language model to intelligently select the most suitable tool and corresponding parameters (such as application name, coordinate points, text content, etc.) to ensure the accuracy of tool calls and the consistency of context.

[0058] S7. Based on the determined tools and parameters, call the corresponding tools to perform actions; among them, for visual-related task operation types such as click and fill, use efficient screen capture technology to obtain the current screen screenshot, use multimodal large model to analyze the task content and screenshot, accurately locate the target content coordinates, and accurately execute the corresponding actions according to the operation type.

[0059] In this embodiment, step S1 of building the MCP service specifically involves: adopting a microservice architecture design, using REST API to implement inter-service communication, and building a highly available MCP service with task scheduling, status management, and log tracking functions.

[0060] The task autonomous execution toolset in step S1 of this embodiment is written in Python and integrates Selenium, Browser-Use and PyAutoGUI libraries, supporting core operations such as software startup, browser control, precise clicking and information entry.

[0061] Among them, the task autonomous execution toolset has the capabilities of parameterized configuration, exception handling, logging and task backtracking, ensuring that tasks are controllable, stable and efficient.

[0062] The specific steps in step S2 of this embodiment, namely deploying the MCP service in the Windows virtual machine environment, are as follows:

[0063] S201, The Windows virtual machine environment must have at least a 4-core CPU, 8GB of memory, sufficient storage space, and a Python 3.8+ environment must be set up and configured;

[0064] S202. Use conda to create an independent virtual environment to isolate project dependencies and install the required Flask, Requests and Selenium dependency packages.

[0065] S203. Register the MCP service as a system service to enable background auto-start;

[0066] S204. Through firewall policies and port mapping configurations, ensure the network security and accessibility of services, and provide reliable support for the subsequent operation of automated tasks.

[0067] In this embodiment, step S3 involves constructing a task autonomous execution workflow agent capable of understanding, planning, and scheduling tasks, as detailed below:

[0068] S301. Knowledge base construction: Common tasks are embedded and represented based on the m3e-large model and stored in the Milvus vector database, supporting semantic matching and fast retrieval.

[0069] S302, Task Understanding and Planning: Combining a custom task prompt word template, the large language model Qwen2.5-72B-Instruct-AWQ is called to perform semantic parsing of user intent and generate a clear task plan;

[0070] S303, Scheduling and Control: Dynamically schedule MCP services according to the task logic chain, and provide feedback and adjustment on the execution status.

[0071] In step S5 of this embodiment, the user's specific needs are deeply analyzed by the intelligent agent, and the user's specific needs are accurately parsed into a clear and operable list of task steps, as follows:

[0072] The intelligent agent searches the knowledge base and attempts to match the task template that best matches the requirements.

[0073] If no task template closely matches the requirements, the Qwen2.5-72B large model and customized prompts are used to perform in-depth analysis of the requirements, extract the operational intent, target components and process structure, and output a standardized list of task steps.

[0074] In step S7 of this embodiment, the analysis of the task content and screenshot using a multimodal large model is as follows: image recognition and feature extraction are performed on the acquired screenshot, the task content is converted into text features, and fused with image features. Through training and inference of the Qwen2.5-VL-72B-Instruct-AWQ multimodal model, the coordinate position of the target content in the screenshot is determined.

[0075] Example 2:

[0076] This embodiment provides a full-scenario task autonomous execution system based on a large model. This system is used to implement the full-scenario task autonomous execution method based on a large model as described in Embodiment 1. The system includes:

[0077] The MCP service module is used to build MCP services and provide a set of tools for autonomous task execution.

[0078] The service deployment module is used to deploy the MCP service in a Windows virtual machine environment;

[0079] The intelligent agent module is used to build intelligent agents for autonomous task execution workflows, enabling task understanding, planning, and scheduling.

[0080] The requirement receiving module is used to receive specific requirements input by the user.

[0081] The requirement parsing module is used to deeply analyze user requirements using a large language model and parse them into an actionable task list.

[0082] The tool and parameter matching module is used to match the required tools and parameters based on the task content.

[0083] The task execution module is used to invoke tools to perform actions. For click and input operations, it combines screen capture technology and multimodal large model analysis to locate the target coordinates and execute the operation. The task execution module includes a screen capture submodule and a multimodal analysis submodule. The screen capture submodule is used to capture screenshots. The multimodal analysis submodule is used to analyze the task content and screenshots using the Qwen2.5-VL-72B-Instruct-AWQ multimodal model to locate the coordinates of the target content.

[0084] Example 3:

[0085] This embodiment also provides an electronic device, including: a memory and a processor;

[0086] The memory stores the instructions executed by the computer.

[0087] The processor executes the computer execution instructions stored in the memory, causing the processor to execute the autonomous execution method for full-scene tasks based on a large model in any embodiment of the present invention.

[0088] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0089] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0090] Example 4:

[0091] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the autonomous task execution method based on a large model in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or device may read and execute the program code stored in the storage medium.

[0092] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0093] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0094] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0095] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous execution of tasks across all scenarios based on a large model, characterized in that, The method is as follows: Build an MCP service and develop a set of task autonomous execution tools that cover various basic operation commands such as opening software, launching a browser, precise clicking, and information entry; Deploy the MCP service in a Windows virtual machine environment to ensure stable service operation; Construct an intelligent agent capable of autonomously executing task workflows, with the ability to understand, plan, and schedule tasks; Specific requirements received from user input: The user inputs the task objective through natural language, and the intelligent agent receives and records the user's specific requirements as a target guide for subsequent task generation and execution. By deeply analyzing the user's specific needs through intelligent agents, the user's specific needs are accurately parsed into a clear and actionable list of task steps; Based on the parsed task step list and combined with the MCP service tool registration information, the system intelligently matches and obtains the required tools and their corresponding parameters. In other words, it calls the language model to intelligently select the most suitable tool and corresponding parameters to ensure the accuracy and context consistency of tool calls. Based on the determined tools and parameters, the corresponding tools are invoked to perform actions. For visually related task operations such as clicking and filling, efficient screen capture technology is used to obtain the current screen screenshot. Multimodal large model is used to analyze the task content and screenshot, accurately locate the coordinates of the target content, and accurately execute the corresponding actions according to the operation type.

2. The method for autonomous execution of full-scenario tasks based on a large model according to claim 1, characterized in that, The specific steps to build an MCP service are as follows: adopt a microservice architecture design, use REST API to implement inter-service communication, and build a highly available MCP service with task scheduling, status management and log tracking functions.

3. The method for autonomous execution of full-scenario tasks based on a large model according to claim 1, characterized in that, The task autonomous execution toolset is written in Python and integrates Selenium, Browser-Use and PyAutoGUI libraries, supporting core operations such as software startup, browser control, precise clicks and information entry. Among them, the task autonomous execution toolset has the capabilities of parameterized configuration, exception handling, logging and task backtracking, ensuring that tasks are controllable, stable and efficient.

4. The method for autonomous execution of full-scenario tasks based on a large model according to claim 1, characterized in that, The specific steps for deploying the MCP service in a Windows virtual machine environment are as follows: The Windows virtual machine environment must have at least a 4-core CPU, 8GB of memory, sufficient storage space, and a Python 3.8+ environment set up and configured. Use conda to create an independent virtual environment to isolate project dependencies and install the required Flask, Requests, and Selenium dependencies. Register the MCP service as a system service to enable it to start automatically in the background; By configuring firewall policies and port mapping, the network security and accessibility of services are ensured, providing reliable support for the subsequent execution of automated tasks.

5. The method for autonomous execution of full-scenario tasks based on a large model according to claim 1, characterized in that, The specific steps for constructing a task-autonomous workflow agent with the ability to understand, plan, and schedule tasks are as follows: Knowledge base construction: Common tasks are embedded and represented based on the m3e-large model and stored in the Milvus vector database, supporting semantic matching and fast retrieval; Task understanding and planning: Combining custom task prompt word templates, the large language model Qwen2.5-72B-Instruct-AWQ is invoked to perform semantic parsing of user intent and generate a clear task plan; Scheduling and Control: Dynamically schedule MCP services according to the task logic chain, and provide feedback and adjustments on the execution status.

6. The method for autonomous execution of full-scenario tasks based on a large model according to claim 1, characterized in that, By deeply analyzing the user's specific needs through intelligent agents, the user's specific needs are accurately parsed into a clear and actionable list of task steps, as follows: The intelligent agent searches the knowledge base and attempts to match the task template that best matches the requirements. If no task template closely matches the requirements, the Qwen2.5-72B large model and customized prompts are used to perform in-depth analysis of the requirements, extract the operational intent, target components and process structure, and output a standardized list of task steps.

7. The autonomous execution method for full-scenario tasks based on a large model according to any one of claims 1-6, characterized in that, The analysis of task content and screenshots using a multimodal large model is as follows: image recognition and feature extraction are performed on the acquired screenshots, the task content is converted into text features, and fused with image features. Through training and inference of the Qwen2.5-VL-72B-Instruct-AWQ multimodal model, the coordinate position of the target content in the screenshot is determined.

8. A full-scenario autonomous task execution system based on a large model, characterized in that, This system is used to implement the autonomous execution method for full-scene tasks based on a large model as described in any one of claims 1 to 7; the system includes: The MCP service module is used to build MCP services and provide a set of tools for autonomous task execution. The service deployment module is used to deploy the MCP service in a Windows virtual machine environment; The intelligent agent module is used to build intelligent agents for autonomous task execution workflows, enabling task understanding, planning, and scheduling. The requirement receiving module is used to receive specific requirements input by the user. The requirement parsing module is used to deeply analyze user requirements using a large language model and parse them into an actionable task list. The tool and parameter matching module is used to match the required tools and parameters based on the task content. The task execution module is used to invoke tools to perform actions. For click and input operations, it combines screen capture technology and multimodal large model analysis to locate the target coordinates and execute the operation. The task execution module includes a screen capture submodule and a multimodal analysis submodule. The screen capture submodule is used to capture screenshots. The multimodal analysis submodule is used to analyze the task content and screenshots using the Qwen2.5-VL-72B-Instruct-AWQ multimodal model to locate the coordinates of the target content.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the autonomous execution method for full-scene tasks based on a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the autonomous execution method for full-scene tasks based on a large model as described in any one of claims 1 to 7.

Images