A GUI agent system based on MCP protocol
By using a GUI intelligent agent system based on the MCP protocol, tasks are broken down into basic subtasks and operation instructions are dynamically matched, which solves the shortcomings of existing GUI systems in understanding user intent and planning and executing tasks, and achieves efficient and stable office automation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING TAIJI INFORMATION SYST TECH CO LTD
- Filing Date
- 2026-02-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing GUI-based office automation systems have shortcomings in understanding user intent and planning and executing tasks. They struggle to handle unstructured input, lack dynamic task decomposition and load balancing capabilities, resulting in low execution efficiency and poor robustness.
A GUI intelligent agent system based on the MCP protocol is adopted. By establishing a task description model, complex tasks are decomposed into basic subtasks, and operation instructions are dynamically matched in an extensible instruction component library. Combined with intelligent agent construction and cluster computing, load balancing and task migration are achieved, thereby improving execution efficiency and stability.
It improves the efficiency of office task execution, reduces execution deviations, ensures the continuity and stability of system services, and adapts to the flexible needs of complex multi-step operations.
Smart Images
Figure CN122120302A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of GUI intelligent agent technology, and in particular to a GUI intelligent agent system based on the MCP protocol. Background Technology
[0002] In the current field of office automation, graphical user interface (GUI)-based automation technologies, such as Robotic Process Automation (RPA), have been widely adopted. However, as enterprises deepen their digital transformation and business scenarios become increasingly complex, the limitations of existing systems are becoming more and more apparent.
[0003] First, at the user intent understanding level, existing systems heavily rely on preset rules or precise script instructions. They have weak processing capabilities for unstructured inputs such as voice commands and natural language descriptions, and lack deep semantic analysis and task element extraction capabilities. This results in insufficient system intelligence, requiring users to express their needs in a highly structured manner that conforms to specific formats, leading to a high learning and usage threshold and making it difficult to adapt to flexible and impromptu office needs.
[0004] Secondly, at the task planning and execution level, mainstream solutions mostly adopt pre-arranged fixed processes. This model lacks the ability to dynamically decompose tasks and generate instruction sequences, and cannot intelligently plan and adjust according to real-time context and specific task objectives. At the same time, task execution usually lacks awareness and dynamic scheduling of backend computing resources, making it difficult to achieve load balancing and optimal efficiency. When faced with complex, multi-step cross-application operations, its robustness and execution efficiency are often difficult to guarantee. Summary of the Invention
[0005] The purpose of this application is to provide a GUI intelligent agent system based on the MCP protocol to solve the above-mentioned technical problems, aiming to improve the efficiency of office task execution.
[0006] In some embodiments of this application, complex task requirements are broken down into multiple basic subtasks by establishing a task description model. The most suitable operation instructions are dynamically matched for each subtask in an extensible "instruction component library". Through the flexible arrangement of basic instructions, complex business processes can be dealt with. Algorithm matching ensures that each operation step is highly consistent with the task objective, reduces execution deviation, and improves the execution efficiency of office tasks.
[0007] In some embodiments of this application, intelligent agents are constructed to quantify the execution load of each subtask, thereby achieving load balancing of cluster computing capabilities, avoiding single-point bottlenecks, and automatically migrating tasks to healthy nodes when some nodes experience performance fluctuations or failures, ensuring the overall service continuity and stability of the system.
[0008] In some embodiments of this application, a GUI intelligent agent system based on the MCP protocol is provided, including:
[0009] The front-end extension unit is used to collect office demand data and generate task demand packages based on the office demand data; The cloud-based central control unit is used to build intelligent agents; The intelligent agent is used to set the sequence of operation instructions according to the task requirement package; The MCP connection unit is used to transmit interactive data between the front-end expansion unit and the cloud central control unit; The front-end extension unit includes: Extension modules are used to build an interactive extension structure with the terminal browser; The extension module is also used to execute operation instruction sequences through interactive extension structures.
[0010] In some embodiments of this application, the front-end extension unit further includes: The requirement processing module is used to generate task text libraries and interference text libraries based on historical data. Generate a speech sub-model based on a library of interfering texts; Generate task sub-models based on the task text library; A semantic parsing model is established based on the speech sub-model and the task sub-model; The requirement processing module is also used to generate task requirement packages based on office requirement data and semantic parsing models.
[0011] In some embodiments of this application, the front-end extension unit further includes: The uplink transmission module is used to acquire the execution feedback data of the operation instruction sequence and generate uplink data packets based on the execution results; The storage module is used to establish a storage library; The environment perception module is used to acquire execution feedback data of the sequence of operation instructions.
[0012] In some embodiments of this application, the demand processing module is further configured to: Determine the category of office demand data; If the office requirement data is voice data, generate a first-level processing instruction; The speech sub-model converts speech data into text data according to the primary processing instructions; The task sub-model extracts task keywords and related content from text data; Generate a task requirement package based on the extraction results; If the office requirement data is text data, generate a secondary processing instruction; The task sub-model extracts task keywords and related content from the office demand data based on the secondary processing instructions; A task requirement package is generated based on the extraction results.
[0013] In some embodiments of this application, the cloud-based central control unit includes: The resource awareness module is used to configure multiple computing nodes; The resource awareness module is also used to acquire the running status data of each computing node; Build modules to create task description models; The building module is also used to generate multiple instruction sub-components based on historical data; Build an instruction sub-component library based on all instruction sub-components; The building module is also used to establish a scheduling sub-model based on all computing nodes; Based on the task description model, instruction component library, and scheduling sub-model, an intelligent agent is constructed. The downlink module is used to generate downlink data packets based on the sequence of operation instructions.
[0014] In some embodiments of this application, the intelligent agent is used to set a sequence of operation instructions according to the task requirement package, including: The task description model reads the task requirement package; Multiple sub-tasks are generated based on the read results; Select the target subtask sequentially from all the required subtasks; Establish an instruction sub-array A, A=(a1, a2, ..., ai, ..., an) based on the instruction sub-array library, where ai is the i-th instruction sub-array and n is the number of instruction sub-arrays; Generate the fit values between the target subtask and each instruction component; Set the operation instructions for the target subtask based on the instruction component corresponding to the maximum value among all matching values; Generate operation instructions for each required subtask in sequence; Generate a first-level operation sequence based on all required sub-tasks; The sequence of operation instructions is set according to the first-level operation sequence and all operation instructions.
[0015] In some embodiments of this application, the intelligent agent is further used for: Select the sub-tasks to be evaluated sequentially from all the required sub-tasks; Generate the execution load values for the subtasks to be evaluated; Establish a sequence of computing nodes B, B=(b1,b2…bi…bm), where bi is the i-th computing node and m is the number of computing nodes; Select the execution computing node of the to-be-evaluated subtask from the computing node sequence B according to the execution load value; Set the execution computing nodes of each demand subtask in sequence.
[0016] In some embodiments of the present application, the selection of the execution computing node of the to-be-evaluated subtask includes: Select target computing nodes in sequence according to the computing node sequence B; Obtain the operation status data of the target computing node and generate the operation computing power value of the target computing node; Generate the operation computing power values of each computing node in sequence; Generate the matching value between the to-be-evaluated subtask and each computing node according to all the operation computing power values and the execution load value of the to-be-evaluated subtask; Set the computing node corresponding to the maximum value among all the matching values as the execution computing node of the to-be-evaluated subtask.
[0017] In some embodiments of the present application, the storage module is further used for: Establish a business subsystem sequence W, W = (w1, w2... wi... wr), where wi is the ith business subsystem; r is the number of business subsystems; Set wi as the target subsystem in sequence according to the business subsystem sequence W; Generate the operation association value between the target subsystem and each instruction sub-component; Select multiple associated instruction sub-components of the target subsystem according to all the operation association values; Generate the call paths of each associated instruction sub-component; Establish a storage sub-library of the target operating system according to all the call paths; Set the storage sub-libraries of each business subsystem in sequence; Construct a storage library according to all the storage sub-libraries.
[0018] In some embodiments of the present application, the selection of multiple associated instruction sub-components of the target subsystem includes: Preset an operation association value threshold C1; If C1 < ci, (i = 1, 2... n), set the ith instruction sub-component as the associated instruction sub-component of the target subsystem; Where, ci is the operation association value between the ith instruction sub-component and the target subsystem; n is the number of instruction sub-components.
[0019] Compared with the prior art, the beneficial effects of a GUI agent system based on the MCP protocol in an embodiment of the present application are as follows: By establishing a task description model, complex task requirements are broken down into multiple basic subtasks. The most suitable operation instructions are dynamically matched for each subtask in an extensible "instruction component library". Through the flexible arrangement of basic instructions, complex business processes can be dealt with. Algorithm matching ensures that each operation step is highly consistent with the task objective, reduces execution deviation, and improves the execution efficiency of office tasks.
[0020] By constructing intelligent agents to quantify the execution load of each subtask, load balancing of cluster computing power is achieved, avoiding single-point bottlenecks. When the performance of some nodes fluctuates or fails, tasks can be automatically migrated to healthy nodes, ensuring the overall service continuity and stability of the system. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of the structure of a GUI intelligent agent system based on the MCP protocol in a preferred embodiment of this application. Detailed Implementation
[0022] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.
[0023] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0024] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0025] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0026] like Figure 1 As shown, a preferred embodiment of this application provides a GUI intelligent agent system based on the MCP protocol, comprising: The front-end extension unit is used to collect office demand data and generate task demand packages based on the office demand data; The cloud-based central control unit is used to build intelligent agents; The intelligent agent is used to set the sequence of operation instructions according to the task requirements package; The MCP connection unit is used to transmit interactive data between the front-end expansion unit and the cloud central control unit; The front-end extension unit includes: Extension modules are used to build an interactive extension structure with the terminal browser; The extension module is also used to execute sequences of operation instructions through interactive extension structures.
[0027] Specifically, the extension module generates an interactive extension structure by establishing a long-lived connection with the browser rendering process using the debugging protocol. This interactive extension structure can then be used to convert the decoded MCP operation commands into native browser control commands.
[0028] Specifically, the MCP connection unit is responsible for the formatting, efficient compression, and reliable transmission of all data exchanged between the cloud and the endpoint.
[0029] Specifically, the interactive extension structure also obtains voice data through the browser's audio application programming interface (API) and text data through the browser's text application architecture.
[0030] Specifically, the front-end extension unit also includes: The requirement processing module is used to generate task text libraries and interference text libraries based on historical data. Generate a speech sub-model based on a library of interfering texts; Generate task sub-models based on the task text library; A semantic parsing model is established based on the speech sub-model and the task sub-model; The requirements processing module is also used to acquire office requirements data and generate task requirement packages based on the office requirements data; The semantic parsing module is used to obtain office requirement data; The semantic parsing module is used to generate task requirement packages based on office requirement data and semantic parsing models; The uplink transmission module is used to acquire the execution feedback data of the operation instruction sequence and generate uplink data packets based on the execution results; The storage module is used to establish a storage library; The environment perception module is used to acquire execution feedback data of the sequence of operation instructions.
[0031] Specifically, the uplink transmission module is also used to receive the downlink operation command sequence from the cloud central control unit, unpack, decode and verify it, and distribute it to the extension module. The extension module translates the abstract operation commands, such as element positioning, clicking and input, into a series of native browser control command sequences and injects them for execution.
[0032] Specifically, historical data refers to acquired historical office demand data (including historical input voice and text data). By filtering and analyzing the voice data in the historical data, multiple useless voice words are generated (e.g., repeated words, personal colloquialisms, environmental noise, echoes, etc.). An interference text library is constructed based on all useless voice words, and multiple voice features are generated (e.g., rapid intonation corresponds to urgent tasks; features such as speech rate and pause intervals are extracted). A voice feature library is built, and a voice sub-model is established using the interference text library and the sentiment text library. The voice sub-model is used to perform initial processing and analysis of the voice data, converting it into text data and determining the urgency of the task.
[0033] Specifically, by filtering and analyzing text data in historical records, multiple task keywords (such as summarizing, filtering, reporting, and other words highly related to office tasks) are generated, and a task text library is constructed based on all task keywords.
[0034] Specifically, the task sub-model in the semantic parsing model is based on the BERT-based pre-trained language model and is specifically optimized for office scenarios. On the one hand, a dedicated corpus covering core enterprise office scenarios such as financial statements, personnel approvals, data aggregation, and document collaboration is constructed. This corpus defines multiple core intents, including data statistics, document generation, process approval, and information query, as well as multiple constraints such as time range, data format, recipient, permission requirements, data range, and operation type. On the other hand, the model structure is modified, replacing the original output layer with an "intent classification head and constraint extraction head." The intent classification head uses a fully connected layer and Softmax activation, while the constraint extraction head uses a CRF layer for sequence labeling. Training is conducted using the AdamW optimizer, freezing the underlying parameters to retain general semantic understanding capabilities while fine-tuning the upper-layer parameters to adapt to the needs of office scenarios. The optimized model can transform unstructured input into standardized structured results. For example, if a user inputs "Generate the Q3 sales department Excel report and send it to Manager Zhang before 6 PM tomorrow," the model can accurately output the core intent of "document generation and data statistics," along with corresponding constraints such as time, data range, and format.
[0035] Specifically, the interference text library, speech feature library, and task text library can be updated periodically to improve the extraction accuracy of key task elements of the semantic parsing model, and enhance the naturalness of human-computer interaction and the accuracy of intent recognition.
[0036] Specifically, the environment perception module can receive visual patch features from 3-5 consecutive screenshots, learn inter-frame dependencies through a temporal self-attention mechanism, and accurately capture element position movement and state switching (such as buttons changing from disabled to enabled, windows popping up and closing).
[0037] Specifically, a lightweight temporal Transformer module is introduced after the GUI visual encoder to build a corresponding environment awareness module. Simultaneously, a dynamic change detector is added. By calculating the cosine similarity of visual patches from adjacent frames, significantly different regions are automatically marked, and these regions are prioritized for localization verification, greatly improving response speed in dynamic scenes. At the training level, based on the original 1M screenshot training data, 300K dynamic scene samples containing element state labels (enabled / disabled, visible / hidden, selected / selected) are added. State information is naturally integrated into the multi-patch supervision process, allowing the environment awareness module to automatically filter out non-interactive elements while learning to locate target regions.
[0038] Specifically, the environment perception module addresses the limitations of a fixed 28×28 pixel patch granularity in fine-grained element localization and complex interference scenarios by employing an adaptive granularity strategy to optimize the patch generation mechanism: based on the element density of the input image, it generates 8×8 pixel fine-grained patches in high-density areas (such as toolbars and icon groups) to accurately capture extremely small elements smaller than 10×10 pixels (such as CAD software control points and browser icons), and generates 32×32 pixel coarse-grained patches in low-density areas (such as blank backgrounds and large buttons), thus balancing positioning accuracy and computational efficiency.
[0039] Specifically, the extension module can directly associate semantic instructions with visual regions based on the sequence of operation instructions. First, it applies a self-attention layer to the visual patch features extracted by the VLM visual encoder, aggregating the semantics of spatially related blocks. Then, it uses two independent MLPs to... <actor>The hidden state of the token and the contextualized patch features are projected into a shared embedding space. Attention scores are then calculated to form a spatial activation map, generating multiple candidate operation regions in a single forward pass. During training, the extension module employs a multi-patch supervision strategy, labeling all visual patches partially or completely covered by ground truth bounding boxes as positive samples and the rest as negative samples. This effectively tolerates spatial ambiguity in GUI interactions and avoids excessive penalty for reasonable operation changes from single-point predictions. Furthermore, a lightweight grounded validator, acting as a decision refinement layer, receives instructions and visually labeled screenshots, predicts whether the labeled regions match the task intent, and selects the optimal solution from multiple candidate regions through cross-entropy loss training, further improving localization accuracy.
[0040] Specifically, the demand processing module is also used for: Determine the category of office demand data; If the office requirement data is voice data, generate a first-level processing instruction; The speech sub-model converts speech data into text data according to the primary processing instructions; The task sub-model extracts task keywords and related content from text data; Generate a task requirement package based on the extraction results; If the office requirement data is text data, generate a secondary processing instruction; The task sub-model extracts task keywords and related content from the office demand data based on the secondary processing instructions; A task requirement package is generated based on the extraction results.
[0041] Specifically, the speech sub-model removes redundant noise and useless words from the original speech data according to the first-level processing instructions, and converts the removed speech data into text data, thereby achieving in-depth purification and intent refinement of colloquial speech.
[0042] Specifically, the task sub-model identifies and filters task keywords in the text data, while retaining the contextual content (i.e. related content) of the identified task keywords, and generates a task requirement package based on the filtering results.
[0043] Specifically, the related content of task keywords is the relevant context. By adding a dimensional context awareness mechanism to the task sub-model, the accuracy of intent recognition can be further improved.
[0044] It is understandable that in the above embodiments, a front-end and back-end separation architecture based on the MCP protocol is adopted. The front-end extension unit is responsible for collecting user and environmental needs and executing commands; the cloud-based central control unit is used for intelligent agent construction and task planning. The MCP protocol, as a standardized communication bridge, ensures the standardization, traceability, and security of interactive data. The front-end extension unit, by building interactive extension modules and embedding them into various browser environments, achieves non-intrusive intelligent enhancement of the office system, improving the system's compatibility and usability.
[0045] In a preferred embodiment of this application, the cloud-based central control unit includes: The resource awareness module is used to configure multiple computing nodes; The resource awareness module is also used to acquire the running status data of each computing node; Build modules to create task description models; The module is also used to generate multiple instruction sub-components based on historical data; Build an instruction sub-component library based on all instruction sub-components; The module is also used to build a scheduling sub-model based on all computing nodes; Based on the task description model, instruction component library, and scheduling sub-model, an intelligent agent is constructed. The downlink module is used to generate downlink data packets based on the sequence of operation instructions.
[0046] Specifically, the resource awareness module establishes multiple computing nodes based on the hardware devices (GPU, CPU, NPU) of the office system, and sets the collection indicators for each computing node. These collection indicators include, but are not limited to, parameters that map the operating status of the hardware devices, such as load, network bandwidth, cache utilization, core temperature, video memory utilization, and task queue length.
[0047] Specifically, multiple basic tasks (data processing, document editing, message sending, etc.) are generated by analyzing historical data, and operation instructions corresponding to each basic task are constructed. Multiple instruction sub-components are generated based on all basic tasks, where each instruction sub-component represents the operation instructions for a technical task.
[0048] Specifically, an office scenario task dataset (including a large number of office instructions) is constructed. Based on the optimization and iteration results of the office scenario task dataset, a task description model is generated. This task description model can accurately output structured task description objects. For example, if the task requirement package is "summarize 10 Excel files and generate a report", the task description model can break it down into four sub-tasks: reading the Excel reports, summarizing the data, generating the report, and sending emails.
[0049] Specifically, the intelligent agent is used to set a sequence of operation instructions based on the task requirements package, including: The task description model reads the task requirement package; Multiple sub-tasks are generated based on the read results; Select the target subtask sequentially from all the required subtasks; Establish an instruction sub-array A, A=(a1, a2, ..., ai, ..., an) based on the instruction sub-array library, where ai is the i-th instruction sub-array and n is the number of instruction sub-arrays; Generate the fit values between the target subtask and each instruction component; Set the operation instructions for the target subtask based on the instruction component corresponding to the maximum value among all matching values; Generate operation instructions for each required subtask in sequence; Generate a first-level operation sequence based on all required sub-tasks; The sequence of operation instructions is set according to the first-level operation sequence and all operation instructions.
[0050] Specifically, a matching value is generated by analyzing the similarity between the target subtask and the basic task corresponding to the current instruction sub-component. The higher the similarity, the larger the matching value. The mapping relationship between the two can be set according to historical parameters.
[0051] Specifically, the downlink module caches ten operation instructions by default to avoid queue overflow.
[0052] It is understood that in the above embodiments, by establishing a task description model, complex task requirements are broken down into multiple basic subtasks, and the most suitable operation instructions are dynamically matched for each subtask in an extensible "instruction component library". Through the flexible arrangement of basic instructions, complex business processes can be dealt with, and algorithm matching ensures that each operation step is highly consistent with the task objective, reducing execution deviations and improving the execution efficiency of office tasks.
[0053] In a preferred embodiment of this application, the intelligent agent is further used for: Select the sub-tasks to be evaluated sequentially from all the required sub-tasks; Generate the execution load values for the subtasks to be evaluated; Establish a sequence of computing nodes B, B=(b1,b2…bi…bm), where bi is the i-th computing node and m is the number of computing nodes; Select the execution computing node for the subtask to be evaluated from the computing node sequence B based on the execution load value; Set the execution and computation nodes for each requirement subtask in sequence.
[0054] Specifically, the execution load value is set according to the computing resources required by the subtask to be evaluated; the more computing resources required, the larger the corresponding execution load value. The mapping relationship between the two can be set based on historical parameters.
[0055] Specifically, the execution computation nodes of the subtask to be evaluated are selected, including: Select the target computation nodes sequentially according to the computation node sequence B; Acquire the running status data of the target computing node and generate the running computing power value of the target computing node; The computing power values of each computing node are generated sequentially; The matching values between the subtasks to be evaluated and each computing node are generated based on the total computing power value and the execution load value of the subtasks to be evaluated. The computation node corresponding to the maximum value among all matched values is set as the execution computation node of the subtask to be evaluated.
[0056] Specifically, when calculating the computing power of the target node, a "weighted moving average" algorithm is used to smoothly collect data (the preferred weight parameters in this application are: weight: most recent = 0.4, previous 2 = 0.3, previous 3 = 0.3) to avoid misjudgments caused by hardware load fluctuations.
[0057] Specifically, when calculating the computing power of a target node, historical indicators such as historical failure rate and task completion success rate are collected to generate the health of the target node. The higher the historical failure rate and the lower the task completion success rate, the higher the corresponding health. The mapping relationship between the two can be set according to historical parameters.
[0058] Specifically, a corresponding compensation coefficient is set based on the health status of the target node. The higher the health status, the larger the corresponding compensation coefficient. The mapping relationship between the two can be set based on historical parameters. The computing power value of the target node is then corrected using the compensation coefficient.
[0059] Specifically, a matching mapping table of computing power value and execution load value is constructed based on historical operating parameters. This generates a matching value between the computing power value of each computing node and the subtask to be evaluated. The larger the matching value, the more suitable the current computing node is for executing the subtask to be evaluated.
[0060] Specifically, when the computing node load is too high, the agent recalculates the matching value and migrates unexecuted subtasks to idle nodes; when parallel subtask data is missing, ready subtasks are executed first, and the missing subtasks are executed after the data is supplemented; when the target changes, subtasks are re-decomposed (e.g., a new "format conversion" subtask is added), and the operation instruction sequence is updated. At the same time, clear adjustment priority rules are set: target layer adjustment > resource layer adjustment > data layer adjustment; urgent task adjustment > regular task adjustment. It is understood that in the above embodiments, by constructing intelligent agents to quantify the execution load of each subtask, the load balancing of the cluster computing capacity is achieved, avoiding single-point bottlenecks. When the performance of some nodes fluctuates or fails, the tasks can be automatically migrated to healthy nodes, ensuring the overall service continuity and stability of the system.
[0061] In a preferred embodiment of this application, the storage module is further used for: Establish a sequence W, W=(w1, w2…wi…wr) of business subsystems, where wi is the i-th business subsystem and r is the number of business subsystems; Based on the business subsystem sequence W, wi is sequentially set as the target subsystem; Generate the operational association values between the target subsystem and each instruction component; Select multiple associated instruction components of the target subsystem based on all operation association values; Generate the call paths for each associated instruction component; Establish a storage sub-library for the target operating system based on all call paths; Configure the storage sub-databases for each business subsystem in sequence; Construct a storage library based on all storage sub-libraries.
[0062] Specifically, the call path refers to the process of calling specific data objects and database fields within the target subsystem based on the historical execution records of the currently associated instruction subsystem. By establishing call paths for each associated instruction subsystem, instruction execution can skip the time-consuming exploration and trial-and-error process and directly access the correct data source.
[0063] Specifically, historical execution records refer to the data obtained by combining screenshots, UI element attributes (type, label, position), window hierarchy information, and real user interaction records, such as clicks, inputs, and gestures, to provide samples for the model to learn human operational patterns. This allows the intelligent agent to perform precise operations in dynamic interfaces based on the operational needs corresponding to associated instruction components. Examples include automatically locating the search bar and entering keywords in a webpage, or finding a specific button in a desktop application to perform copy and paste operations.
[0064] Specifically, multiple associated instruction components of the target subsystem are selected, including: Preset operation association value threshold C1; If C1 < ci, (i = 1, 2…n), set the i-th instruction sub-component as the associated instruction sub-component of the target subsystem; Where, ci is the operation association value between the i-th instruction sub-component and the target subsystem; n is the number of instruction sub-components.
[0065] Specifically, the business subsystem includes but is not limited to each independent business area of office systems such as enterprise email, human resources subsystem, financial subsystem, etc.
[0066] Specifically, if the operation instruction corresponding to the current instruction sub-component has been executed in the target subsystem, set the operation association value between the target subsystem and the current instruction sub-component to 1, and if not, set the operation association value to 0.
[0067] Specifically, the value range of the operation association value threshold in this application is (0, 1).
[0068] According to the first concept of this application, by establishing a task description model, complex task requirements are disassembled to generate multiple basic sub-tasks, and the most suitable operation instructions are dynamically matched for each sub-task in the scalable "instruction sub-component library". It can cope with complex business processes through the flexible arrangement of basic instructions, ensure a high degree of consistency between each operation step and the task objective through algorithm matching, reduce execution deviation, and improve the execution efficiency of office tasks.
[0069] According to the second concept of this application, by constructing an agent to quantify the execution load of each sub-task, the load balance of the cluster computing ability is achieved, avoiding a single-point bottleneck. When the performance of some nodes fluctuates or fails, the task can be automatically migrated to healthy nodes to ensure the service continuity and stability of the overall system.
[0070] The above are only the preferred implementation manners of this application. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of this application, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as the protection scope of this application.< / actor>
Claims
1. A GUI intelligent agent system based on the MCP protocol, characterized in that, include: The front-end extension unit is used to collect office demand data and generate task demand packages based on the office demand data; Cloud-based central control unit, used to build intelligent agents; The intelligent agent is used to set the sequence of operation instructions according to the task requirement package; The MCP connection unit is used to transmit interactive data between the front-end expansion unit and the cloud central control unit; The front-end extension unit includes: Extension modules are used to build an interactive extension structure with the terminal browser; The extension module is also used to execute operation instruction sequences through interactive extension structures.
2. The GUI intelligent agent system based on the MCP protocol as described in claim 1, characterized in that, The front-end extension unit also includes: The requirement processing module is used to generate task text libraries and interference text libraries based on historical data. Generate a speech sub-model based on a library of interfering texts; Generate task sub-models based on the task text library; A semantic parsing model is established based on the speech sub-model and the task sub-model; The requirement processing module is used to generate task requirement packages based on office requirement data and semantic parsing models.
3. The GUI intelligent agent system based on the MCP protocol as described in claim 2, characterized in that, The front-end extension unit also includes: The uplink transmission module is used to acquire the execution feedback data of the operation instruction sequence and generate uplink data packets based on the execution results; The storage module is used to establish a storage library; The environment perception module is used to acquire execution feedback data of the sequence of operation instructions.
4. The GUI intelligent agent system based on the MCP protocol as described in claim 3, characterized in that, The demand processing module is also used for: Determine the category of office demand data; If the office requirement data is voice data, generate a first-level processing instruction; The speech sub-model converts speech data into text data according to the primary processing instructions; The task sub-model extracts task keywords and related content from text data; Generate a task requirement package based on the extraction results; If the office requirement data is text data, generate a secondary processing instruction; The task sub-model extracts task keywords and related content from the office demand data based on the secondary processing instructions; A task requirement package is generated based on the extraction results.
5. The GUI intelligent agent system based on the MCP protocol as described in claim 3, characterized in that, The cloud-based central control unit includes: The resource awareness module is used to configure multiple computing nodes; The resource awareness module is also used to acquire the running status data of each computing node; Build modules to create task description models; The building module is also used to generate multiple instruction sub-components based on historical data; Build an instruction sub-component library based on all instruction sub-components; The building module is also used to establish a scheduling sub-model based on all computing nodes; Based on the task description model, instruction component library, and scheduling sub-model, an intelligent agent is constructed. The downlink module is used to generate downlink data packets based on the sequence of operation instructions.
6. The GUI intelligent agent system based on the MCP protocol as described in claim 5, characterized in that, The intelligent agent is used to set a sequence of operation instructions according to the task requirement package, including: The task description model reads the task requirement package; Multiple sub-tasks are generated based on the read results; Select the target subtask sequentially from all the required subtasks; Establish an instruction sub-array A, A=(a1, a2, ..., ai, ..., an) based on the instruction sub-array library, where ai is the i-th instruction sub-array and n is the number of instruction sub-arrays; Generate the fit values between the target subtask and each instruction component; Set the operation instructions for the target subtask based on the instruction component corresponding to the maximum value among all matching values; Generate operation instructions for each required subtask in sequence; Generate a first-level operation sequence based on all required sub-tasks; Set the operation instruction sequence according to the primary operation sequence and all operation instructions.
7. The GUI intelligent agent system based on the MCP protocol as described in claim 6, characterized in that, The agent is also used for: Sequentially select the sub-tasks to be evaluated from all demand sub-tasks; Generate the execution load value of the sub-task to be evaluated; Establish a sequence of computing nodes B, B = (b1, b2…bi…bm), where bi is the i-th computing node; m is the number of computing nodes; Select the execution computing node of the sub-task to be evaluated from the sequence of computing nodes B according to the execution load value; Sequentially set the execution computing nodes of each demand sub-task.
8. The GUI intelligent agent system based on the MCP protocol as described in claim 7, characterized in that, The selection of the execution computing node of the sub-task to be evaluated includes: Sequentially select target computing nodes from the sequence of computing nodes B; Obtain the operation status data of the target computing node and generate the operation computing power value of the target computing node; Sequentially generate the operation computing power values of each computing node; Generate the matching value between the sub-task to be evaluated and each computing node according to all operation computing power values and the execution load value of the sub-task to be evaluated; Set the computing node corresponding to the maximum value among all matching values as the execution computing node of the sub-task to be evaluated.
9. The GUI intelligent agent system based on the MCP protocol as described in claim 8, characterized in that, The storage module is also used for: Establish a sequence of business subsystems W, W = (w1, w2…wi…wr), where wi is the i-th business subsystem; r is the number of business subsystems; Sequentially set wi as the target subsystem according to the sequence of business subsystems W; Generate the operation association value between the target subsystem and each instruction sub-component; Select multiple associated instruction sub-components of the target subsystem according to all operation association values; Generate the call path of each associated instruction sub-component; Establish a storage sub-library of the target operating system according to all call paths; Sequentially set the storage sub-libraries of each business subsystem; Construct a storage library according to all storage sub-libraries.
10. The GUI intelligent agent system based on the MCP protocol as described in claim 9, characterized in that, The selection of multiple associated instruction sub-components of the target subsystem includes: Preset an operation association value threshold C1; If C1 < ci, (i = 1, 2…n), set the i-th instruction sub-component as the associated instruction sub-component of the target subsystem; where ci is the operation association value between the i-th instruction sub-component and the target subsystem; n is the number of instruction sub-components.