Multi-agent collaborative information processing method and device, equipment and medium

Abstract

The invention provides a multi-agent collaborative information processing method and device, equipment and a medium, relates to the technical field of information and communication, and is suitable for the field of financial science and technology and the field of medical health. The method comprises the following steps: the terminal responds to acquired target operation information, and performs screenshot on an original interface to obtain an original interface screenshot; sending the target operation information to a cloud planning agent to obtain a target planning scheme; performing action decision on the target planning scheme and the original interface screenshot through an edge execution agent to obtain a target planning action sequence; executing the first target planning action on the original interface to obtain a first interface screenshot; and performing interface verification on the first interface screenshot through the edge observation agent, and then selecting to update the original interface or update the first interface. According to the method, the three agents are uniformly coordinated to form a continuous self-adaptive, self-error correction and self-optimization multi-agent collaborative architecture, so that the information processing accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of information and communication technology, and is applicable to the fields of financial technology and healthcare. In particular, it relates to a method, apparatus, device, and medium for information processing involving multiple agents. Background Technology

[0002] An intelligent agent is an autonomous entity capable of perceiving its environment, making decisions, and executing actions. The core functions of an intelligent agent include perception (through sensors or data input), decision-making (based on rules, models, or learning), and action (by performing operations or influencing the environment). For example, in the insurance scenario within the fintech field, if a user inputs the command "Open the insurance app, find settings, and enable a certain function" on their mobile phone, an intelligent agent can process this command to redirect the user to the desired interface. Similarly, in the digital healthcare service scenario within the healthcare field, if a user inputs the command "Open the smart healthcare platform, find the appointment booking function," an intelligent agent can process this command to redirect the user to the desired interface.

[0003] Related technologies typically utilize cloud-based intelligent agents centered around multimodal large language models for complex information reasoning. However, cloud-based intelligent agents have limited understanding of screen interfaces such as mobile phone screens, leading to performance degradation when processing command information related to screen interfaces. Although related technologies also exist that use terminal intelligent agents deployed on terminal devices for information reasoning, these terminal intelligent agents exhibit poor performance in reasoning about complex command information. Summary of the Invention

[0004] The main objective of this application is to propose a multi-agent collaborative information processing method, apparatus, device, and medium. By unifying and coordinating three agents, a multi-agent collaborative architecture that is continuously adaptive, self-correcting, and self-optimizing is formed, thereby improving the accuracy of information processing.

[0005] To achieve the above objectives, a first aspect of this application proposes a multi-agent collaborative information processing method applied to a terminal, wherein the terminal is deployed with an edge execution agent and an edge observation agent, the method comprising:

[0006] In response to obtaining the target operation information, a screenshot of the original interface is taken to obtain the original interface screenshot;

[0007] The target operation information is sent to the cloud-based planning agent; wherein, the target operation information is used by the cloud-based planning agent to perform global planning, obtain a target planning scheme, and send the target planning scheme to the terminal;

[0008] In response to receiving the target planning scheme, the edge execution agent makes action decisions on the target planning scheme and the original interface screenshot to obtain a target planning action sequence; wherein, the target planning action sequence includes a first target planning action and a second target planning action;

[0009] The first target planning action is executed on the original interface to obtain the first interface, and a screenshot of the first interface is taken to obtain the first interface screenshot;

[0010] The edge-observation agent performs interface verification on the first interface screenshot to obtain a first action feedback category; wherein, the first action feedback category includes a first interface verification passed category or a first interface verification failed category;

[0011] The original interface is updated based on the category of failure to pass the first interface verification, or the first interface is updated based on the category of success to pass the first interface verification and the second target planning action.

[0012] Optionally, updating the original interface based on the category of failed verification on the first interface includes:

[0013] In response to the first interface verification failing category, the edge observation agent performs semantic compression processing on the first interface screenshot to obtain the structured text of the first interface.

[0014] The edge observation agent sends the structured text of the first interface to the cloud planning agent; wherein, the cloud planning agent is used to perform global planning based on the structured text of the first interface and the target operation information to obtain an updated target planning scheme.

[0015] The edge execution agent updates the original interface according to the target planning scheme to obtain the target interface.

[0016] Optionally, the step of performing semantic compression processing on the first interface screenshot through the edge observation agent to obtain the structured text of the first interface includes:

[0017] Perform interface structure recognition on the first interface screenshot to obtain the first interface structure description text;

[0018] Perform text recognition on the first interface screenshot to obtain the text describing the content of the first interface;

[0019] By comparing and analyzing the original interface screenshot and the first interface screenshot, the text describing the state change of the first interface is obtained.

[0020] The structured text of the first interface is generated based on the first interface structure description text, the first interface content description text, and the first interface state change description text.

[0021] Optionally, sending the structured text of the first interface to the cloud-based planning agent via the edge observation agent includes:

[0022] The edge observation agent sends the structured text of the first interface to the memory module of the cloud planning agent; wherein, the memory module is used to store the target operation information;

[0023] The memory module, after storing the structured text of the first interface, triggers the reflection module of the cloud-based planning agent. The reflection module is used to perform global planning based on the structured text of the first interface and the target operation information to obtain an updated target planning scheme.

[0024] Optionally, updating the first interface based on the first interface verification category and the second target planning action includes:

[0025] In response to the interface verification passing category, the second target planning action is executed on the first interface to obtain the second interface, and a screenshot of the second interface is taken to obtain the second interface screenshot;

[0026] The edge-observation agent performs interface verification on the second interface screenshot to obtain a second action feedback category; wherein, the second action feedback category includes a second interface verification pass category or a second interface verification fail category;

[0027] The first interface is updated based on the category of failure to pass verification of the second interface, or the second interface is used as the target interface based on the category of success to pass verification of the second interface.

[0028] Optionally, the step of verifying the first interface screenshot through the edge observation agent to obtain the first action feedback category includes:

[0029] Receive reference interface status information sent by the cloud-based planning agent; wherein, the reference interface status information is the interface status information that the cloud-based planning agent predicts will appear after executing the target planning scheme;

[0030] Perform interface state analysis on the first interface screenshot to obtain the first interface state information;

[0031] The first action feedback category is obtained by comparing and analyzing the reference interface status information and the first interface status information.

[0032] Optionally, the step of using the edge execution agent to make action decisions on the target planning scheme and the original interface screenshot to obtain the target planning action sequence includes:

[0033] The target operation is obtained by extracting the operation from the target planning scheme through the edge execution agent;

[0034] The edge execution agent performs text encoding on the target planning scheme to obtain task text features;

[0035] The edge execution agent performs visual encoding on the original interface screenshot to obtain interface state features;

[0036] The edge execution agent performs image-text alignment on the task text features and interface state features to obtain the interface interaction area.

[0037] The target planned action is generated based on the target operation and the interface interaction area, and the target planned action sequence is obtained by merging the individual target planned actions.

[0038] To achieve the above objectives, a second aspect of this application proposes a multi-agent collaborative information processing device applied to a terminal, wherein the terminal is equipped with an edge execution agent and an edge observation agent, and the device includes:

[0039] The information determination module is used to take a screenshot of the original interface in response to obtaining the target operation information;

[0040] The scheme planning module is used to send the target operation information to the cloud planning agent according to the deep inference task; wherein, the target operation information is used by the cloud planning agent to perform global planning, obtain the target planning scheme, and send the target planning scheme to the terminal;

[0041] An action decision module is used to respond to receiving the target planning scheme by performing action decisions on the target planning scheme and the original interface screenshot through the edge execution agent to obtain a target planning action sequence; wherein, the target planning action sequence includes a first target planning action and a second target planning action;

[0042] The first interface update module is used to execute the first target planning action on the original interface to obtain the first interface, and to take a screenshot of the first interface to obtain a screenshot of the first interface.

[0043] The action feedback module is used to perform interface verification on the first interface screenshot through the edge observation agent to obtain a first action feedback category; wherein, the first action feedback category includes a first interface verification passed category or a first interface verification failed category.

[0044] The second interface update module is used to update the original interface according to the category of the first interface verification failure, or to update the first interface according to the category of the first interface verification success and the second target planning action.

[0045] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the multi-agent collaborative information processing method described in the first aspect.

[0046] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-agent cooperative information processing method described in the first aspect.

[0047] The multi-agent collaborative information processing method, device, electronic device, and computer-readable storage medium proposed in this application operate on a multi-agent collaborative architecture comprising a cloud-based planning agent, an edge-based execution agent, and an edge-based observation agent. The terminal sends the acquired target operation information to the cloud-based planning agent. Upon receiving the target operation information, the cloud-based planning agent plans a target operation scheme based on the target operation information. Next, the edge-based execution agent makes action decisions based on the target operation scheme and an original interface screenshot to obtain a target-planned action sequence and executes the actions in the target-planned action sequence. Then, the observation edge agent verifies the interface to obtain a first action feedback category. The original interface or the first interface is updated based on the first action feedback category. Thus, this application, by unifying and coordinating the three agents, forms a continuously adaptive, self-correcting, and self-optimizing multi-agent collaborative architecture, improving the accuracy of information processing. Attached Figure Description

[0048] Figure 1 This is a flowchart of the multi-agent collaborative information processing method provided in the embodiments of this application;

[0049] Figure 2 yes Figure 1 The flowchart for step 103 in the document;

[0050] Figure 3 yes Figure 1The flowchart for step 105 in the document;

[0051] Figure 4 yes Figure 1 The flowchart for step 106 in the document;

[0052] Figure 5 yes Figure 4 The flowchart for step 401 in the document;

[0053] Figure 6 yes Figure 1 Another flowchart of step 106 in the process;

[0054] Figure 7 This is a block diagram of the module structure of the multi-agent collaborative information processing device provided in the embodiments of this application;

[0055] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] AI Agent: An application capable of autonomously planning, invoking external tools for execution, and reflecting on the results based on task objectives and external information. AI agents are designed around task objectives and business processes in terms of implementation, operational form, and behavioral characteristics. AI agents include cloud-based planning agents, edge-based execution agents, and edge-based observation agents.

[0061] Cloud-based planning agents are intelligent agents deployed in the cloud for planning solutions. They often incorporate multimodal, large language models.

[0062] Edge execution agents are agents deployed on the edge and used for action decision-making and action execution. Edge execution agents are often equipped with small language models.

[0063] Edge observation agent: An agent that is deployed on the terminal and used to provide action feedback.

[0064] Multimodal Large Language Model (MLLM): This is a type of artificial intelligence model that can understand and process multiple different types of data.

[0065] Miniature or Smaller Language Models (MSLM) are language models that are smaller in scale and have fewer parameters compared to large pre-trained language models (such as GPT-4 and BERT).

[0066] The current development of mobile intelligent agent systems faces a significant challenge: an imbalance in capabilities between cloud-based and edge-execution agents. Cloud-based agents, centered on multimodal large language models, possess powerful task understanding, long-range planning, and complex logical reasoning capabilities, enabling them to translate high-level instructions into multi-step executable operation sequences. However, these models are bulky, reliant on high-performance servers and stable networks, and their application on mobile devices faces bottlenecks such as high latency, high energy consumption, and high communication costs. Simultaneously, cloud-based planning agents have weak direct understanding of the terminal screen interface. Since training data is mostly general images, they struggle to accurately identify minute, dense, and dynamically changing elements within the terminal screen interface (also known as the user interface). Therefore, when executing specific operations, they often rely on auxiliary detection modules, such as optical character recognition or object recognition networks, leading to complex processing flows, delayed responses, and increased error rates, making it difficult to meet the demands of real-time mobile interaction.

[0067] Terminal agents deployed on mobile devices represent the other extreme. These agents, finely tuned to adapt to mobile interaction environments, can directly read screen content and perform touch-based operations, offering significant advantages in low latency and low power consumption. However, these models have limited reasoning capabilities, struggle to understand the overall task objective in complex scenarios, and lack the ability to structurally decompose multi-step operations. If deviations occur during execution, the model often cannot self-correct, easily falling into repetitive errors or complete failure. Furthermore, current edge-cloud collaborative architectures mostly employ a one-way command delivery from the cloud to the edge, with passive execution. The lack of effective result feedback and status reporting mechanisms prevents the cloud from knowing the actual execution status and dynamically adjusting strategies, resulting in low system efficiency, insufficient success rate, and difficulty in supporting the stable execution of complex mobile tasks.

[0068] Based on this, the embodiments of this application propose a multi-agent collaborative information processing method, a multi-agent collaborative information processing device, an electronic device, and a computer-readable storage medium. By unifying and coordinating three agents, a multi-agent collaborative architecture that is continuously adaptive, self-correcting, and self-optimizing is formed, thereby improving the accuracy of information processing.

[0069] The multi-agent collaborative information processing method provided in this application can be applied to terminals and the cloud, or it can be software running in the cloud. The cloud can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or it can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the multi-agent collaborative information processing method, etc., but is not limited to the above forms.

[0070] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0071] This application provides a multi-agent collaborative information processing method, a multi-agent collaborative information processing device, an electronic device, and a computer-readable storage medium. The specific embodiments are described below. First, the multi-agent collaborative information processing method in this application is described.

[0072] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to user identity or characteristics, such as user interface data, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0073] Reference Figure 1 , Figure 1 This is an optional flowchart of a multi-agent collaborative information processing method provided in the embodiments of this application. The method is applied to a terminal, which is equipped with an edge execution agent and an edge observation agent. The method may include, but is not limited to, steps 101 to 106.

[0074] Step 101: In response to obtaining the target operation information, take a screenshot of the original interface to obtain the original interface screenshot;

[0075] Step 102: Send the target operation information to the cloud planning agent; wherein, the target operation information is used by the cloud planning agent to perform global planning, obtain the target planning scheme, and send the target planning scheme to the terminal;

[0076] Step 103: In response to receiving the target planning scheme, the edge execution agent makes action decisions on the target planning scheme and the original interface screenshot to obtain the target planning action sequence; wherein, the target planning action sequence includes the first target planning action and the second target planning action;

[0077] Step 104: Execute the first target planning action on the original interface to obtain the first interface, and take a screenshot of the first interface to obtain the first interface screenshot.

[0078] Step 105: The edge observation agent performs interface verification on the first interface screenshot to obtain the first action feedback category; wherein, the first action feedback category includes the first interface verification passed category or the first interface verification failed category.

[0079] Step 106: Update the original interface according to the category of failed verification of the first interface, or update the first interface according to the category of passed verification of the first interface and the action planned by the second goal.

[0080] Steps 101 to 106, as illustrated in this embodiment, operate within a multi-agent collaborative architecture comprising a cloud-based planning agent, an edge execution agent, and an edge observation agent. The terminal sends the acquired target operation information to the cloud-based planning agent. Upon receiving the target operation information, the cloud-based planning agent plans a target operation scheme based on this information. Next, the edge execution agent makes action decisions based on the target operation scheme and an original interface screenshot to obtain a target-planned action sequence, and then executes the actions within that sequence. Finally, the observation edge agent verifies the interface to obtain a first action feedback category. The original interface is then updated, or a first interface is updated, based on the first action feedback category. Thus, by coordinating the three agents in a unified manner, this application forms a continuously adaptive, self-correcting, and self-optimizing multi-agent collaborative architecture, improving the accuracy of information processing.

[0081] In step 101 of some embodiments, in response to obtaining the target operation information, the terminal takes a screenshot of the original interface to obtain a screenshot of the original interface. The target operation information refers to the operation or request issued by the user to the terminal (also called the terminal device) through the interface, voice, text, etc., aimed at instructing the system or application to perform a specific task or provide a certain service. The target operation information can be used to represent the instruction or command information that the target user expects to jump from the original interface to the target interface.

[0082] The native interface refers to the screen interface on a terminal device (such as a mobile phone, computer, etc.). The native interface has multiple elements, including: 1. Status bar: Displays basic information such as time, signal strength, Wi-Fi status, battery level, and notification icons. 2. Navigation bar: Includes back button, home button, and multitasking / switching buttons. 3. Title bar / toolbar: Displays the application's title and often includes a back button, menu, search button, and action buttons. 4. Buttons: Provide interactive elements for users to click, such as "Submit," "Cancel," and "Confirm." 5. Text boxes: Areas for users to input information, such as search boxes and login forms. 6. Labels: Display text information for description or prompts. 7. Icons: Visual symbols representing operations, content categories, or states. 8. Pop-ups: Used for confirmation, prompts, or input. 9. Gesture or swipe areas: Support user operations via swipes or gestures. 10. Floating action buttons: Often used to highlight key operations, such as the "Add" button.

[0083] You can use screenshot software on your terminal device to take a screenshot of the original interface and obtain a screenshot of the original interface.

[0084] In the insurance scenario within the fintech field, users input the target action information on their mobile phone interface, such as "open the insurance app, find the settings, and enable a certain function." This information is then processed by an intelligent agent to redirect the user's mobile phone interface to the desired insurance function.

[0085] In the context of digital healthcare services, users input the target operation information "open the smart healthcare platform and find the appointment registration function" into the terminal interface. This information is then processed by an intelligent agent to redirect the terminal interface to the user's desired registration interface.

[0086] In step 102 of some embodiments, the terminal sends the target operation information to the cloud-based planning agent. The target operation information is used by the cloud-based planning agent to perform global planning, obtain a target planning scheme, and then send the target planning scheme to the terminal. In conjunction with the above, the cloud-based planning agent uses a multimodal large language model to perform global planning on the target operation information to obtain the target planning scheme.

[0087] In one embodiment, during the global planning process, the cloud-based planning agent first performs semantic analysis on the target operation information to obtain the target task, which includes a deep reasoning task. The cloud-based planning agent then performs global planning on the target operation information based on the deep reasoning task. The deep reasoning task indicates that the target operation information needs to be deeply reasoned.

[0088] In another embodiment, the target task may further include a non-deep reasoning task. The cloud-based planning agent extracts the target operation from the target operation information based on the non-deep reasoning task and generates a target planning scheme based on the target operation. The non-deep reasoning task indicates that the target operation information does not need to be subjected to deep reasoning.

[0089] In one embodiment, the information processing method can also be applied to the cloud, where a cloud-based planning agent and a central scheduler are deployed. The central scheduler is deployed on a control platform or cloud console. The central scheduler is responsible for receiving task requests from upper-layer services and, as a distributor, coordinating task allocation between the cloud-based planning agent and the edge execution agent. In this embodiment, the central scheduler is embedded in the task initialization module of the cloud-based planning agent and possesses basic task parsing and classification capabilities. Its function is to determine the type of model to be invoked and the location of its computing resources based on the task content. After receiving the target operation information, the central scheduler pre-parses the target operation information, classifying the task intent based on keywords, intent structure, and context state. If the target operation information contains multiple sub-objectives, requires cross-screen path planning, or involves ambiguous language (such as "after opening the insurance app, find settings and enable a certain function"), it is identified as a deep reasoning task and transferred to the multimodal large language model of the cloud-based planning agent for global planning. Conversely, if the target operation information has explicit operation instructions (such as "click the button" or "enter a mobile phone number") or belongs to a specific sub-task step planned in the cloud, it is directly dispatched by the central scheduler to the lightweight small language model of the edge execution agent for execution. The judgment criteria mainly include a combination of factors such as the complexity of the task language, whether the interface state is stable, whether reasoning jump is required, and the failure rate of historical tasks. Dynamic assignment is achieved through the assistance of a lightweight rule engine or prior model.

[0090] In one embodiment, the information processing method executed in the cloud may further include: sending target operation information to a terminal based on a non-deep reasoning task, and having the terminal use a small language model of an edge execution agent to plan the target operation information to obtain a target planning scheme.

[0091] In step 103 of some embodiments, an edge execution agent makes action decisions based on the target planning scheme and the original interface screenshot to obtain a target planning action sequence. The target planning action sequence includes multiple target planning actions executed in a sequential order. In one example, the target planning action sequence includes a first target planning action and a second target planning action. In another example, the target planning action sequence may also include a first target planning action, a second target planning action, a third target planning action, and a fourth target planning action, etc. The edge execution agent can use a small language model to make action decisions based on the target planning scheme and the original interface screenshot to obtain the target planning action sequence.

[0092] In one embodiment, after receiving a target planning scheme (which may include multiple operations) from the cloud, the edge execution agent combines the current interface screenshot with its internally fine-tuned small language model to make local action decisions. Specifically, the model inputs the operation information (such as "click the 'Confirm' button") and the current interface screenshot, aligns the corresponding interaction area (such as the specific coordinates of the button) through the alignment of visual features and task text, and then generates specific low-level action instructions, such as Tap(x,y) and Swipe(x1,y1,x2,y2), and executes them immediately on the local device. This process is completed entirely at the edge, without uploading images to the cloud for parsing, effectively avoiding communication bottlenecks and latency.

[0093] In one embodiment, reference is made to Figure 2 Step 103 may include:

[0094] Step 201: Extract the target operation from the target planning scheme through the edge execution agent;

[0095] Step 202: The target planning scheme is text-encoded by the edge execution agent to obtain task text features;

[0096] Step 203: Visually encode the original interface screenshot using an edge execution agent to obtain interface state features;

[0097] Step 204: Align the task text features and interface state features with the edge execution agent to obtain the interface interaction area.

[0098] Step 205: Generate target planned actions based on the target operation and the interface interaction area, and merge the target planned actions to obtain the target planned action sequence.

[0099] Specifically, edge execution agents can use small language models to generate target planning action sequences. These small language models are obtained by fine-tuning general-purpose small-scale multimodal language models (such as Mini-InternVL) for mobile device GUI tasks. First, a high-quality training corpus is built, containing tens of thousands of task samples collected from real mobile devices. Each sample consists of a screenshot, operation intent, actual action, and post-operation interface state, covering high-frequency interaction patterns such as button clicks, text input, swipe navigation, and app switching. Image data, after feature extraction by a visual encoder, is fed into the model along with the task text input for alignment training, forming a perception-decision capability that links images and text. The fine-tuning process employs a two-stage strategy: the first stage is supervised training, using manually labeled data to perform end-to-end operation prediction training, enabling the model to infer the correct operation intent based on interface semantics; the second stage introduces interface fine-tuning and human feedback (such as RLHF) mechanisms, running the model on a simulator and collecting its operation performance, feeding back successful and failed examples into the training to strengthen the model's ability to identify and correct key interaction states. The resulting small language model can not only understand the current screen content, but also accurately generate the next operation instructions based on the task steps. It has strong generalization ability and lightweight deployment advantages, making it suitable for running on mobile terminals or edge devices.

[0100] The advantage of the embodiments of steps 201 to 205 described above is that they can improve the accuracy of action decisions and are conducive to improving the accuracy of information processing.

[0101] In step 104 of some embodiments, a first target planning action is performed on the original interface to obtain a first interface, and a screenshot of the first interface is taken to obtain a screenshot of the first interface. In one example, the first target planning action includes a first target operation and a first interface interaction area; the process of performing the first target planning action on the original interface to obtain the first interface may include: in response to the first target operation being a text deletion operation, selecting text in the text input box of the first interface interaction area to obtain candidate text to be deleted; performing the first target operation on the candidate text to be deleted to obtain the first interface. In this way, support for deletion operations can be provided, which, compared with related technologies that can only support click, swipe, input, and other operations, enhances the flexibility of operation and thus improves the applicability of information processing.

[0102] In one example, after receiving the "delete text" instruction through a fine-tuned small language model, the edge execution agent, based on the current screen state, identifies the location of the currently focused input box and generates an equivalent clearing action sequence, such as simulating a long press of the delete key, selecting and clearing text, and other low-level operations. The model already includes numerous instances of text input and modification during training, thus possessing the ability to recognize the complete operation flow of "input error—clear—re-enter." Therefore, when the cloud-based planning agent specifies the need to delete input content in the task steps, the edge agent can accurately identify the target input box and perform the clearing locally, without relying on complex interface structure analysis or secondary OCR judgment, resulting in higher execution efficiency and stability. The fundamental reason why existing models generally lack a "clearing mechanism" lies in the limitations of their operational space design. Existing model training schemes typically only include atomic actions such as "inputting text" or "clicking," ignoring common modification behaviors in text input tasks. This leads to situations where, once an input error occurs, the model cannot roll back or replace the content, resulting in repeated attempts or even task failure. This application explicitly introduces "deleting text" as an independent action in the operational space and adds relevant supervised samples during model training, enabling the model to learn and execute deletion operations. Simultaneously, the cloud-based intelligent agent supports explicitly listing "clearing existing content" as a subtask during the task planning phase and transmitting it to the edge, forming a clear instruction chain. This provides support for the "clearing mechanism" from both design and execution levels, overcoming the structural deficiencies in fault tolerance inherent in existing technologies.

[0103] In step 105 of some embodiments, the edge-observation agent performs interface verification on the first interface screenshot to obtain a first action feedback category. The first action feedback category includes a first interface verification passed category or a first interface verification failed category.

[0104] In one embodiment, reference is made to Figure 3 Step 105 may include:

[0105] Step 301: Receive reference interface status information sent by the cloud-based planning agent;

[0106] Step 302: Analyze the interface state of the first interface screenshot to obtain the first interface state information;

[0107] Step 303: Compare and analyze the reference interface status information and the first interface status information to obtain the first action feedback category.

[0108] In step 301, the reference interface state information is the interface state information predicted by the cloud-based planning agent after the execution of the target planning scheme. The cloud-based planning agent uses a multimodal large language model to predict the interface state of the target planning scheme to obtain the reference interface state information. If the target planning scheme includes multiple operations, the reference interface state information includes the reference sub-interface state information corresponding to each operation.

[0109] In step 302, the edge execution agent can use a small language model to perform interface state analysis on the first interface screenshot to obtain the first interface state information.

[0110] It should be noted that UI state information refers to a collection of dynamic information and states reflecting the current application, system, or page within the user interface. UI state information helps users understand the application's operational status, content changes, and the current state of the interface.

[0111] In step 303, theoretically, the reference interface state information should be basically the same as the first interface state information. Therefore, the edge execution agent can use a small language model to compare and analyze the reference interface state information and the first interface state information to obtain the comparison and analysis results. If the comparison and analysis results indicate that the two are consistent, the first action feedback category is the first interface verification passed category. If the comparison and analysis results indicate that the two are inconsistent, the first action feedback category is the first interface verification failed category.

[0112] It's important to note that the edge execution agent itself does not directly receive the reference interface state information generated by the cloud. This reference interface state information primarily serves the edge observation agent and the cloud's feedback and reflection mechanisms. After executing an action, the edge observation agent compares the current screen state (first interface state information) with the reference interface state information to determine if the operation was successful and feeds the result back to the cloud. If the operation fails, the cloud will use its memory and reflection modules to trace the cause of the error based on the difference between the expected and observed results, and activate the replanning process. Therefore, the reference interface state information, as a reference standard for verification and correction, provides crucial support for the entire system to form a closed loop of perception-verification-feedback-correction.

[0113] The advantage of the embodiments of steps 301 to 303 described above is that by performing self-correction and timely feedback through the edge observation agent, the accuracy of information processing is improved. Furthermore, the edge observation agent is independent of the edge execution agent and the cloud planning agent, reducing the adverse effects of self-correction on action decision-making and scheme planning.

[0114] In step 106 of some embodiments, the original interface is updated according to the category of failing the first interface verification, or the first interface is updated according to the category of passing the first interface verification and the second target planning action.

[0115] In one embodiment, reference is made to Figure 4 Step 106, which updates the original interface based on the category of failed verification on the first interface, may include:

[0116] Step 401: In response to the first interface verification failure category, the first interface screenshot is semantically compressed by the edge observation agent to obtain the structured text of the first interface.

[0117] Step 402: The structured text of the first interface is sent to the cloud planning agent through the edge observation agent; wherein, the cloud planning agent is used to perform global planning based on the structured text of the first interface and the target operation information to obtain the updated target planning scheme.

[0118] Step 403: The edge execution agent updates the original interface according to the target planning scheme to obtain the target interface.

[0119] In step 401, the edge observation agent can use a pre-understanding model to perform semantic compression processing on the first interface screenshot to obtain structured text for the first interface. The pre-understanding model is essentially a lightweight visual language model deployed on an edge device, a component of the edge observation agent, used to compress the screenshot into a concise text description. The pre-understanding model can be a Qwen2-VL-2B model, specifically designed to reduce image transmission costs while preserving key semantic information.

[0120] In one example, a multimodal small language model (such as Qwen2-VL-2B) that supports text and image input and output can be deployed locally as a pre-understanding model. A screenshot of the mobile phone screen is prepared as input, and reasoning is performed in combination with specific prompts (such as "Please describe the interactive elements visible on the screen"). Finally, the structured text output by the model is used for task verification and memory update.

[0121] In one embodiment, reference is made to Figure 5 Step 401 may include:

[0122] Step 501: Perform interface structure recognition on the first interface screenshot to obtain the first interface structure description text;

[0123] Step 502: Perform text recognition on the screenshot of the first interface to obtain the text describing the content of the first interface;

[0124] Step 503: Compare and analyze the original screenshot and the first screenshot to obtain the text describing the state changes of the first screen.

[0125] Step 504: Generate the structured text of the first interface based on the first interface structure description text, the first interface content description text, and the first interface state change description text.

[0126] Specifically, semantic compression is performed on the first screenshot to extract core information such as the user interface (UI) structure, text content, and state changes, and then converts it into structured text to be fed back to the cloud. During the compression process, the number of processing tokens required for the first screenshot is reduced from thousands to less than hundreds, significantly reducing bandwidth usage and computational overhead, enabling more frequent and real-time data interaction between the edge and the cloud.

[0127] The advantage of the embodiments of steps 501 to 504 described above is that they can improve the communication efficiency between the edge observation agent and the cloud planning agent.

[0128] In one embodiment, step 402 may include: sending the structured text of the first interface to the memory module of the cloud planning agent through the edge observation agent; wherein, the memory module is used to store target operation information, and after storing the structured text of the first interface, the memory module triggers the reflection module of the cloud planning agent, and the reflection module is used to perform global planning based on the structured text of the first interface and the target operation information to obtain an updated target planning scheme.

[0129] Specifically, this embodiment implements the technology by introducing a "memory module" and a "reflective module" working collaboratively within the cloud-based planning agent. When the edge observation agent determines that the current execution step has not achieved the expected result, it immediately calls the "pre-understanding model" to compress the screen state at the time of failure into a structured text representation, and writes it along with the historical context corresponding to that step into the cloud-based memory module. The memory module stores the semantic state changes after each step in a time-series manner, forming a complete execution trajectory. Subsequently, the reflection module is automatically activated. It takes the initial instructions of the current task, the operation steps before the failure point, and the semantic trajectory in memory as input, and uses a multimodal large language model to perform error reasoning analysis to determine whether the failure was caused by interface changes, process interruption, or operational deviation. Based on this, the model selectively adjusts the task path after the failed step and regenerates a new target planning scheme and expected result (referencing interface state information). This process does not rely on human intervention, has self-awareness and self-correction capabilities, and ensures that the task can continue to advance under complex or dynamic interfaces, demonstrating the reflective adaptive capability under multi-agent collaboration.

[0130] In step 403, the specific implementation of the embodiments of steps 103 to 106 is carried out, which will not be repeated here.

[0131] In one embodiment, reference is made to Figure 6 Step 106, which involves updating the first interface based on the first interface verification, category verification, and second goal planning action, may include:

[0132] Step 601: In response to the interface verification passing category, execute the second target planning action on the first interface to obtain the second interface, and take a screenshot of the second interface to obtain the second interface screenshot.

[0133] Step 602: The edge observation agent performs interface verification on the second interface screenshot to obtain the second action feedback category; wherein, the second action feedback category includes the second interface verification passed category or the second interface verification failed category;

[0134] Step 603: Update the first interface according to the category of the second interface that failed verification, or use the second interface as the target interface according to the category of the second interface that passed verification.

[0135] The specific implementation processes of steps 601 to 603 and steps 104 to 106 are basically the same, and will not be described in detail here.

[0136] The advantage of the embodiments of steps 601 to 603 described above is that even if the second action feedback category is the second interface verification failure category, the original interface does not need to be updated again, but can continue to be updated from the first interface. This can reduce the workload of replanning by the cloud planning agent, which is conducive to saving processing resources and thus improving information processing efficiency.

[0137] Understandably, when a deviation occurs in task execution, the edge observation agent proactively writes simplified semantic results into the cloud-based memory module, triggering the planning agent's reflection mechanism. This, combined with historical trajectories, enables fault analysis and strategy reconstruction, forming an adaptive closed-loop system of "perception—memory—feedback—replanning." Through this structure, the embodiments of this application not only retain the global reasoning advantages of the cloud model but also fully leverage the execution efficiency of the edge model, enabling complex mobile tasks to achieve higher success rates and system stability while maintaining response speed.

[0138] In one example, the information processing method provided in this application may include the following steps:

[0139] (1) The cloud-based planning agent interfaces with a unified scheduling platform or business system to receive target operation information triggered by the user or automated processes. This target operation information is sent to the cloud in a structured format via an interface, typically compared to the original interface uploaded by the terminal device. Figure 1 This information forms the input. Upon receiving instructions and initial environmental information, the cloud-based planning agent immediately enters the task parsing and planning process to prepare for generating a specific operation plan.

[0140] (2) The target operation information itself is multimodal, including natural language descriptions, such as "complete real-name authentication in the bank app", as well as original screenshot images (reflecting the current screen state), optional OCR-recognized text (extracting elements such as buttons, labels, and input boxes), and sometimes also the result of the previous operation or device context information. Through the fusion of these textual and visual information, the cloud-based planning agent can accurately determine the user's intent and the current interface state.

[0141] (3) To enhance the understanding of the interface, the multimodal large language model used by the cloud planning agent was fine-tuned during training by incorporating a large amount of real GUI task data. The model not only accepts labeled images from real App interfaces, but also introduces paired data between operation behavior trajectories, task intentions and interface changes, which significantly improves its ability to identify and process the interface structure, task logic and abnormal states of mobile devices, and makes up for the weakness of general large models in UI understanding.

[0142] (4) After the task is broken down, the cloud-based AI agent encapsulates each operation and its corresponding expected result into structured information and sends it to the edge execution AI agent. The structured information includes the target planning scheme and the key changes that the interface should show after the operation is completed. The target planning scheme specifically includes the operation type (such as clicking a coordinate, entering text, etc.), the identification method of the target element (such as button text or position), and the operation parameters (such as the content to be entered). The edge AI agent executes the task accordingly and compares the execution result with the edge observation AI agent to see if it meets the expectations, thus achieving precise and closed-loop mobile automation.

[0143] In summary, this application can achieve at least the following beneficial effects: (1) It constructs an intelligent architecture that integrates task decomposition, local execution, result feedback, semantic compression, and dynamic replanning. By deploying observation agents on the edge side and introducing a "pre-understanding model", the execution results at the image level can be transformed into low-cost text semantic information, realizing real-time monitoring and efficient transmission of the execution status. This mechanism not only effectively solves the bandwidth and token consumption problem in the process of visual data transmission, but also provides reliable context clues for the cloud, supporting it to conduct historical reflection and task correction, opening up the perception channel between execution and reasoning, and forming a new type of "reflective agent loop". (2) In terms of model collaboration strategy, this application proposes a task-granular dynamic scheduling architecture, which entrusts complex cognitive planning tasks to cloud planning agents with strong reasoning capabilities, while sinking high-frequency, low-latency specific operation tasks to edge execution agents that can be deployed on edge devices, thereby taking into account both computing resource utilization and execution efficiency. This functional division of labor among heterogeneous models has not been realized or is insufficiently applied in traditional solutions. In particular, the introduction of text deletion action expands the operational semantic boundary of the existing mobile agent system, enabling it to modify content and undo context for the first time, thus improving its flexibility and robustness in practical applications. (3) By unifying and coordinating the three agents, a multi-agent automated system that is continuously adaptive, self-correcting, and self-optimizing is formed. Even with limited deployment conditions on mobile devices, it can still complete highly complex tasks with low resource consumption, demonstrating strong practical value and promising prospects for promotion.

[0144] Please see Figure 7 This application also provides a multi-agent collaborative information processing apparatus, which can implement the above-described multi-agent collaborative information processing method. Figure 7 This is a block diagram of the module structure of a multi-agent collaborative information processing device provided in an embodiment of this application. The device includes:

[0145] The information determination module 701 is used to take a screenshot of the original interface in response to obtaining the target operation information;

[0146] The scheme planning module 702 is used to send the target operation information to the cloud planning agent; wherein, the target operation information is used by the cloud planning agent to perform global planning, obtain the target planning scheme, and send the target planning scheme to the terminal;

[0147] Action decision module 703 is used to respond to receiving a target planning scheme, and to make action decisions on the target planning scheme and the original interface screenshot through an edge execution agent to obtain a target planning action sequence; wherein, the target planning action sequence includes a first target planning action and a second target planning action;

[0148] The first interface update module 704 is used to perform the first target planning action on the original interface to obtain the first interface, and to take a screenshot of the first interface to obtain a screenshot of the first interface.

[0149] The action feedback module 705 is used to perform interface verification on the first interface screenshot through the edge observation agent to obtain a first action feedback category; wherein, the first action feedback category includes a first interface verification passed category or a first interface verification failed category.

[0150] The second interface update module 706 is used to update the original interface according to the first interface verification failure category, or to update the first interface according to the first interface verification success category and the second target planning action.

[0151] It should be noted that the specific implementation of this multi-agent collaborative information processing device is basically the same as the specific implementation of the multi-agent collaborative information processing method described above, and will not be repeated here.

[0152] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned multi-agent collaborative information processing method. This electronic device can be any smart terminal, including mobile phones, tablets, and in-vehicle computers.

[0153] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0154] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0155] The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the multi-agent cooperative information processing method of the embodiments of this application.

[0156] The 803 input / output interface is used to implement information input and output.

[0157] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0158] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0159] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0160] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to realize the above-described multi-agent collaborative information processing method.

[0161] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0163] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0166] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multi-agent collaborative information processing method, characterized in that, Applied to a terminal, wherein the terminal is equipped with an edge execution agent and an edge observation agent, the method includes: In response to obtaining the target operation information, a screenshot of the original interface is taken to obtain the original interface screenshot; The target operation information is sent to the cloud-based planning agent; wherein, the target operation information is used by the cloud-based planning agent to perform global planning, obtain a target planning scheme, and send the target planning scheme to the terminal; In response to receiving the target planning scheme, the edge execution agent makes action decisions on the target planning scheme and the original interface screenshot to obtain a target planning action sequence; wherein, the target planning action sequence includes a first target planning action and a second target planning action; The first target planning action is executed on the original interface to obtain the first interface, and a screenshot of the first interface is taken to obtain the first interface screenshot; The edge-observation agent performs interface verification on the first interface screenshot to obtain a first action feedback category; wherein, the first action feedback category includes a first interface verification passed category or a first interface verification failed category; The original interface is updated based on the category of failure to pass the first interface verification, or the first interface is updated based on the category of success to pass the first interface verification and the second target planning action.

2. The method according to claim 1, characterized in that, The step of updating the original interface based on the category of failed verification on the first interface includes: In response to the first interface verification failing category, the edge observation agent performs semantic compression processing on the first interface screenshot to obtain the structured text of the first interface. The edge observation agent sends the structured text of the first interface to the cloud planning agent; wherein, the cloud planning agent is used to perform global planning based on the structured text of the first interface and the target operation information to obtain an updated target planning scheme. The edge execution agent updates the original interface according to the target planning scheme to obtain the target interface.

3. The method according to claim 2, characterized in that, The semantic compression processing of the first interface screenshot by the edge observation agent to obtain the structured text of the first interface includes: Perform interface structure recognition on the first interface screenshot to obtain the first interface structure description text; Perform text recognition on the first interface screenshot to obtain the text describing the content of the first interface; By comparing and analyzing the original interface screenshot and the first interface screenshot, the text describing the state change of the first interface is obtained. The structured text of the first interface is generated based on the first interface structure description text, the first interface content description text, and the first interface state change description text.

4. The method according to claim 2, characterized in that, The step of sending the structured text of the first interface to the cloud planning agent through the edge observation agent includes: The edge observation agent sends the structured text of the first interface to the memory module of the cloud planning agent; wherein, the memory module is used to store the target operation information; The memory module, after storing the structured text of the first interface, triggers the reflection module of the cloud-based planning agent. The reflection module is used to perform global planning based on the structured text of the first interface and the target operation information to obtain an updated target planning scheme.

5. The method according to any one of claims 1 to 4, characterized in that, The step of updating the first interface based on the verification category of the first interface and the second target planning action includes: In response to the interface verification passing category, the second target planning action is executed on the first interface to obtain the second interface, and a screenshot of the second interface is taken to obtain the second interface screenshot; The edge-observation agent performs interface verification on the second interface screenshot to obtain a second action feedback category; wherein, the second action feedback category includes a second interface verification pass category or a second interface verification fail category; The first interface is updated based on the category of failure to pass verification of the second interface, or the second interface is used as the target interface based on the category of success to pass verification of the second interface.

6. The method according to any one of claims 1 to 4, characterized in that, The step of verifying the first interface screenshot through the edge observation agent to obtain the first action feedback category includes: Receive reference interface status information sent by the cloud-based planning agent; wherein, the reference interface status information is the interface status information that the cloud-based planning agent predicts will appear after executing the target planning scheme; Perform interface state analysis on the first interface screenshot to obtain the first interface state information; The first action feedback category is obtained by comparing and analyzing the reference interface status information and the first interface status information.

7. The method according to any one of claims 1 to 4, characterized in that, The step of performing action decisions on the target planning scheme and the original interface screenshot through the edge execution agent to obtain the target planning action sequence includes: The target operation is obtained by extracting the operation from the target planning scheme through the edge execution agent; The edge execution agent performs text encoding on the target planning scheme to obtain task text features; The edge execution agent performs visual encoding on the original interface screenshot to obtain interface state features; The edge execution agent performs image-text alignment on the task text features and interface state features to obtain the interface interaction area. The target planned action is generated based on the target operation and the interface interaction area, and the target planned action sequence is obtained by merging the individual target planned actions.

8. A multi-agent collaborative information processing device, characterized in that, Applied to a terminal, the terminal being deployed with an edge execution agent and an edge observation agent, the device includes: The information determination module is used to take a screenshot of the original interface in response to obtaining the target operation information; The scheme planning module is used to send the target operation information to the cloud planning agent; wherein, the target operation information is used by the cloud planning agent to perform global planning, obtain the target planning scheme, and send the target planning scheme to the terminal; An action decision module is used to respond to receiving the target planning scheme by performing action decisions on the target planning scheme and the original interface screenshot through the edge execution agent to obtain a target planning action sequence; wherein, the target planning action sequence includes a first target planning action and a second target planning action; The first interface update module is used to execute the first target planning action on the original interface to obtain the first interface, and to take a screenshot of the first interface to obtain a screenshot of the first interface. The action feedback module is used to perform interface verification on the first interface screenshot through the edge observation agent to obtain a first action feedback category; wherein, the first action feedback category includes a first interface verification passed category or a first interface verification failed category. The second interface update module is used to update the original interface according to the category of the first interface verification failure, or to update the first interface according to the category of the first interface verification success and the second target planning action.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the multi-agent cooperative information processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-agent cooperative information processing method according to any one of claims 1 to 7.

Images