Multi-agent cooperation interaction method and interaction system

Abstract

The invention provides a multi-agent cooperation interaction method and system, and the method comprises the steps: firstly, enabling a decision agent to obtain an execution environment and a subtask set corresponding to a to-be-executed instruction in response to the received to-be-executed instruction; the decision-making agent generates a target dynamic interaction graph corresponding to a to-be-executed instruction based on a preset interaction graph and the sub-task set, and decomposes the target dynamic interaction graph into interaction sub-tasks which execute at least one interaction operation with at least one element feature in an execution environment in sequence; and finally, sequentially executing the page interaction agent according to the sequence of the interaction subtasks so as to complete the execution of the to-be-executed instruction. According to the method and the system, compared with a traditional browser automatic script which only depends on a preset rule, complex tasks of multi-step decision making can be processed, the intelligent level of man-machine interaction is remarkably improved, the task application range is remarkably widened, and meanwhile the response speed of task processing is also improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-agent collaborative interaction method and system. BACKGROUND

[0002] With the rapid development of Internet technology and the wide application of artificial intelligence, the use scenarios of automated tools in daily life are increasing. Traditional automated solutions mainly rely on pre-written fixed instruction sequences (i.e., scripts), which simulate user operations to perform specific tasks. For example, in the field of web automation, scripts can be used to automatically fill out forms, click buttons, or perform simple data scraping operations, etc.

[0003] Existing script technology is usually based on pre-set static execution logic, which triggers pre-set operations by identifying the location, attributes, and other characteristics of interface elements. This implementation relies on a stable running environment and lacks the ability to adapt to environmental changes, making it unable to handle unforeseen interaction scenarios that occur during execution.

[0004] In terms of task complexity, traditional script technology also has obvious limitations. Taking an online ticketing system as an example, although scripts can be used to implement fixed process ticketing reservations, they cannot handle tasks that require decision-making, such as resolving the most suitable date, session, etc. information when the user's requirements do not explicitly specify detailed information, or continuing to work normally when the version is updated or the interface is updated. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a multi-agent collaborative interaction method and system. Compared with traditional browser automation scripts that rely solely on pre-set rules, the present application can handle complex tasks involving multiple steps of decision-making, significantly improving the intelligence level and task adaptation range of human-computer interaction, while also improving the response speed of task processing.

[0006] In a first aspect, the embodiments of the present application provide a multi-agent collaborative interaction method, which comprises: In response to receiving an execution instruction, a decision-making agent obtains an execution environment and a task set corresponding to the execution instruction; The decision-making agent generates a target dynamic interaction graph corresponding to the execution instruction based on a pre-set interaction graph and the task set, and decomposes the target dynamic interaction graph into interaction sub-tasks that sequentially perform at least one interaction operation with at least one element feature in the execution environment; The page interaction agent sequentially executes the interaction sub-tasks according to their order to complete the execution of the execution instruction.

[0007] Furthermore, the interactive subtasks in the target dynamic interaction graph include target interactive actions and corresponding expected effects of interactive subtasks; wherein, the expected effect of interactive subtasks represents the target effect that will be triggered when the target interactive action corresponding to the interactive subtask is executed on the target page of the execution environment.

[0008] Furthermore, the preset interaction graph includes all interactive actions in the execution environment and their corresponding expected interactive effects; the steps for creating the preset interaction graph include: At least one interface in the execution environment is obtained, at least one preset interactive action is performed on all interactive coordinate points on the interface, and the interface is captured when at least one element in the interface changes, so as to obtain at least one screenshot of the interface. Extract element features and structured data from at least one interface screenshot, and construct at least one mapping pair between element features, preset interactive actions, and structured data to generate the preset interactive map.

[0009] Furthermore, after each interactive subtask is completed, the interaction method further includes: For each interactive subtask, the decision-making agent obtains the actual execution effect corresponding to the execution of the interactive subtask, and compares the actual execution effect with the preset execution effect in the preset interaction graph to detect interaction anomalies.

[0010] Furthermore, comparing the actual execution effect with the preset execution effect in the preset interaction graph includes: Obtain the actual interaction screenshot triggered when executing the interactive subtask, and compare the actual interaction screenshot with the expected interaction screenshot; wherein, the expected interaction screenshot is the interface screenshot triggered when the same interactive operation is performed on the same interface when constructing the corresponding mapping pair; or, Obtain the actual structured data triggered when executing the interactive subtask, and compare the actual structured data with the expected structured data; wherein, the expected structured data is the structured data obtained when performing the same interactive operation on the same interface when constructing the corresponding mapping pair.

[0011] Furthermore, if the interaction anomaly detection fails, the interaction method further includes: The decision-making agent obtains the difference between the actual execution effect and the preset execution effect, and determines the anomaly detection result; The decision-making agent determines an anomaly repair strategy from the repair strategy database based on the anomaly detection results, and uses the anomaly repair strategy to update the interactive subtasks that have experienced interaction anomalies.

[0012] Furthermore, generating the target dynamic interaction graph corresponding to the instruction to be executed includes: Determine all complete interaction paths that can complete the set of subtasks from the preset interaction graph; The historical execution data of each subtask under each complete interaction path is obtained. Each complete interaction path is evaluated based on the historical execution data, and the optimal path is determined from multiple complete interaction paths. The target dynamic interaction map is generated based on the optimal path.

[0013] Furthermore, the evaluation of each complete interaction path based on the historical execution data, and the determination of the optimal path from multiple complete interaction paths, includes: For each complete interaction path, the performance results of the complete interaction path in terms of time, interaction stability, and resource consumption are determined based on the historical execution data of that complete interaction path. The selection of the optimal path is treated as an optimization problem. Based on the performance results of each complete interaction path in various dimensions, the optimal solution to the optimization problem is sought to obtain at least one optimal path.

[0014] Furthermore, the optimal solution to the optimization problem based on the performance results of each complete interaction path, to obtain at least one optimal path, includes: The performance results of each complete interaction path in each dimension are converted into a path performance vector corresponding to each complete interaction path; wherein, the path performance vector includes the total interaction time, the error rate, and the resource consumption. Traverse all path performance vectors, compare the performance results of each path performance vector in each dimension, filter out at least one complete interaction path that meets the preset performance standard, and select at least one optimal path from the at least one complete interaction path that meets the preset performance standard according to the preset optimization goal.

[0015] Secondly, embodiments of this application also provide a multi-agent collaborative interaction system, the interaction system being used to execute a multi-agent collaborative interaction method, the interaction system including a decision-making agent and a page interaction agent; The decision-making agent is used to obtain the execution environment and the set of subtasks corresponding to the instruction to be executed in response to receiving the instruction to be executed. The decision-making agent is also used to generate a target dynamic interaction graph corresponding to the instruction to be executed based on the preset interaction graph and the set of subtasks, and to decompose the target dynamic interaction graph into interactive subtasks that perform at least one interactive operation with at least one element feature in the execution environment in sequence. The page interaction agent is used to execute the interaction subtasks sequentially to complete the execution of the instruction to be executed.

[0016] The multi-agent collaborative interaction method and system provided in this application firstly, in response to receiving an instruction to be executed, the decision agent acquires the execution environment and the set of subtasks corresponding to the instruction to be executed; the decision agent generates a target dynamic interaction graph corresponding to the instruction to be executed based on a preset interaction graph and the set of subtasks, and decomposes the target dynamic interaction graph into interactive subtasks that sequentially perform at least one interactive operation with at least one element feature in the execution environment; finally, the page interaction agent executes the interactive subtasks sequentially to complete the execution of the instruction to be executed.

[0017] This application introduces a decision-making intelligent agent to perform semantic understanding and contextual analysis of user commands, intelligently identify the execution environment and construct a task framework, generating interactive subtasks based on the characteristics of elements in the execution environment. The page interaction intelligent agent then executes multiple interactive subtasks sequentially to complete the execution of the command to be executed. Compared to traditional browser automation scripts that rely solely on preset rules, this application can handle complex tasks involving multi-step decision-making, significantly improving the intelligence level and task adaptability of human-computer interaction, while also increasing the response speed of task processing.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a multi-agent collaborative interaction method provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of a multi-agent collaborative interaction system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0022] First, the applicable application scenarios for this application will be introduced. This application can be applied to the field of artificial intelligence technology.

[0023] With the rapid development of internet technology and the widespread application of artificial intelligence, automation tools are increasingly being used in daily life. Traditional automation solutions mainly rely on pre-written, fixed sequences of instructions (i.e., scripts), which perform specific tasks by simulating user actions. For example, in the field of web automation, scripts can perform one or more simple operations such as automatically filling out forms, automatically clicking buttons, or performing simple data scraping.

[0024] Existing scripting technologies are typically based on pre-defined static execution logic, triggering pre-defined operations by recognizing the position, attributes, and other characteristics of UI elements. This approach relies on a stable runtime environment, lacks adaptability to environmental changes, and cannot autonomously handle unforeseen interactive scenarios that occur during runtime.

[0025] Research has revealed that traditional scripting techniques also have significant limitations in terms of task complexity. For example, while online ticketing systems can implement fixed-process ticket booking through scripts, they cannot handle tasks requiring decision-making. For instance, they cannot parse the most suitable date and session information when the user's request does not explicitly specify detailed information, nor can they continue to function normally after version updates or interface updates.

[0026] Based on this, the embodiments of this application provide a multi-agent collaborative interaction method that can handle complex tasks involving multi-step decision-making, significantly improving the intelligence level and task adaptability of human-computer interaction, while also increasing the response speed of task processing.

[0027] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-agent collaborative interaction method provided in an embodiment of this application. Figure 1As shown in the embodiments of this application, the interaction method includes: Step 101: In response to receiving the instruction to be executed, the decision agent obtains the execution environment and the set of subtasks corresponding to the instruction to be executed.

[0028] Here, the intelligent agent, or intelligent proxy, is an object trained based on a Large Language Model (LLM) and specifically designed to complete a particular type of task. The decision-making agent is primarily responsible for receiving instructions to be executed, determining a suitable execution environment, and generating a set of subtasks to initiate the subsequent instruction execution process. Instructions to be executed are task requests input by the user to express a specific operational intention. These instructions can be textual or voice information. If the instruction is not textual, it needs to be converted to text. This application does not specifically limit the form or conversion method of the instructions to be executed. The set of subtasks refers to the result of further refining and decomposing the instructions to be executed. Each subtask is an executable task with a clear operational goal and expected result. Multiple subtasks constitute the actual execution path of the complete task. The execution environment refers to the hardware and software platform and its interface on which the instructions to be executed depend during runtime.

[0029] Regarding step 101 above, in specific implementation, when the decision-making agent receives the user's pending instruction, it obtains the execution environment for executing the instruction and the set of subtasks corresponding to the instruction. Here, the execution environment can be directly specified by the pending instruction or obtained by the decision-making agent through content analysis of the pending instruction. Specifically, the decision-making agent determines the execution environment based on a context-based intelligent intent decision-making method, including: obtaining the device type when the user issues the pending instruction, such as a computer / mobile device, the application environment the user is in, such as a webpage, software, or mini-program, and the user's state information, such as at least one of the following: current time, historical behavior, and historical preferences.

[0030] It should be understood that the above information requires user authorization to obtain, and the interaction method and system proposed in this application will not obtain the above information without user authorization. If the user does not authorize at least one of the information, the multi-agent collaborative interaction method will prioritize the scenario with the highest interaction efficiency, the lowest error rate, or the most suitable scenario that can realize the execution of the instruction from all feasible device terminals and all feasible application environments to be executed.

[0031] As an example, the instruction to be executed could be "Buy a high-speed rail ticket from Shanghai to Beijing South tomorrow." The decision-making agent can determine the execution environment based on this instruction, which could be any of the following: a browser on a computer, software on a computer, a webpage on a mobile device, software on a mobile terminal, or a mini-program on a mobile terminal. Each interaction stage is a different page within the execution environment. Based on the instruction to be executed, the following set of subtasks can be determined: selecting the departure point, selecting the destination, selecting the departure date, logging into the official ticketing environment using authorized account information, selecting passengers, and invoking the payment module to perform the payment operation.

[0032] Step 102: The decision-making agent generates a target dynamic interaction graph corresponding to the instruction to be executed based on the preset interaction graph and the set of subtasks, and decomposes the target dynamic interaction graph into interactive subtasks that sequentially perform at least one interactive operation with at least one element feature in the execution environment.

[0033] Specifically, the preset interaction graph includes all interactive actions in the execution environment and their corresponding expected effects. The target dynamic interaction graph includes multiple interaction sub-tasks and the element features required for each interaction sub-task. Element features include, but are not limited to, dynamic or static text, clickable links, buttons, input boxes, floating boxes, and other content presented in each interaction stage.

[0034] Regarding step 102 above, in specific implementation, after the decision-making agent determines the set of subtasks, it uses a pre-constructed preset interaction graph to obtain the interaction actions and expected effects corresponding to each subtask from the preset interaction graph through methods such as searching and matching, generates a target dynamic interaction graph corresponding to the instruction to be executed, and decomposes the target dynamic interaction graph into interactive subtasks that interact with at least one element feature in the execution environment in sequence.

[0035] Here, according to the embodiments provided in this application, each interaction subtask in the target dynamic interaction graph includes a target interaction action and a corresponding expected effect of the interaction subtask. The expected effect of the interaction subtask represents the target effect that will be triggered when the target interaction action corresponding to the interaction subtask is executed on the target page of the execution environment.

[0036] For example, continuing the example in step 101, when the execution environment is a browser on a computer, logging in using authorized account information includes the following sub-tasks: "Enter the user account in the account input control at coordinates (A, B) on the interface", "Enter the password in the password input control at coordinates (B, C)", and "Click the login button at coordinates (E, F)". The corresponding element features can be identified as the account input control, password input control, and login button on the browser page. The expected results of each interactive sub-task are: displaying the corresponding account name in the account input control, displaying the corresponding password in plaintext or ciphertext form in the password input control, and displaying login success.

[0037] For "Select Departure Point", the following subtasks can be included: "Click the departure point selection button at coordinates (G, H) on the interface" and "Select the departure point Shanghai in the departure point information display control at coordinates (I, J)". The element features can be identified as the departure point selection button and the departure point information display control on the browser page. The target interaction action is to click the departure point selection button, and the corresponding expected effect is to pop up the departure point information display control on the browser page. The target interaction action is to select the departure point "Shanghai" in the pop-up departure point information display control, and the corresponding expected effect of the interaction subtask is to display "Shanghai" in the departure point selection control.

[0038] As an optional embodiment, regarding step 102 above, generating the target dynamic interaction graph corresponding to the instruction to be executed includes: Step 1021: Determine all complete interaction paths from the preset interaction graph that can complete the set of subtasks.

[0039] Regarding step 1021 above, in specific implementation, the decision-making agent determines all complete interaction paths that can complete the set of subtasks from the preset interaction graph. An interaction path includes several subtasks. For example, continuing with steps 101-102, one interaction path includes at least the following subtasks: "Enter user account in the account input control at coordinates (A, B) on the interface," "Enter password in the password input control at coordinates (B, C)," "Click the login button at coordinates (E, F)," "Click the departure point selection button at coordinates (G, H) on the interface," and "Select departure point Shanghai in the departure point information display control at coordinates (I, J)." Here, the decision-making agent analyzes the preset interaction graph, searching for all feasible paths from the first subtask to the last subtask.

[0040] Step 1022: Obtain historical execution data of each subtask under each complete interaction path, evaluate each complete interaction path based on the historical execution data, determine the optimal path from multiple complete interaction paths, and generate the target dynamic interaction map based on the optimal path.

[0041] Regarding step 1022 above, in specific implementation, historical execution data for each complete interaction path is acquired, and each complete interaction path is evaluated using the historical execution data. The optimal path is determined from multiple complete interaction paths, and the selected optimal path is transformed into a sequence of several <interaction subtask-expected effect> pairs containing execution order information to generate the target dynamic interaction graph.

[0042] Furthermore, regarding step 1022 above, the evaluation of each complete interaction path based on the historical execution data, and the determination of the optimal path from multiple complete interaction paths, includes: Step 10221: For each complete interaction path, determine the performance results of the complete interaction path in terms of time dimension, interaction stability dimension, and resource consumption dimension based on the historical execution data of the complete interaction path.

[0043] Regarding step 10221 above, in practical implementation, for each determined complete interaction path, the performance results of that complete interaction path in terms of time, interaction stability, and resource consumption are determined based on the historical execution data of that complete interaction path. Specifically, the time dimension is used to measure the total interaction time required for the complete interaction path to complete the corresponding instruction. The interaction stability dimension is used to measure the number or probability of interaction anomalies encountered by the complete interaction path during historical execution. The resource consumption dimension is used to measure the computing resources (e.g., CPU, memory, token overhead, etc.) or network traffic required by the complete interaction path.

[0044] Step 10222: Treat the selection of the optimal path as an optimization problem, and find the optimal solution to the optimization problem based on the performance results of each complete interaction path in each dimension, so as to obtain at least one optimal path.

[0045] Regarding step 10222 above, in specific implementation, the selection of the optimal path is treated as an optimization problem. Based on the performance results of each complete interaction path determined in step 10221 across various dimensions, the optimal solution to the optimization problem is sought to obtain at least one optimal path. The optimal path should be the path that is optimal in terms of required time, anomaly rate, and resource consumption.

[0046] As an optional implementation, for each complete interaction path, firstly, based on the historical execution data of each subtask within that complete interaction path, the scores of each subtask in the aforementioned dimensions are determined. Then, the scores of each dimension are weighted and summed to obtain the score of the complete interaction path, and this score is used as the performance result of the complete interaction path. By comparing the scores of multiple complete interaction paths, the complete interaction path with the highest score is selected as the optimal path.

[0047] As another optional embodiment, regarding step 10222 above, the step of finding the optimal solution to the optimization problem based on the performance results of each complete interaction path to obtain at least one optimal path includes: (1) Convert the performance results of each complete interaction path in each dimension into the path performance vector corresponding to each complete interaction path.

[0048] Regarding step (1) above, in specific implementation, the performance results of each complete interaction path in each dimension are converted into the path performance vector N corresponding to each complete interaction path. i =[S1,S2,S3]. Wherein, in the path performance vector, S1 represents the total interaction time, S2 represents the exception rate, and S3 represents the resource consumption.

[0049] Specifically, when constructing a path performance vector, it is necessary to unify the optimization direction of each dimension. For example, the smaller the values ​​of the dimensions of total interaction time, exception rate, and resource consumption, the better. Therefore, the vector should be constructed based on the total interaction time, exception rate, and resource consumption, rather than using the "success rate" which should be constructed with the highest possible value.

[0050] (2) Traverse all path performance vectors, compare the performance results of each path performance vector in each dimension, filter out at least one complete interaction path that meets the preset performance standard, and select at least one optimal path from at least one complete interaction path that meets the preset performance standard according to the preset optimization goal.

[0051] Regarding step (2) above, in specific implementation, it includes: after determining the path performance vector corresponding to the complete interaction path, traversing all path performance vectors, comparing the performance of each path performance vector in each dimension, and selecting at least one complete interaction path that meets the preset performance standard. Continuing the example in the above steps, since the values ​​of each dimension are better the smaller they are, the dominant strategy for selecting a better complete interaction path includes: at least one dimension of the original first complete interaction path is lower than the corresponding dimension of the second complete interaction path, and the other dimensions of the first complete interaction path are not greater than the other dimensions of the second complete interaction path.

[0052] The process of finding the optimal solution includes: Step S1: Select any path Pi from the set S of all complete interaction paths; Step S2: Traverse all other paths Pj in the set S based on the dominance policy; Step S3: If Pj dominates Pi, then Pi is not the optimal solution. Discard Pi and return to step S1 above, selecting other paths that have not yet been checked. If, after traversal, no path dominates Pi, then Pi is the optimal solution. Repeat steps S1-S3 above until all paths in the set S have been checked.

[0053] It should be understood that at least one optimal path may ultimately be obtained. If there are multiple optimal paths, the path performance vectors corresponding to all optimal paths are displayed in a three-dimensional space. Based on the values ​​of each dimension of the path performance vector of the optimal path, three-dimensional coordinate points (S1, S2, S3) are determined, and connecting these three-dimensional coordinate points forms an optimal curve. Then, according to a preset selection strategy or dynamic real-time requirements, the most suitable target path is selected from all optimal path solutions. For example, if the current instruction to be executed has the highest requirement for execution speed, the shortest complete interaction path can be selected from the optimal path solutions as the final optimal path; if the current system resources are scarce, the complete interaction path with the lowest resource consumption can be selected from the optimal path solutions as the final optimal path.

[0054] As another optional embodiment, the path performance vectors corresponding to all optimal solutions can also be displayed in a lower-dimensional space. For example, when the interaction time is not sensitive, or the interaction time of each optimal path is relatively close, two-dimensional coordinate points (S2, S3) can be established by selecting only the two dimensions of abnormality rate and resource consumption, and then connecting each two-dimensional coordinate point to form a two-dimensional optimal curve. Similarly, two-dimensional coordinates can be established only based on the dimensions of total interaction time and abnormality rate, or only based on the dimensions of total interaction time and resource consumption.

[0055] Thus, based on the above steps, a performance vector is constructed by considering multiple dimensions such as time cost, historical success rate, and resource consumption. The set of non-dominated solutions is then calculated, and finally, the most suitable execution path is selected from this set based on real-time business requirements. Compared to a single path selection method, this application can flexibly respond to different application scenarios, achieving intelligent and dynamic path planning.

[0056] As an optional implementation, when new execution data feedback is available for the complete interaction path, the decision-making agent can update its path evaluation model and optimization strategy, thereby continuously improving its ability to find better paths.

[0057] According to the interaction method provided in this application, the steps for creating the preset interaction graph include: A: Obtain at least one interface in the execution environment, perform at least one preset interactive action on all interactive coordinate points on the interface, and capture the interface when at least one element in the interface changes, thereby obtaining at least one screenshot of the interface.

[0058] Here, a proxy can be deployed in the execution environment to execute one or more preset interactive actions from the operation library on the interface. Preset interactive actions may include clicking a button or link, hovering over a button, gesture operations, or inputting certain content, etc., and this application does not specifically limit these actions. Regarding step A above, in specific implementation, at least one interface in the execution environment is acquired. For each interface, the proxy executes at least one preset interactive action at all interactive coordinate points on that interface. When any element in the interface changes, the interface is captured to obtain at least one interface image. Continuing the example in step 102 above, for example, if the interface is a high-speed rail ticket purchase interface, and the preset interactive action is clicking the departure location selection button, then an element in the interface changes, i.e., the departure location information display control appears. At this time, the interface is captured to obtain an interface image.

[0059] B: Extract element features and structured data from at least one interface screenshot, and construct at least one mapping pair between element features, preset interactive actions, and structured data to generate the preset interactive map.

[0060] Regarding step B above, in specific implementation, based on a computer vision model, all element features are extracted from at least one interface image. Specifically, element features may include the shape, form, pattern, size, coordinates, and text around and above each element, etc., which are not specifically limited in this application. Based on a structured data analysis model, corresponding structured data are extracted from at least one interface. Then, at least one element feature, preset interactive actions, and structured data are stored in a mapping relationship to construct mapping pairs and generate a preset interactive graph.

[0061] As an optional implementation, when performing an environment update, steps A-B above need to be re-executed based on the updated results to dynamically update the preset interaction graph. This allows the decision-making agent to automatically detect interface changes and reconstruct interaction paths in scenarios such as website or application version updates or UI adjustments, effectively solving the problem of traditional automation tools failing due to interface changes.

[0062] S103, the page interaction agent executes the interaction subtasks sequentially according to their order to complete the execution of the instruction to be executed.

[0063] Here, the page interaction agent is mainly used to execute each interaction subtask sequentially in the execution environment. Regarding step S103 above, in specific implementation, the page interaction agent executes each interaction subtask sequentially according to the order of each interaction subtask in the target dynamic interaction graph, thus completing the execution of the instructions to be executed.

[0064] According to the interaction method provided in this application, after each interaction subtask is completed, the interaction method further includes: For each interactive subtask, the decision-making agent obtains the actual execution effect corresponding to the execution of the interactive subtask, and compares the actual execution effect with the preset execution effect in the preset interaction graph to detect interaction anomalies.

[0065] In the specific implementation of the above steps, for each interactive subtask, the decision agent obtains the actual execution effect corresponding to the execution of the interactive subtask, compares the actual execution effect with the preset execution effect in the preset interaction graph, and detects whether the actual execution effect is the same as the expected execution effect in order to perform interaction anomaly detection.

[0066] As an optional embodiment, regarding the above steps, comparing the actual execution effect with the preset execution effect in the preset interaction graph includes: Obtain the actual interaction screenshot triggered when the interactive subtask is executed, and compare the actual interaction screenshot with the expected interaction screenshot.

[0067] Regarding the above steps, in specific implementation, the actual interaction screenshot triggered when executing the interactive subtask is obtained. Simultaneously, the expected interaction screenshot corresponding to the interactive subtask is obtained. The expected interaction screenshot is the interface screenshot triggered when the same interactive operation is performed on the same interface during the construction of the corresponding mapping pair. Interaction anomaly detection is performed by comparing the actual interaction screenshot and the expected interaction screenshot. Specifically, a computer vision model is used for comparison to identify whether element features have been added, deleted, changed in shape, or changed in position. Here, continuing the example from the previous steps, when the interactive subtask is clicking the departure location selection button, a departure location information display control will appear in the expected interaction screenshot. In this case, during interaction anomaly detection, it is necessary to compare whether a departure location information display control with the same shape appears in the same position in the actual interaction screenshot. If it exists, the interactive subtask is considered to be free of interaction anomalies; otherwise, the interactive subtask is considered to have an anomaly.

[0068] As another optional embodiment, regarding the above steps, comparing the actual execution effect with the preset execution effect in the preset interaction graph includes: Obtain the actual structured data triggered when executing the interactive subtask, and compare the actual structured data with the expected structured data.

[0069] Regarding the above steps, in specific implementation, the actual structured data triggered when executing the interactive subtask is obtained. Simultaneously, the expected structured data corresponding to the interactive subtask is also obtained. Expected structured data is the structured data obtained when performing the same interactive operation on the same interface during the construction of the corresponding mapping pair. Interaction anomaly detection is performed by comparing the actual structured data with the expected structured data. Specifically, a structured data analysis model is used to identify whether any elements have been deleted, added, or changed.

[0070] According to the interaction method provided in this application, if the interaction anomaly detection fails, the interaction method further includes: I: The decision-making agent obtains the difference between the actual execution effect and the preset execution effect, and determines the anomaly detection result.

[0071] II: The decision-making agent determines an anomaly repair strategy from the repair strategy database based on the anomaly detection results, and uses the anomaly repair strategy to update the interactive subtasks that have experienced interaction anomalies.

[0072] Regarding steps I-II above, in practical implementation, if an interaction anomaly is detected in an interactive subtask, the difference between the actual execution effect and the preset execution effect is obtained to determine the anomaly detection result. Then, the anomaly repair strategy corresponding to the anomaly detection result is determined from the repair strategy database. The repair strategy database can store information such as anomaly condition judgments, retry logic, and retry conditions. The decision-making agent repairs the interaction anomaly according to the anomaly repair strategy and updates the interactive subtask that experienced the anomaly.

[0073] As an optional implementation, if the exception repair strategy fails to fix the interaction exception, it indicates that the exception is not caused at the interaction operation level. It is necessary to check for database errors (such as incorrect account passwords), network errors, permission configuration errors, and business logic errors (such as a product purchase transaction showing "sold out"). In this case, the execution of the pending instruction should be stopped, an error report or error message should be generated, and the decision-making agent should provide feedback to the user through text, sound, images, etc. If the error is a database error, the decision-making agent can use other data in the database to replan the interaction sequence and execute it. For network errors, retries can be used; this application does not limit the error repair method.

[0074] Following these steps, the decision-making agent can compare the actual execution results with the expected effects in real time, and initiate anomaly attribution and repair processes when deviations are detected. This closed-loop mechanism not only ensures the accuracy of task execution but also supports automatic recovery from common problems such as coordinate offsets, missing elements, and network errors, greatly reducing the need for manual intervention.

[0075] The multi-agent collaborative interaction method provided in this application firstly, in response to receiving an instruction to be executed, the decision agent acquires the execution environment and the set of subtasks corresponding to the instruction to be executed; the decision agent generates a target dynamic interaction graph corresponding to the instruction to be executed based on a preset interaction graph and the set of subtasks, and decomposes the target dynamic interaction graph into interactive subtasks that sequentially perform at least one interactive operation with at least one element feature in the execution environment; finally, the page interaction agent executes the interactive subtasks sequentially to complete the execution of the instruction to be executed.

[0076] This application introduces a decision-making intelligent agent to perform semantic understanding and contextual analysis of user commands, intelligently identify the execution environment and construct a task framework, generating interactive subtasks based on the characteristics of elements in the execution environment. The page interaction intelligent agent then executes multiple interactive subtasks sequentially to complete the execution of the command to be executed. Compared to traditional browser automation scripts that rely solely on preset rules, this application can handle complex tasks involving multi-step decision-making, significantly improving the intelligence level and task adaptability of human-computer interaction, while also increasing the response speed of task processing.

[0077] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a multi-agent collaborative interaction system provided in an embodiment of this application. The interaction system is used to execute a multi-agent collaborative interaction method, such as... Figure 2 As shown, the interactive system 200 includes a decision-making agent 201 and a page interaction agent 202.

[0078] The decision-making agent 201 is used to obtain the execution environment and the set of subtasks corresponding to the instruction to be executed in response to receiving the instruction to be executed. The decision-making agent 201 is also used to generate a target dynamic interaction graph corresponding to the instruction to be executed based on the preset interaction graph and the set of subtasks, and to decompose the target dynamic interaction graph into interactive subtasks that perform at least one interactive operation with at least one element feature in the execution environment in sequence. The page interaction agent 202 is used to execute the interaction subtasks sequentially to complete the execution of the instruction to be executed.

[0079] Furthermore, the interactive subtasks in the target dynamic interaction graph include target interactive actions and corresponding expected effects of interactive subtasks; wherein, the expected effect of interactive subtasks represents the target effect that will be triggered when the target interactive action corresponding to the interactive subtask is executed on the target page of the execution environment.

[0080] Furthermore, the preset interaction graph includes all interactive actions in the execution environment and their corresponding expected effects; the decision-making agent 201 is also used to create the preset interaction graph through the following steps: At least one interface in the execution environment is obtained, at least one preset interactive action is performed on all interactive coordinate points on the interface, and the interface is captured when at least one element in the interface changes, so as to obtain at least one screenshot of the interface. Extract element features and structured data from at least one interface screenshot, and construct at least one mapping pair between element features, preset interactive actions, and structured data to generate the preset interactive map.

[0081] Furthermore, after each interactive subtask is completed, the decision-making agent 201 is also used to: For each interactive subtask, the actual execution effect corresponding to the execution of the interactive subtask is obtained, and the actual execution effect is compared with the preset execution effect in the preset interaction graph to detect interaction anomalies.

[0082] Furthermore, when comparing the actual execution effect with the preset execution effect in the preset interaction graph, the decision-making agent 201 is also used to: Obtain the actual interaction screenshot triggered when executing the interactive subtask, and compare the actual interaction screenshot with the expected interaction screenshot; wherein, the expected interaction screenshot is the interface screenshot triggered when the same interactive operation is performed on the same interface when constructing the corresponding mapping pair; or, Obtain the actual structured data triggered when executing the interactive subtask, and compare the actual structured data with the expected structured data; wherein, the expected structured data is the structured data obtained when performing the same interactive operation on the same interface when constructing the corresponding mapping pair.

[0083] Furthermore, if the interaction anomaly detection fails, the decision-making agent 201 is also configured to: Obtain the difference between the actual execution effect and the preset execution effect to determine the anomaly detection result; Based on the anomaly detection results, an anomaly repair strategy is determined from the repair strategy database, and the anomaly repair strategy is used to update the interactive subtasks that have experienced interaction anomalies.

[0084] Furthermore, when generating the target dynamic interaction graph corresponding to the instruction to be executed, the decision-making agent 201 is also used to: Determine all complete interaction paths that can complete the set of subtasks from the preset interaction graph; The historical execution data of each subtask under each complete interaction path is obtained. Each complete interaction path is evaluated based on the historical execution data, and the optimal path is determined from multiple complete interaction paths. The target dynamic interaction map is generated based on the optimal path.

[0085] Furthermore, when the decision-making agent 201 evaluates each complete interaction path based on the historical execution data and determines the optimal path from multiple complete interaction paths, the decision-making agent 201 is also used to: For each complete interaction path, the performance results of the complete interaction path in terms of time, interaction stability, and resource consumption are determined based on the historical execution data of that complete interaction path. The selection of the optimal path is treated as an optimization problem. Based on the performance results of each complete interaction path in various dimensions, the optimal solution to the optimization problem is sought to obtain at least one optimal path.

[0086] Furthermore, when the decision-making agent 201 is used to find the optimal solution to the optimization problem based on the performance results of each complete interaction path to obtain at least one optimal path, the decision-making agent 201 is also used to: The performance results of each complete interaction path in each dimension are converted into a path performance vector corresponding to each complete interaction path; wherein, the path performance vector includes the total interaction time, the error rate, and the resource consumption. Traverse all path performance vectors, compare the performance results of each path performance vector in each dimension, filter out at least one complete interaction path that meets the preset performance standard, and select at least one optimal path from the at least one complete interaction path that meets the preset performance standard according to the preset optimization goal.

[0087] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0088] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the multi-agent collaborative interaction method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0089] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the multi-agent collaborative interaction method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0090] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0092] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0094] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0096] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-agent collaborative interaction method, characterized in that, The interaction method includes: In response to receiving a command to be executed, the decision-making agent obtains the execution environment and the set of subtasks corresponding to the command to be executed; The decision-making agent generates a target dynamic interaction graph corresponding to the instruction to be executed based on a preset interaction graph and the set of subtasks, and decomposes the target dynamic interaction graph into interactive subtasks that sequentially perform at least one interactive operation with at least one element feature in the execution environment. The page interaction agent executes the interaction subtasks sequentially to complete the execution of the instructions to be executed.

2. The interaction method according to claim 1, characterized in that, The interactive subtasks in the target dynamic interaction graph include target interactive actions and corresponding expected effects of the interactive subtasks; wherein, the expected effect of the interactive subtasks represents the target effect that will be triggered when the target interactive action corresponding to the interactive subtask is executed on the target page of the execution environment.

3. The interaction method according to claim 1, characterized in that, The preset interaction graph includes all interactive actions in the execution environment and their corresponding expected interactive effects; The steps for creating the preset interactive graph include: At least one interface in the execution environment is obtained, at least one preset interactive action is performed on all interactive coordinate points on the interface, and the interface is captured when at least one element in the interface changes, so as to obtain at least one screenshot of the interface. Extract element features and structured data from at least one interface screenshot, and construct at least one mapping pair between element features, preset interactive actions, and structured data to generate the preset interactive map.

4. The interaction method according to claim 3, characterized in that, After each interactive subtask is completed, the following is also included: For each interactive subtask, the decision-making agent obtains the actual execution effect corresponding to the execution of the interactive subtask, and compares the actual execution effect with the preset execution effect in the preset interaction graph to detect interaction anomalies.

5. The interaction method according to claim 4, characterized in that, The step of comparing the actual execution effect with the preset execution effect in the preset interaction graph includes: Obtain the actual interaction screenshot triggered when executing the subtask, and compare the actual interaction screenshot with the expected interaction screenshot; wherein, the expected interaction screenshot is the interface screenshot triggered when the same interactive operation is performed on the same interface when constructing the corresponding mapping pair; or, Obtain the actual structured data triggered when executing the subtask, and compare the actual structured data with the expected structured data; wherein, the expected structured data is the structured data obtained when performing the same interactive operation on the same interface when constructing the corresponding mapping pair.

6. The interaction method according to claim 4, characterized in that, If the interaction anomaly detection fails, the interaction method further includes: The decision-making agent obtains the difference between the actual execution effect and the preset execution effect, and determines the anomaly detection result; The decision-making agent determines an anomaly repair strategy from the repair strategy database based on the anomaly detection results, and uses the anomaly repair strategy to update the interactive subtasks that have experienced interaction anomalies.

7. The interaction method according to claim 1, characterized in that, The generation of the target dynamic interaction graph corresponding to the instruction to be executed includes: Determine all complete interaction paths that can complete the set of subtasks from the preset interaction graph; The historical execution data of each subtask under each complete interaction path is obtained. Each complete interaction path is evaluated based on the historical execution data, and the optimal path is determined from multiple complete interaction paths. The target dynamic interaction map is generated based on the optimal path.

8. The interaction method according to claim 7, characterized in that, The process of evaluating each complete interaction path based on the historical execution data and selecting the complete interaction path with the highest score as the optimal path includes: For each complete interaction path, the performance results of the complete interaction path in terms of time, interaction stability, and resource consumption are determined based on the historical execution data of that complete interaction path. The selection of the optimal path is treated as an optimization problem. Based on the performance results of each complete interaction path in various dimensions, the optimal solution to the optimization problem is sought to obtain at least one optimal path.

9. The interaction method according to claim 8, characterized in that, The optimization problem is solved by applying the performance results of each complete interaction path to obtain at least one optimal path, including: The performance results of each complete interaction path in each dimension are converted into a path performance vector corresponding to each complete interaction path; wherein, the path performance vector includes the total interaction time, the error rate, and the resource consumption. Traverse all path performance vectors, compare the performance results of each path performance vector in each dimension, filter out at least one complete interaction path that meets the preset performance standard, and select at least one optimal path from the at least one complete interaction path that meets the preset performance standard according to the preset optimization goal.

10. A multi-agent collaborative interaction system, characterized in that, The interaction system is used to execute the multi-agent collaborative interaction method as described in any one of claims 1 to 9, wherein the interaction system includes a decision-making agent and a page interaction agent; The decision-making agent is used to obtain the execution environment and the set of subtasks corresponding to the instruction to be executed in response to receiving the instruction to be executed. The decision-making agent is also used to generate a target dynamic interaction graph corresponding to the instruction to be executed based on the preset interaction graph and the set of subtasks, and to decompose the target dynamic interaction graph into interactive subtasks that perform at least one interactive operation with at least one element feature in the execution environment in sequence. The page interaction agent is used to execute the interaction subtasks sequentially to complete the execution of the instruction to be executed.

Images