Graphical user interface operation method and device based on end-cloud multi-model cooperation and electronic equipment

Abstract

The application discloses a kind of based on end cloud multi-model cooperation's graphical user interface operation method and device and electronic equipment, the method includes: obtaining the user input at client, and using first machine learning model carries out semantic recognition, obtains instruction recognition result;When being identified as operation instruction, current interface image is obtained and image unit is divided;According to instruction semantic information, the semantic correlation between each image unit and operation instruction is calculated, to determine target image unit set;Operation instruction, target image unit set and corresponding semantic correlation are sent to cloud server, and instruction interface feature and multiple candidate operation schemes are generated using second machine learning model;At least some candidate operation schemes are executed by client, and operation result information is obtained;Again, the second machine learning model calculates the relevance of operation result information and operation instruction and corresponding candidate operation scheme, and determines final interface operation result.The application can improve the accuracy, robustness and execution efficiency of graphical user interface automatic operation.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a graphical user interface operation method, apparatus and electronic device based on edge-cloud multi-model collaboration. Background Technology

[0002] With the development of artificial intelligence, image recognition, and graphical user interface (GUI) interaction technologies, technical solutions based on natural language input-driven GUI operations are gaining increasing attention. These technologies are typically applied in scenarios such as smart terminal control, application automation, intelligent assistants, human-machine collaborative operation, and automated processing of complex software tasks. Their goal is to convert user-inputted natural language commands into specific operational processes for the GUI, thereby reducing the burden of manual operation for users and improving interaction efficiency.

[0003] In existing technologies, a common implementation involves the terminal device receiving text or voice commands input by the user, performing semantic parsing on the commands, and then combining this with a screenshot of the current interface, control recognition results, or interface structure information to determine the target control to be operated on. This process then generates interface actions such as clicking, inputting, swiping, and selecting. Other solutions employ cloud-based models to jointly analyze the interface images and command information uploaded by the terminal, identifying interactive elements in the current interface, inferring user intent, and outputting corresponding automated operation steps, thereby achieving task execution for the graphical user interface.

[0004] However, in graphical user interface (GUI) operation scenarios, user input typically exhibits characteristics such as flexible semantic expression, abstract task objectives, implicit constraints, and non-unique execution paths. Simply relying on terminal-side models or single-session interface analysis results often fails to accurately determine the interface areas and operation objects truly relevant to user commands. Especially when there are numerous interface elements, complex hierarchical structures, frequent dynamic page changes, or semantically similar controls, existing technologies are prone to problems such as inaccurate target element positioning, unreasonable task decomposition, and incorrect operation sequence inference, thereby affecting the accuracy of interface operation results.

[0005] Furthermore, many existing solutions typically focus on directly generating action or fixed operation procedures based on a single recognition result, lacking sufficient support for comparing, filtering, and dynamically adjusting multiple candidate operation solutions. When a candidate operation path fails to execute, or the resulting interface state is inconsistent with the user's original operation intention, the solutions often lack a closed-loop processing mechanism for determining the correlation of candidate operation solutions based on the execution result, iteratively updating, and re-executing. This results in a low success rate and poor robustness in complex task scenarios.

[0006] Meanwhile, in scenarios where edge and cloud collaborative processing is required, existing technologies do not adequately consider the collaborative relationships between edge semantic recognition, interface image unit filtering, cloud element segmentation, candidate task generation, execution result correlation calculation, and candidate operation scheme updating. It is still difficult to achieve a good balance between computational resource consumption, interface understanding accuracy, and operation execution success rate.

[0007] Therefore, there is a need in the existing technology for a technical solution that can improve the understanding of interface elements, candidate tasks and execution results during the operation of a graphical user interface under an edge-cloud collaborative architecture. Summary of the Invention

[0008] This application provides a graphical user interface operation method, apparatus, and electronic device based on edge-cloud multi-model collaboration, so as to improve the accuracy, adaptability, and execution success rate of automatic operation of graphical user interface.

[0009] According to an embodiment of this application, a graphical user interface operation method based on end-to-cloud multi-model collaboration is provided. The method includes: acquiring user input at a client, and using a first machine learning model to perform semantic recognition on the user input to obtain an instruction recognition result, wherein the instruction recognition result is at least used to characterize whether the user input is an operation instruction and the corresponding instruction semantic information; if the instruction recognition result characterizes the user input as an operation instruction, acquiring a current interface image, and dividing the current interface image into image units based on the instruction semantic information to obtain multiple image units; calculating the semantic correlation between each image unit and the operation instruction according to the instruction semantic information, and determining a target image unit set from the multiple image units based on the semantic correlation corresponding to each image unit, wherein the target image unit set includes image units with a semantic correlation greater than an image unit correlation threshold; sending the operation instruction, the target image unit set, and the semantic correlation corresponding to each target image unit to a cloud server, so as to use a second machine learning model to perform element partitioning based on the operation instruction, the target image unit set, and the semantic correlation corresponding to each target image unit to generate instruction interface features corresponding to the operation instruction, wherein the instruction interface features are used to characterize the semantic association between the operation instruction and the target image unit set; and so on. A second machine learning model is used to calculate the correlation between the interface features of each instruction and the preset operation task, as well as the order probability between each operation task in the candidate operation scheme, to generate multiple candidate operation schemes. Each candidate operation scheme includes multiple operation tasks with a correlation greater than the task correlation threshold, and the order of the operation tasks in each candidate operation scheme is the order with an order probability greater than the task order probability threshold. The client executes at least some of the candidate operation schemes to obtain operation result information corresponding to each candidate operation scheme. The operation result information includes at least the result interface image after execution, execution status information, and result association information corresponding to the executed operation task. The operation instruction, operation result information, and the candidate operation schemes corresponding to the operation result information are sent to the cloud server, and the second machine learning model is used to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on the operation instruction, operation result information, and the candidate operation schemes corresponding to the operation result information. Among the operation result information with a correlation greater than the result correlation threshold, the operation result information with the highest correlation is determined as the interface operation result corresponding to the operation instruction.

[0010] In some implementations, when all correlations are less than the result correlation threshold, the method further includes: updating the candidate operation schemes using a second machine learning model based on correlations and candidate operation schemes; and having the client execute the updated candidate operation schemes until at least one correlation is greater than the result correlation threshold. This approach allows for replanning and iterative execution based on execution feedback when the initial candidate operation schemes fail to produce satisfactory results.

[0011] In some implementations, the process of using a first machine learning model to perform semantic recognition on user input to obtain instruction recognition results includes: extracting at least one of textual semantic features, task intent features, and action description features from the user input; based on the extracted features, using the first machine learning model to determine whether the user input is an operation instruction, and when the user input is an operation instruction, extracting at least one of the target object, operation action, constraint conditions, and task objective corresponding to the user input as instruction semantic information.

[0012] In some implementations, the process of dividing the current interface image into image units based on instruction semantic information to obtain multiple image units includes: performing at least one image segmentation process on the current interface image to obtain multiple candidate regions; and merging or subdividing the multiple candidate regions by combining at least one of the text region, control region, icon region, menu region and window region in the current interface image to obtain multiple image units.

[0013] In some implementations, the process of calculating the semantic correlation between each image unit and the operation instruction based on the instruction semantic information includes: extracting at least one of the visual features, text features, position features, and interface structure features corresponding to each image unit; and calculating the semantic correlation between each image unit and the operation instruction based on the degree of matching between the features corresponding to each image unit and the instruction semantic information.

[0014] In some implementations, the process of using a second machine learning model to calculate the correlation between each instruction interface feature and a preset operation task, as well as the order probability between each operation task in the candidate operation scheme, to generate multiple candidate operation schemes includes: determining multiple candidate operation tasks associated with the operation instruction based on each instruction interface feature; calculating the task correlation between each candidate operation task and the instruction interface feature; for candidate operation tasks whose task correlation is greater than a task correlation threshold, calculating the order probability between each candidate operation task by combining at least one of the dependency relationship, pre- and post-constraint relationship, and historical execution order among the candidate operation tasks; and generating multiple candidate operation schemes by combining the task correlation and order probability.

[0015] In some implementations, the process of the client executing at least some of the candidate operation schemes from a plurality of candidate operation schemes includes: sorting the plurality of candidate operation schemes according to at least one of the scheme priority, scheme confidence, resource consumption prediction value and historical success rate; and executing at least some of the candidate operation schemes from the plurality of candidate operation schemes according to the sorting result to obtain operation result information corresponding to each candidate operation scheme.

[0016] In some implementations, the result-related information in the operation result information includes at least one of the following: interface change information corresponding to the executed operation task, difference information before and after the operation, information corresponding to newly added interface elements in the result interface image, success mark information, failure mark information, or waiting status information corresponding to the executed operation task.

[0017] In some implementations, the process of using a second machine learning model to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on the operation instruction, operation result information, and candidate operation schemes corresponding to the operation result information includes: calculating the instruction result correlation between the operation result information and the operation instruction; calculating the scheme execution consistency between the operation result information and the corresponding candidate operation scheme; calculating the state rationality of the operation result information in the current execution stage; and determining the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on at least two of the instruction result correlation, scheme execution consistency, and state rationality.

[0018] In some implementations, the process of updating candidate operation schemes based on relevance and candidate operation schemes using a second machine learning model includes: determining target candidate operation schemes corresponding to operation result information with relevance lower than the result relevance threshold; and adjusting at least one of the following in the target candidate operation scheme: operation task order, operation task parameters, operation task combination method, and candidate operation scheme priority, based on at least one of the failed operation task, low-confidence operation task, or abnormal state operation task in the target candidate operation scheme, to obtain the updated candidate operation scheme.

[0019] In some implementations, before sending the target image unit set to the cloud server, the method further includes: detecting privacy-sensitive information on the current interface image or the target image unit set; if privacy-sensitive information is detected, masking, blurring, replacing, or desensitizing the image area containing the privacy-sensitive information before sending it to the cloud server.

[0020] According to embodiments of this application, a graphical user interface operation device based on edge-cloud multi-model collaboration is also provided. The device includes an input recognition module, an image acquisition and segmentation module, a correlation determination module, a feature generation module, a scheme generation module, an execution acquisition module, a correlation calculation module, and a result output module. Specifically, the input recognition module acquires user input from the client and performs semantic recognition on the user input using a first machine learning model to obtain an instruction recognition result; the image acquisition and segmentation module acquires the current interface image when the instruction recognition result indicates that the user input is an operation instruction, and divides the current interface image into image units based on the instruction semantic information to obtain multiple image units; the correlation determination module calculates the semantic correlation between each image unit and the operation instruction based on the instruction semantic information, and determines a target image unit set from the multiple image units based on the semantic correlation corresponding to each image unit; the feature generation module sends the operation instruction, the target image unit set, and the semantic correlation corresponding to each target image unit to a cloud server, so as to... The system utilizes a second machine learning model to generate instruction interface features corresponding to the operation instructions; a scheme generation module generates multiple candidate operation schemes using the second machine learning model; an execution acquisition module allows the client to execute at least some of the candidate operation schemes to obtain operation result information corresponding to each candidate operation scheme; a correlation calculation module sends the operation instructions, operation result information, and corresponding candidate operation schemes to a cloud server to calculate the correlation between the operation result information, the operation instructions, and the corresponding candidate operation schemes using the second machine learning model; and a result output module identifies the operation result information with the highest correlation among the operation result information with a correlation greater than the result correlation threshold as the interface operation result corresponding to the operation instructions.

[0021] In some embodiments, the above-described apparatus further includes a scheme update module and an iterative execution module. The scheme update module is used to update the candidate operation scheme based on the correlation and candidate operation scheme using a second machine learning model when all correlations are less than the result correlation threshold. The iterative execution module is used by the client to execute the updated candidate operation scheme until at least one correlation is greater than the result correlation threshold.

[0022] In some implementations, the input recognition module can also be used to: extract at least one of text semantic features, task intent features, and action description features from the user input; based on the extracted features, use a first machine learning model to determine whether the user input is an operation instruction, and when the user input is an operation instruction, extract at least one of the target object, operation action, constraint conditions, and task objective corresponding to the user input as instruction semantic information.

[0023] In some implementations, the image acquisition and segmentation module can also be used to: perform at least one image segmentation process on the current interface image to obtain multiple candidate regions; and combine or subdivide the multiple candidate regions by combining at least one of the text region, control region, icon region, menu region and window region in the current interface image to obtain multiple image units.

[0024] In some implementations, the correlation determination module can also be used to: extract at least one of the visual features, text features, position features, and interface structure features corresponding to each image unit; and calculate the semantic correlation between each image unit and the operation instruction based on the degree of matching between the features corresponding to each image unit and the instruction semantic information.

[0025] In some implementations, the scheme generation module can also be used to: determine multiple candidate operation tasks associated with operation instructions based on the characteristics of each instruction interface; calculate the task correlation between each candidate operation task and the instruction interface characteristics; for candidate operation tasks whose task correlation is greater than the task correlation threshold, calculate the order probability between each candidate operation task by combining at least one of the dependency relationship, the pre- and post-constraint relationship and the historical execution order between each candidate operation task; and generate multiple candidate operation schemes based on the combination of task correlation and order probability.

[0026] In some implementations, the correlation calculation module can also be used to: calculate the correlation between operation result information and operation instructions; calculate the consistency of operation execution between operation result information and corresponding candidate operation schemes; calculate the rationality of the state of operation result information in the current execution stage; and determine the correlation between operation result information and operation instructions and corresponding candidate operation schemes based on at least two of instruction result correlation, scheme execution consistency and state rationality.

[0027] In some implementations, the scheme update module can also be used to: determine the target candidate operation scheme corresponding to the operation result information with a correlation lower than the result correlation threshold; and based on at least one of the failed operation tasks, low-confidence operation tasks, or abnormal state operation tasks in the target candidate operation scheme, adjust at least one of the operation task order, operation task parameters, operation task combination method, and candidate operation scheme priority in the original candidate operation scheme to obtain the updated candidate operation scheme.

[0028] This application also provides an electronic device, including one or more processors and a memory, the memory being used to store one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the above-described graphical user interface operation method based on edge-cloud multi-model collaboration.

[0029] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described graphical user interface operation method based on edge-cloud multi-model collaboration.

[0030] The graphical user interface operation method, apparatus, and electronic device based on edge-cloud multi-model collaboration provided in this application involve the client acquiring user input and performing semantic recognition using a first machine learning model. After recognizing the input as an operation command, the client acquires the current interface image and divides it into image units. Further, based on the semantic information of the command, the client calculates the semantic correlation between each image unit and the operation command to determine the target image unit set. The operation command, the target image unit set, and the corresponding semantic correlation are then sent to the cloud server. A second machine learning model performs element division and generates command interface features. Based on this, the client calculates the correlation between each command interface feature and a preset operation task, as well as the order probability between each operation task in the candidate operation schemes, to generate multiple candidate operation schemes. The client executes at least some of these candidate operation schemes to obtain operation result information. The second machine learning model then calculates the correlation based on the operation command, operation result information, and corresponding candidate operation schemes, and determines the final interface operation result from the operation result information where the correlation requirement is met. Compared to existing technologies where natural language-driven graphical user interface (GUI) operations suffer from inaccurate interface element positioning, insufficient understanding of the mapping relationship between complex interfaces and abstract user intents, limited candidate operation paths, lack of effective judgment of execution results, and low success rate in complex scenarios, this application combines edge-side semantic recognition and target image unit screening with cloud-side multi-model task planning and result judgment. This effectively reduces interference from irrelevant interface content, improves the accuracy of target interface region positioning, enhances the understanding of complex GUI structures and user operation intents, and improves the matching degree between interface operation results and the user's true intent by generating multiple candidate operation schemes and combining them with execution results for correlation judgment. This improves the accuracy, robustness, and execution efficiency of automatic GUI operations.

[0031] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0032] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of an embodiment of the graphical user interface operation method based on edge-cloud multi-model collaboration provided in this application; Figure 2 A schematic diagram of the structure of an embodiment of the graphical user interface operation device based on edge-cloud multi-model collaboration provided in this application; Figure 3 A schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0033] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0034] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish different objects and should not be construed as limiting the order of importance, sequence, or quantity. Unless otherwise expressly stated, the terms "comprising," "including," etc., are intended to cover non-exclusive inclusion, such that a process, method, apparatus, or system that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, apparatus, or system.

[0035] It should also be noted that, in the embodiments of this application, the client can be a mobile phone, tablet computer, laptop computer, vehicle terminal, wearable device, or other terminal device with graphical user interface display and interaction capabilities; the cloud server can be a single server, server cluster, distributed computing node, or other server-side device with model inference capabilities. The first machine learning model and the second machine learning model can be different instances of the same type of model, or they can be different types of machine learning models, such as one or more combinations of text semantic understanding models, visual language models, multimodal inference models, or task planning models.

[0036] Furthermore, the “graphical user interface operation” mentioned in this application may include, but is not limited to, click, double-click, long press, swipe, drag, input, check, switch, expand, collapse, return, confirm, submit and other operation forms; “operation task” can be understood as an executable sub-task divided to complete the user’s instruction; “candidate operation scheme” can be understood as an executable task sequence formed by combining multiple operation tasks in a specific order.

[0037] First Embodiment With the development of artificial intelligence, natural language understanding, and image recognition technologies, the use of natural language input to drive the graphical user interface (GUI) of terminal devices to automatically perform operations has been gradually applied in scenarios such as intelligent assistants, application automation control, complex software operation assistance, and cross-page task execution. In these scenarios, users typically want to express their operational intentions in a conversational and natural manner, allowing the system to automatically recognize the semantics of the instructions, locate interface elements, and complete the corresponding GUI operations.

[0038] In existing technologies, common solutions typically parse user input on the terminal side and, combined with screenshots, control trees, or interface element recognition results, directly determine the object to be operated on and generate the operation action. Some solutions also send interface images and user commands collected by the terminal to the cloud, where a cloud model analyzes the data and returns operation suggestions or automated execution results. These solutions can achieve a certain degree of automation in simple, single-step scenarios with relatively fixed interface structures.

[0039] However, in real-world graphical user interface (GUI) scenarios, user input often exhibits characteristics such as flexible expression, incomplete semantic information, abstract operational goals, and implicit constraints. Furthermore, GUIs themselves suffer from a large number of elements, dynamically changing layouts, complex interaction hierarchies, and numerous similar controls. Therefore, existing technologies are prone to problems such as inaccurate identification of interface target elements, unreasonable task decomposition, unstable task execution path selection, and difficulty in effectively judging execution results. Particularly in complex task scenarios, if the planned operation path fails to complete the task, existing technologies typically lack a closed-loop mechanism for re-judging, re-planning, and re-executing based on the execution result, thus affecting the overall success rate and robustness.

[0040] To address the aforementioned issues, this application proposes a graphical user interface (GUI) operation scheme based on edge-cloud multi-model collaboration. On the client side, a first machine learning model performs semantic recognition on user input and, combined with the current interface image, performs image unit segmentation and filtering to first identify the local interface content more relevant to the user command. Then, the operation command, the target image unit set, and their semantic relevance are sent to the cloud server, where a second machine learning model performs element segmentation, command interface feature generation, and candidate operation scheme generation. The client then executes the candidate operation scheme and obtains the operation result information. Finally, the cloud side determines the final interface operation result based on the correlation between the operation result information, the operation command, and the candidate operation scheme, or, if the correlation is insufficient, updates the candidate operation scheme and continues execution.

[0041] Therefore, this application effectively balances client-side processing efficiency with cloud-side multimodal understanding capabilities. On one hand, semantic recognition and image unit filtering are performed on the client side first, reducing irrelevant interface information transmitted to the cloud and lowering the cloud analysis burden. On the other hand, the cloud side generates candidate operation schemes based on richer model capabilities and performs correlation calculations on the execution results, improving the accuracy of task understanding and result judgment in complex interface scenarios. Furthermore, by introducing multiple candidate operation schemes and a dynamic update mechanism based on result correlation, this application can also improve the success rate, adaptability, and robustness of graphical user interface operations in complex task scenarios.

[0042] Second Embodiment Figure 1 This paper illustrates a graphical user interface operation method based on edge-cloud multi-model collaboration according to an embodiment of this application. The method can be executed collaboratively by a client and a cloud server. The client is primarily responsible for input acquisition, edge-side recognition, interface image processing, execution of candidate operation schemes, and result collection. The cloud server is primarily responsible for multimodal understanding, candidate scheme generation, and correlation calculation. Figure 1 As shown in the embodiments of this application, the graphical user interface operation method based on edge-cloud multi-model collaboration may include: S101. Obtain user input from the client and use the first machine learning model to perform semantic recognition on the user input to obtain the instruction recognition result.

[0043] In step S101, the client can obtain user input through text input boxes, voice input interfaces, shortcut entry points, or other interactive methods. If the user input is voice information, the client can first convert the voice into text through a voice recognition module, and then provide the text content to the first machine learning model for semantic recognition. The first machine learning model can be deployed locally on the client or it can be a lightweight client-side model used to quickly determine whether the current user input belongs to an operation command for the graphical user interface.

[0044] The instruction recognition result can at least be used to characterize whether the user input is an operation instruction and the corresponding semantic information of the instruction. For example, it can include at least two parts: first, the input type determination result, used to determine whether the current user input is an operation instruction that can be executed by the system; second, the semantic parsing result, used to represent the semantic information in the user input, such as the operation target, target object, execution constraints, execution scenario, or task purpose. For example, if a user inputs "Help me turn off the notification permission for this page," the first machine learning model can identify that the input belongs to an operation instruction and extract "notification permission" as the target object and "turn off" as the operation action.

[0045] In this embodiment, user input may contain omissions, pronouns, or context-dependent elements. For example, expressions such as "open it," "return to the previous page," and "continue the previous operation" may be used. The first machine learning model in this embodiment can also combine the current session context of the client, the recent interface state, or historical interaction information for joint analysis to improve the accuracy of recognizing the true intent of user input.

[0046] Furthermore, in this embodiment, at least one of the text semantic features, task intent features, and action description features in the user input can be extracted; based on the extracted features, a first machine learning model is used to determine whether the user input is an operation instruction, and when the user input is an operation instruction, at least one of the target object, operation action, constraint conditions, and task objective corresponding to the user input is extracted as instruction semantic information.

[0047] In this embodiment, text semantic features can represent keywords, phrase dependencies, or syntactic semantic relationships in the input content; task intent features can represent the type of task the user wishes to perform, such as querying, jumping, editing, submitting, or on / off control; action description features can represent the action semantics corresponding to specific interface actions, such as clicking, selecting, inputting, switching, or deleting. By jointly extracting and modeling these features, the first machine learning model's ability to understand complex natural language instructions can be enhanced.

[0048] S102. When the instruction recognition result indicates that the user input is an operation instruction, the current interface image is acquired, and the current interface image is divided into image units based on the instruction semantic information to obtain multiple image units.

[0049] In step S102, the client can take a screenshot of the currently displayed interface to form an image of the current interface. The current interface image can include visible area information from the current application page, system page, overlay page, pop-up page, or nested subpage. To better understand the correspondence between the current interface and operation commands, the current interface image can be divided into image units to break down the entire page image into multiple analyzable local areas.

[0050] Image unit segmentation can be achieved using methods based on visual segmentation, layout analysis, control detection, text region recognition, or interface structure parsing. Each image unit can correspond to a local area, control, icon, text region, menu item, window region, or a region formed by a combination of multiple adjacent elements in the interface. By segmenting the image into images, the analysis of the interface can be transformed from a whole-page level to a local unit level, thus providing a finer-grained object foundation for subsequent semantic relevance calculations.

[0051] In this embodiment, the image unit segmentation can also be adaptively processed based on the instruction semantic information. For example, when the instruction semantic information contains object types such as "button," "input box," "switch," and "drop-down item," the image unit segmentation process can improve the segmentation accuracy of the corresponding type of interface area; when the instruction semantic information contains position constraints such as "top right corner," "bottom," and "list," the corresponding area of ​​the interface can also be segmented with finer granularity.

[0052] In addition, in this embodiment of the application, at least one image segmentation process can be performed on the current interface image to obtain multiple candidate regions; and multiple candidate regions can be merged or subdivided by combining at least one of the text region, control region, icon region, menu region and window region in the current interface image to obtain multiple image units.

[0053] In this embodiment, image segmentation processing may include segmentation methods based on pixel edges, color regions, control borders, layout structures, or detection box outputs. Merging or subdividing candidate regions can avoid semantic confusion caused by excessively large image units or insufficient context caused by excessively small image units, thereby achieving a balance between analysis accuracy and computational efficiency.

[0054] S103. Based on the semantic information of the instruction, calculate the semantic correlation between each image unit and the operation instruction, and determine the target image unit set from multiple image units based on the semantic correlation of each image unit.

[0055] In this step, the client can extract feature information that reflects the meaning of the interface content for each image unit, and match this feature information with the instruction semantic information obtained in step S101 to calculate the semantic correlation between the image unit and the operation instruction. This semantic correlation can represent the degree of matching between a certain image unit and the user's current operation intention.

[0056] The semantic relevance of image units can be obtained in various ways. For example, the text recognition results in an image unit can be matched with the target object in the instruction; the visual appearance features of an image unit can be matched with the type of the target control; or a comprehensive judgment can be made by combining structural information such as the position, hierarchy, and adjacency of the image unit in the interface. Ultimately, the client can filter image units from multiple image units whose semantic relevance is greater than the image unit relevance threshold, and combine these image units into a target image unit set. The target image unit set includes image units whose semantic relevance is greater than the image unit relevance threshold. In this way, the entire page interface image can be "denoised" and "focused" on the device side first, excluding interface content that is obviously unrelated to the current operation command, and only retaining the interface area that is more likely to be related to the current operation. This reduces the data scale of subsequent cloud processing and improves the accuracy of subsequent element division and task planning.

[0057] In addition, in this embodiment, at least one of the visual features, text features, position features and interface structure features corresponding to each image unit can be extracted respectively; based on the degree of matching between the features corresponding to each image unit and the semantic information of the instruction, the semantic correlation between each image unit and the operation instruction can be calculated.

[0058] In this embodiment, visual features may include color distribution, shape outline, icon style, or control appearance; text features may include character content, keywords, text semantic vectors, or text categories; positional features may include the coordinate position, area, hierarchy, or relative position of an image unit; and interface structure features may include parent-child relationships, proximity relationships, arrangement relationships, or belonging regions between units. Joint analysis of multiple features helps improve the accuracy of image unit selection.

[0059] S104. The operation instructions, the target image unit set, and the semantic relevance of each target image unit are sent to the cloud server so that the second machine learning model can perform element partitioning based on the operation instructions, the target image unit set, and the semantic relevance of each target image unit to generate instruction interface features corresponding to the operation instructions.

[0060] In this step, the client can send the filtered set of target image units and their associated information to the cloud server. Since what is sent at this point is no longer a complete interface image, but rather a set of target image units filtered for relevance, the amount of data received and processed by the cloud is reduced, while retaining the interface content closely related to the current operation command. The cloud server can then perform deeper multimodal analysis on these target image units based on a second machine learning model.

[0061] Element segmentation can be understood as further identifying, classifying, and modeling the relationships between interface elements in a set of target image units. For example, a second machine learning model can identify buttons, input boxes, switches, labels, menu items, or icons contained in each image unit and analyze the relationships between these elements. Based on this, the generated command interface features can be used to characterize "which interface elements, element combinations, or interaction paths are more likely to correspond to the current operation command in the current interface environment."

[0062] The command interface features can take the form of vectors, graph structures, feature maps, or other internal representations that can be further processed by a second machine learning model. These features can characterize not only the relationship between operation commands and individual interface elements, but also the combination, hierarchical, and path relationships between operation commands and multiple elements, providing a foundation for the generation of subsequent candidate operation tasks.

[0063] Furthermore, in this embodiment of the application, before sending the target image unit set to the cloud server, the method according to this embodiment of the application may further include: detecting privacy-sensitive information on the current interface image or the target image unit set; if privacy-sensitive information is detected, masking, blurring, replacing, or desensitizing the image area containing the privacy-sensitive information before sending it to the cloud server.

[0064] In this embodiment, privacy-sensitive information may include user accounts, mobile phone numbers, ID card numbers, addresses, chat content, payment information, facial images, or other sensitive display content. By performing privacy detection and anonymization on the device side, data security and user privacy protection can be improved while ensuring the effectiveness of end-to-cloud collaboration.

[0065] S105. Use the second machine learning model to calculate the correlation between each instruction interface feature and the preset operation task, as well as the order probability between each operation task in the candidate operation scheme, so as to generate multiple candidate operation schemes.

[0066] In this step, the cloud server can identify multiple executable tasks corresponding to the current operation command based on the command interface features generated in step S104. For example, for the operation command "turn off notification permissions", possible operation tasks include "enter the settings page", "open the notification settings subpage", "locate the notification permission switch", and "turn off the corresponding switch", etc. Different interface structures or application paths may have different task decomposition methods and different execution orders.

[0067] To generate multiple candidate operation schemes, the second machine learning model first calculates the correlation between the features of each instruction interface and tasks in a pre-defined operation task library, filtering out candidate operation tasks with high relevance. Then, based on the dependencies, order constraints, and historical execution experience among these candidate operation tasks, the probability of forming a certain sequential combination of tasks is calculated. Finally, multiple tasks that meet the requirements of task relevance and order probability are combined into multiple candidate operation schemes. Each candidate operation scheme includes multiple operation tasks with a relevance greater than a task relevance threshold, and the order of the operation tasks in each candidate operation scheme is such that their order probability is greater than a task order probability threshold. This method of generating multiple candidate operation schemes is more robust than generating only a single path. This is because in complex interface scenarios, there may be different entry points, different operation sequences, or different branch pages to complete the same user intent. Pre-generating multiple candidate operation schemes provides alternatives for subsequent execution and result judgment.

[0068] Furthermore, in this embodiment, multiple candidate operation tasks associated with operation instructions can be determined based on the characteristics of each instruction interface; the task correlation between each candidate operation task and the instruction interface characteristics can be calculated; for candidate operation tasks whose task correlation is greater than the task correlation threshold, the order probability between each candidate operation task can be calculated by combining at least one of the dependency relationship, the pre- and post-constraint relationship and the historical execution order between each candidate operation task; and multiple candidate operation schemes can be generated based on the combination of task correlation and order probability.

[0069] For example, preset operation tasks can be pre-built into an operation task library, and the tasks in the task library can be organized according to application type, page type, control type, or action type. Secondly, the machine learning model can also incorporate historical success cases to refine the sequential probabilities, thereby improving the rationality and executability of candidate operation solutions.

[0070] S106. The client executes at least some of the candidate operation schemes from multiple candidate operation schemes to obtain operation result information corresponding to each candidate operation scheme.

[0071] In this step, after the cloud server generates candidate operation plans, it can send these plans to the client, which will then execute at least some of them. The client can call automated operation interfaces, input simulation interfaces, touch event injection interfaces, or system auxiliary function interfaces to execute interface operations according to the order of the operation tasks in the candidate operation plans.

[0072] During execution, the client can collect corresponding operation result information for each candidate operation plan. This result information can include the result interface image after execution, execution status information, and result association information corresponding to the executed operation task. Execution status information reflects whether each operation task was executed successfully, whether it timed out, whether an exception page was triggered, or whether it entered a waiting state; the result interface image is used to further analyze whether the current execution result meets the user's expectations.

[0073] In some implementations, the client does not necessarily need to execute all candidate operation schemes. Instead, it can prioritize executing candidate operation schemes with higher scores, higher success rates, or lower resource consumption. When an existing candidate operation scheme produces a better result, the execution of subsequent schemes can also be stopped early to reduce resource consumption.

[0074] Furthermore, in this embodiment, multiple candidate operation schemes can be sorted according to at least one of the scheme priority, scheme confidence, resource consumption prediction value, and historical success rate; at least some of the candidate operation schemes can be executed according to the sorting result to obtain operation result information corresponding to each candidate operation scheme.

[0075] For example, the priority of a solution can be calculated by a second machine learning model based on task relevance and order probability; the confidence level of a solution can represent the degree to which the system believes a candidate operation solution can complete the target task; the predicted resource consumption can represent the time, computing resources, or number of interaction steps required to execute the solution; and the historical success rate can be derived from historical execution data statistics. By ranking candidate operation solutions and prioritizing the execution of better solutions, overall execution efficiency can be improved.

[0076] In addition, in this embodiment of the application, the result association information in the operation result information includes at least one of the following: interface change information corresponding to the executed operation task, difference information before and after the operation, information corresponding to newly added interface elements in the result interface image, success mark information, failure mark information or waiting status information corresponding to the executed operation task.

[0077] For example, interface change information can reflect results such as page jumps, control state changes, text changes, and the appearance or disappearance of pop-ups; differences before and after an operation can be obtained by comparing the interface images or structural information before and after execution; information on newly added interface elements can help determine whether a new page state has been entered; success flags, failure flags, and waiting status information can be used to identify the state of the current execution stage, facilitating subsequent correlation calculations.

[0078] S107. Send the operation instruction, operation result information, and candidate operation schemes corresponding to the operation result information to the cloud server, so as to use the second machine learning model to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation schemes based on the operation instruction, operation result information, and candidate operation schemes corresponding to the operation result information.

[0079] In this step, the client can send the operation result information corresponding to each executed candidate operation scheme to the cloud server. The cloud server can use a second machine learning model to jointly analyze the original operation instruction, the candidate operation scheme itself, and the operation result information after execution, thereby calculating whether the operation result truly realizes the user's intent and whether the result is consistent with the candidate operation scheme.

[0080] Relevance can be understood as a comprehensive evaluation metric used to measure the degree of matching between the result of a candidate operation and the user's original operation goal, while also measuring whether the execution result meets the expected effect of the corresponding candidate operation. For example, if the user's intention is to "turn off notification permissions," and the interface does indeed show that the notification switch is off after execution, the relevance can be high; conversely, if execution only redirects to the settings page but does not turn off the switch, the relevance can be low.

[0081] By incorporating the operation result, operation command, and candidate operation schemes into the analysis simultaneously, we can avoid incorrectly judging execution success based solely on interface changes, and also avoid ignoring the actual execution result based solely on the expectation of the candidate scheme itself. This approach improves the accuracy of result determination.

[0082] Furthermore, in this embodiment, the correlation between the operation result information and the operation instruction can be calculated; the consistency of the operation result information and the corresponding candidate operation scheme can be calculated; the rationality of the state of the operation result information in the current execution stage can be calculated; and the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme can be determined based on at least two of the instruction result correlation, scheme execution consistency and state rationality.

[0083] For example, instruction result relevance reflects whether the result conforms to the semantics of the user instruction; scheme execution consistency reflects whether the result is consistent with the expected output of the candidate operation scheme; state rationality reflects whether the current result conforms to the state that should appear in the execution stage, such as intermediate pages, confirmation pop-ups, or final completion pages. By comprehensively judging multiple dimensions, the relevance calculation can be made more reliable.

[0084] S108. Among the operation result information with a correlation greater than the result correlation threshold, determine the operation result information with the highest correlation as the interface operation result corresponding to the operation instruction.

[0085] In this step, the cloud server can compare the operation results of multiple executed candidate operation schemes, filter out candidate results with a relevance greater than the result relevance threshold, and further determine the operation result with the highest relevance among these candidate results as the final interface operation result. In this way, even if there are multiple acceptable results, the optimal result can be selected based on the degree of relevance.

[0086] The final interface operation result can be returned to the client for result display, subsequent automatic operation connection, user confirmation prompts, or task completion records. For the user, this means that the system does not simply "execute an action," but rather selects the execution result that best matches their original intention after comprehensively comparing multiple candidate paths.

[0087] In this embodiment, if only one operation result information has a correlation greater than the result correlation threshold, the result corresponding to that operation result information can be directly determined as the final interface operation result; if there are multiple results with correlation greater than the result correlation threshold, the one with the highest correlation can be selected, and if necessary, factors such as scheme priority, execution time or resource consumption can be further combined for auxiliary judgment.

[0088] Furthermore, in this embodiment, when all correlations are less than the result correlation threshold, a second machine learning model is used to update the candidate operation scheme based on correlation and candidate operation scheme; the updated candidate operation scheme is executed by the client until at least one correlation is greater than the result correlation threshold.

[0089] For example, if multiple results obtained from the initial execution fail to meet the acceptable standard, it indicates that the current candidate operation scheme may have problems such as unreasonable task selection, unreasonable task order, or inaccurate task parameter settings. At this time, the second machine learning model can update the candidate operation scheme based on the correlation distribution, failure information, abnormal states, and intermediate interface changes corresponding to each candidate operation scheme, and regenerate an operation path that is more suitable for the current task scenario.

[0090] Furthermore, in this embodiment, a target candidate operation scheme can be determined corresponding to the operation result information with a correlation lower than the result correlation threshold; based on at least one of the failed operation task, low confidence operation task, or abnormal state operation task in the target candidate operation scheme, at least one of the operation task order, operation task parameters, operation task combination method, and candidate operation scheme priority in the original candidate operation scheme is adjusted to obtain an updated candidate operation scheme.

[0091] For example, if a candidate operation fails at a specific task node—such as failing to find the target control, accidentally triggering another page, or entering an abnormal state—then only the relevant task node can be updated locally, instead of completely overturning the entire path. This improves iterative update efficiency and convergence speed in complex scenarios.

[0092] The graphical user interface operation method based on edge-cloud multi-model collaboration provided in this application improves subsequent processing efficiency by using a first machine learning model on the client side to perform semantic recognition of user input and relevance filtering of interface image units, thereby filtering out a large amount of irrelevant interface content on the client side. By using a second machine learning model on the cloud side to perform element partitioning, instruction interface feature generation, candidate operation task filtering, candidate operation scheme construction, and result correlation calculation, the method enhances the understanding of the mapping relationship between complex graphical user interfaces and abstract operation intentions. By generating multiple candidate operation schemes based on task relevance and task order probability, and selecting the result after execution based on the correlation between the operation result information, operation instructions, and candidate operation schemes, the method improves the matching degree between the final interface operation result and the user's true intention. By dynamically updating candidate operation schemes and continuing execution when the correlation does not meet the requirements, the method enhances the adaptability and robustness in complex interface operation scenarios. By performing privacy-sensitive information detection and de-identification processing before sending interface image-related data, the method balances processing effectiveness and data security during edge-cloud collaboration.

[0093] Third Embodiment Figure 2 This is a schematic diagram of the structure of an embodiment of the graphical user interface operation device based on edge-cloud multi-model collaboration provided in this application. Figure 2 As shown in the embodiments of this application, the graphical user interface operation device based on edge-cloud multi-model collaboration may include: an input recognition module 21, an image acquisition and segmentation module 22, a correlation determination module 23, a feature generation module 24, a scheme generation module 25, an execution acquisition module 26, a correlation calculation module 27, and a result output module 28. The above modules can be implemented by software, hardware, or a combination of both, and can be integrated into the same processing platform or distributed across multiple computing nodes. For example, in this embodiment, the input recognition module 21, image acquisition and segmentation module 22, correlation determination module 23, and feature generation module 24 can be located in the client, while the scheme generation module 25, execution acquisition module 26, correlation calculation module 27, and result output module 28 can be located in the cloud server.

[0094] The input recognition module 21 can be used to obtain user input from the client and use the first machine learning model to perform semantic recognition on the user input to obtain the instruction recognition result.

[0095] The input recognition module 21 can accept various input sources such as text input, voice input, gesture input, or shortcut input, and convert the collected content into an input format suitable for processing by the first machine learning model. If the input is in the form of voice, the input recognition module 21 can also call the voice recognition unit to convert the voice into text before performing semantic recognition. The command recognition results output by this module can be directly used in subsequent image processing and task planning processes.

[0096] In this embodiment, the input recognition module 21 can also perform joint semantic recognition by combining the current application state, previous interaction information or historical task context, thereby improving the ability to understand omitted expressions, pronoun expressions and continuous conversational commands.

[0097] Furthermore, in this embodiment, at least one of the text semantic features, task intent features, and action description features in the user input can be extracted; based on the extracted features, a first machine learning model is used to determine whether the user input is an operation instruction, and when the user input is an operation instruction, at least one of the target object, operation action, constraint conditions, and task objective corresponding to the user input is extracted as instruction semantic information.

[0098] In this embodiment, the input recognition module 21 can have a built-in feature extraction unit or call a local semantic analysis unit to complete the structured representation of the user input. This provides a clearer target orientation for subsequent interface image analysis.

[0099] The image acquisition and segmentation module 22 can be used to acquire the current interface image when the instruction recognition result represents the user input as an operation instruction, and to segment the current interface image into image units based on the instruction semantic information to obtain multiple image units.

[0100] The image acquisition and segmentation module 22 can call the system screenshot interface, the interface rendering buffer reading interface, or the application's internal page capture interface to obtain the current interface image. After obtaining the interface image, this module can divide the interface image into multiple analyzable image units based on the page structure, visual boundaries, control detection results, or text region detection results.

[0101] In practice, this module can also select different segmentation strategies based on different application types or page types. For example, for settings pages, segmentation can be prioritized based on list items; for form pages, segmentation can be prioritized based on input boxes, labels, and button areas; and for pop-up pages, the title area, content area, and confirmation / cancel control area can be prioritized.

[0102] In addition, in this embodiment of the application, at least one image segmentation process can be performed on the current interface image to obtain multiple candidate regions; and multiple candidate regions can be merged or subdivided by combining at least one of the text region, control region, icon region, menu region and window region in the current interface image to obtain multiple image units.

[0103] In this embodiment, the image acquisition and segmentation module 22 can dynamically adjust the region according to the segmentation result to obtain a set of image units that are more suitable for semantic matching.

[0104] The correlation determination module 23 can be used to calculate the semantic correlation between each image unit and the operation instruction according to the instruction semantic information, and determine the target image unit set from multiple image units based on the semantic correlation corresponding to each image unit. The target image unit set includes image units whose semantic correlation is greater than the image unit correlation threshold.

[0105] The relevance determination module 23 can extract text content, visual features, location information, and structural relationship information for each image unit, and match them with the semantic information of the instruction output by the input recognition module 21 to determine the degree of relevance between each image unit and the current operation instruction. Subsequently, this module can filter out more relevant image units to form a target image unit set.

[0106] By setting up a relevance determination module 23 on the client side, data filtering can be completed before it is sent to the cloud. This reduces network transmission overhead and also reduces the cloud's analysis burden on irrelevant areas, thereby improving overall processing efficiency.

[0107] In addition, in this embodiment, at least one of the visual features, text features, position features and interface structure features corresponding to each image unit can be extracted respectively; based on the degree of matching between the features corresponding to each image unit and the semantic information of the instruction, the semantic correlation between each image unit and the operation instruction can be calculated.

[0108] In this embodiment of the application, the relevance determination module 23 can achieve semantic relevance scoring through multi-feature fusion to improve the accuracy of target image unit selection.

[0109] The feature generation module 24 can be used to send the operation instructions, the target image unit set, and the semantic relevance of each target image unit to the cloud server, so as to use the second machine learning model to perform element partitioning based on the operation instructions, the target image unit set, and the semantic relevance of each target image unit, and generate instruction interface features corresponding to the operation instructions.

[0110] The feature generation module 24 serves as a collaborative interface module between the client and the cloud server. This module is responsible for sending the filtered information from the client side to the cloud, and also corresponds to the cloud side's process of element segmentation and command interface feature generation based on the second machine learning model. In implementation, the command interface features can be generated by the cloud and returned to the client for caching, or they can be retained in the cloud for further use in subsequent scheme generation.

[0111] The feature generation module 24 can combine the local interface understanding results generated by the client with the multimodal reasoning capabilities in the cloud, thereby improving the ability to model the relationship between complex pages and abstract user intent.

[0112] In addition, in this embodiment of the application, before sending the target image unit set to the cloud server, the following can be done: to detect privacy-sensitive information on the current interface image or the target image unit set; if privacy-sensitive information is detected, the image area containing the privacy-sensitive information is masked, blurred, replaced, or desensitized and encoded before being sent to the cloud server.

[0113] In this embodiment, the feature generation module 24 can also work in conjunction with the edge privacy detection unit to ensure the security of data received in the cloud.

[0114] The scheme generation module 25 can be used to calculate the correlation between each instruction interface feature and the preset operation task, as well as the order probability between each operation task in the candidate operation scheme, using the second machine learning model to generate multiple candidate operation schemes. Each candidate operation scheme includes multiple operation tasks with a correlation greater than the task correlation threshold, and the order of the operation tasks in each candidate operation scheme is the order with an order probability greater than the task order probability threshold.

[0115] The solution generation module 25 is deployed on the cloud server side. It can call task planning models, rule reasoning models, or multimodal large models based on the characteristics of the command interface, and output multiple executable candidate operation solutions. Each candidate operation solution can be composed of multiple operation tasks in a specific order, and can be accompanied by auxiliary information such as priority, confidence level, or expected result description.

[0116] Since there may be multiple feasible paths for the same user intent, the solution generation module 25 can generate multiple candidate solutions simultaneously for subsequent execution and verification on the client side. This multi-candidate planning approach can improve adaptability in complex scenarios.

[0117] Furthermore, in this embodiment, multiple candidate operation tasks associated with operation instructions can be determined based on the characteristics of each instruction interface; the task correlation between each candidate operation task and the instruction interface characteristics can be calculated; for candidate operation tasks whose task correlation is greater than the task correlation threshold, the order probability between each candidate operation task can be calculated by combining at least one of the dependency relationship, the pre- and post-constraint relationship and the historical execution order between each candidate operation task; and multiple candidate operation schemes can be generated based on the combination of task correlation and order probability.

[0118] In this embodiment of the application, the scheme generation module 25 can access the operation task library and the historical execution knowledge base to support more accurate modeling of the relationship between task combinations and sequences.

[0119] The execution acquisition module 26 is used by the client to execute at least some of the candidate operation schemes among multiple candidate operation schemes to obtain operation result information corresponding to each candidate operation scheme. The operation result information includes at least the result interface image after execution, execution status information, and result association information corresponding to the executed operation task.

[0120] Although the data acquisition module 26 is executed Figure 2 While the middleware resides on the cloud server side, in an edge-cloud collaborative architecture, its function can be understood as the cloud initiating scheme scheduling, the client actually executing the operation, collecting the execution results, and then feeding them back to the cloud. This module can be responsible for the execution scheduling of candidate operation schemes, execution status management, and result collection process coordination.

[0121] To improve execution efficiency, the execution acquisition module 26 can control the client to prioritize certain schemes based on their priority or historical success rate, rather than blindly executing all schemes. This module can also decide to stop the execution of other schemes early once a candidate scheme has yielded high-quality results.

[0122] Furthermore, in this embodiment, multiple candidate operation schemes can be sorted according to at least one of the scheme priority, scheme confidence, resource consumption prediction value, and historical success rate; at least some of the candidate operation schemes can be executed according to the sorting result to obtain operation result information corresponding to each candidate operation scheme.

[0123] In this embodiment, the execution acquisition module 26 can serve as the core of the end-to-cloud collaborative scheduling, and can uniformly control execution resources, execution order and execution stop conditions.

[0124] In addition, in this embodiment of the application, the result association information in the operation result information includes at least one of the following: interface change information corresponding to the executed operation task, difference information before and after the operation, information corresponding to newly added interface elements in the result interface image, success mark information, failure mark information or waiting status information corresponding to the executed operation task.

[0125] In this embodiment of the application, the execution acquisition module 26 can continuously record the interface changes during the execution process and feed back the structured results to the cloud for subsequent correlation calculations.

[0126] The correlation calculation module 27 can be used to send the operation instruction, operation result information and the candidate operation scheme corresponding to the operation result information to the cloud server, so as to use the second machine learning model to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on the operation instruction, operation result information and the candidate operation scheme corresponding to the operation result information.

[0127] The correlation calculation module 27 can comprehensively evaluate the results obtained after executing each candidate operation scheme and output a correlation score or correlation level. The input of this module includes not only the execution result itself, but also the original operation instruction and the corresponding candidate operation scheme. Therefore, its evaluation result can more comprehensively reflect whether a certain result truly meets the user's needs.

[0128] By setting up the correlation calculation module 27, we can avoid simply using "page changes" or "task completion" as the basis for success judgment, and instead place more emphasis on the consistency between the result and the user's actual goal.

[0129] Furthermore, in this embodiment, the correlation between the operation result information and the operation instruction can be calculated; the consistency of the operation result information and the corresponding candidate operation scheme can be calculated; the rationality of the state of the operation result information in the current execution stage can be calculated; and the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme can be determined based on at least two of the instruction result correlation, scheme execution consistency and state rationality.

[0130] In this embodiment, the correlation calculation module 27 can combine multi-dimensional evaluation indicators to output a more stable result score.

[0131] The result output module 28 can be used to determine the operation result information with the highest correlation among the operation result information with a correlation greater than the result correlation threshold as the interface operation result corresponding to the operation instruction.

[0132] The result output module 28 can filter out the results that meet the requirements from multiple candidate results, and further select the best result as the final output. This module can send the final result to the client for display, or trigger subsequent business processes, such as continuing to execute subsequent automatic tasks, recording task completion logs, or prompting the user for confirmation.

[0133] In practical implementation, the result output module 28 can also be linked with the scheme update logic. When a qualified result exists, the result output module 28 is responsible for outputting the final result; when no qualified result exists, the scheme update-related functions will handle the further processing.

[0134] Furthermore, in this embodiment, when all correlations are less than the result correlation threshold, a second machine learning model can be used to update the candidate operation scheme based on correlation and candidate operation scheme; the updated candidate operation scheme can then be executed by the client until at least one correlation is greater than the result correlation threshold.

[0135] In this embodiment, the device may further include a scheme update module and an iterative execution module. The scheme update module is used to infer the weak links in the candidate operation schemes based on the low correlation results, and update the task order, task parameters, task combination method, or priority of the candidate operation schemes; the iterative execution module is used to control the client to continue execution according to the updated candidate operation schemes until a qualified result is obtained.

[0136] Furthermore, in this embodiment, a target candidate operation scheme can be determined corresponding to the operation result information with a correlation lower than the result correlation threshold; based on at least one of the failed operation task, low confidence operation task, or abnormal state operation task in the target candidate operation scheme, at least one of the operation task order, operation task parameters, operation task combination method, and candidate operation scheme priority in the original candidate operation scheme is adjusted to obtain an updated candidate operation scheme.

[0137] In this embodiment, the result output module 28 and the scheme update module can form a closed-loop control relationship to improve the device's adaptability in complex graphical user interface operation scenarios.

[0138] Fourth embodiment The above describes the internal functions and structure of a graphical user interface operating device based on edge-cloud multi-model collaboration, which can be implemented as an electronic device. Figure 3 A schematic diagram illustrating the structure of an embodiment of the electronic device provided in this application. (See attached diagram.) Figure 3 As shown, the electronic device includes a memory 31 and a processor 32.

[0139] Memory 31 is used to store programs. In addition to the programs described above, memory 31 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0140] The memory 31 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0141] The processor 32 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural network processor (NPU), or an artificial intelligence (AI) chip. The processor 32 is coupled to the memory 31 and executes the program stored in the memory 31. When the program runs, it executes the graphical user interface operation method based on edge-cloud multi-model collaboration described in Embodiment 2 above.

[0142] Furthermore, such as Figure 3 As shown, the electronic device may also include other components such as a communication component 33, a power supply component 34, an audio component 35, and a display 36. Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown.

[0143] Communication component 33 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 33 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 33 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0144] Power supply component 34 provides power to various components of the electronic device. Power supply component 34 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0145] Audio component 35 is configured to output and / or input audio signals. For example, audio component 35 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 31 or transmitted via communication component 33. In some embodiments, audio component 35 also includes a speaker for outputting audio signals.

[0146] Display 36 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can detect not only the boundaries of touch or swipe actions, but also the duration and pressure associated with the touch or swipe operation.

[0147] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A graphical user interface operation method based on edge-cloud multi-model collaboration, characterized in that, include: The system obtains user input from the client and uses a first machine learning model to perform semantic recognition on the user input to obtain an instruction recognition result. The instruction recognition result is used to characterize whether the user input is an operation instruction and the instruction semantic information corresponding to the user input. When the instruction recognition result indicates that the user input is an operation instruction, the current interface image is acquired, and the current interface image is divided into image units based on the instruction semantic information to obtain multiple image units; Based on the instruction semantic information, the semantic correlation between each image unit and the operation instruction is calculated respectively, and based on the semantic correlation corresponding to each image unit, a target image unit set is determined from the plurality of image units, wherein the target image unit set includes image units whose semantic correlation is greater than the image unit correlation threshold; The operation instruction, the target image unit set, and the semantic relevance of each target image unit are sent to the cloud server. The second machine learning model is used to perform element partitioning based on the operation instruction, the target image unit set, and the semantic relevance of each target image unit to generate instruction interface features corresponding to the operation instruction. The instruction interface features are used to characterize the semantic association between the operation instruction and the target image unit set. The second machine learning model is used to calculate the correlation between each instruction interface feature and the preset operation task, as well as the order probability between each operation task in the candidate operation scheme, to generate multiple candidate operation schemes. Each candidate operation scheme includes multiple operation tasks with a correlation greater than the task correlation threshold, and the order of the operation tasks in each candidate operation scheme is the order with an order probability greater than the task order probability threshold. The client executes at least some of the multiple candidate operation schemes to obtain operation result information corresponding to each candidate operation scheme. The operation result information includes at least the result interface image after execution, execution status information, and result association information corresponding to the executed operation task. The operation instruction, the operation result information, and the candidate operation scheme corresponding to the operation result information are sent to the cloud server so that the second machine learning model can be used to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on the operation instruction, the operation result information, and the candidate operation scheme corresponding to the operation result information. Among the operation result information with a correlation greater than the result correlation threshold, the operation result information with the highest correlation is determined as the interface operation result corresponding to the operation instruction.

2. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, Also includes: When all correlations are less than the result correlation threshold, the second machine learning model is used to update the candidate operation scheme based on the correlations and the candidate operation scheme; The updated candidate operation scheme is executed by the client and the correlation calculation and determination process is performed by the cloud server until at least one correlation is greater than the result correlation threshold.

3. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, The step of using a first machine learning model to perform semantic recognition on the user input to obtain instruction recognition results includes: Extract at least one of the text semantic features, task intent features, and action description features from the user input; Based on the extracted features, the first machine learning model is used to determine whether the user input is an operation command. When the user input is an operation command, at least one of the target object, operation action, constraint condition and task objective corresponding to the user input is extracted as the command semantic information.

4. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, The current interface image is divided into multiple image units based on the instruction semantic information, including: Perform at least one image segmentation process on the current interface image to obtain multiple candidate regions; By combining at least one of the text area, control area, icon area, menu area, and window area in the current interface image, the multiple candidate areas are merged or subdivided to obtain the multiple image units.

5. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, The step of calculating the semantic correlation between each image unit and the operation instruction based on the instruction semantic information includes: Extract at least one of the visual features, text features, positional features, and interface structure features corresponding to each image unit; Based on the degree of matching between the features corresponding to each image unit and the semantic information of the instruction, the semantic correlation between each image unit and the operation instruction is calculated.

6. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, The step of using the second machine learning model to calculate the correlation between each instruction interface feature and the preset operation task, as well as the order probability between each operation task in the candidate operation scheme, to generate multiple candidate operation schemes includes: Based on the characteristics of each instruction interface, determine multiple candidate operation tasks associated with the operation instruction; Calculate the task correlation between each of the candidate operation tasks and the instruction interface features; For candidate operation tasks whose task relevance is greater than the task relevance threshold, the order probability between each candidate operation task is calculated by combining at least one of the following: the dependency relationship between each candidate operation task, the pre- and post-constraint relationship, and the historical execution order. The multiple candidate operation schemes are generated based on the combination of task relevance and sequential probability.

7. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, The execution of at least some of the candidate operation schemes by the client includes: The candidate operation schemes are ranked according to at least one of the following: scheme priority, scheme confidence, predicted resource consumption, and historical success rate. Execute at least some of the candidate operation schemes from the plurality of candidate operation schemes according to the sorting results, so as to obtain operation result information corresponding to each candidate operation scheme.

8. The graphical user interface operation method based on edge-cloud multi-model collaboration according to claim 1, characterized in that, The step of using the second machine learning model to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on the operation instruction, the operation result information, and the candidate operation scheme corresponding to the operation result information includes: Calculate the instruction result correlation between the operation result information and the operation instruction; Calculate the consistency of the operation result information with the corresponding candidate operation scheme in terms of scheme execution; Calculate the rationality of the state of the operation result information in the current execution stage; Based on at least two of the following: the relevance of the instruction result, the consistency of the scheme execution, and the rationality of the state, the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme is determined.

9. A graphical user interface operating device based on edge-cloud multi-model collaboration, characterized in that, include: An input recognition module is used to acquire user input at the client and perform semantic recognition on the user input using a first machine learning model to obtain an instruction recognition result, wherein the instruction recognition result is used at least to characterize whether the user input is an operation instruction and the instruction semantic information corresponding to the user input; The image acquisition and segmentation module is used to acquire the current interface image when the instruction recognition result indicates that the user input is an operation instruction, and to segment the current interface image into image units based on the instruction semantic information to obtain multiple image units. The correlation determination module is used to calculate the semantic correlation between each image unit and the operation instruction according to the instruction semantic information, and to determine a target image unit set from the plurality of image units based on the semantic correlation corresponding to each image unit, wherein the target image unit set includes image units whose semantic correlation is greater than the image unit correlation threshold; The feature generation module is used to generate instruction interface features corresponding to the operation instruction by using a second machine learning model to perform element partitioning based on the operation instruction, the target image unit set, and the semantic relevance of each target image unit. The instruction interface features are used to characterize the semantic association between the operation instruction and the target image unit set. The scheme generation module is used to calculate the correlation between each instruction interface feature and the preset operation task, and the order probability between each operation task in the candidate operation scheme, using the second machine learning model, so as to generate multiple candidate operation schemes. Each candidate operation scheme includes multiple operation tasks with a correlation greater than the task correlation threshold, and the order of the operation tasks in each candidate operation scheme is the order with an order probability greater than the task order probability threshold. The execution acquisition module is used to execute at least some of the multiple candidate operation schemes at the client to obtain operation result information corresponding to each candidate operation scheme. The operation result information includes at least the result interface image after execution, execution status information, and result association information corresponding to the executed operation task. The correlation calculation module is used at the cloud server to use the second machine learning model to calculate the correlation between the operation result information and the operation instruction and the corresponding candidate operation scheme based on the operation instruction, the operation result information and the candidate operation scheme corresponding to the operation result information; The result output module is used to determine the operation result information with the highest correlation among the operation result information with a correlation greater than the result correlation threshold as the interface operation result corresponding to the operation instruction.

10. An electronic device, characterized in that, include: Memory, used to store programs; A processor is configured to run the program stored in the memory to perform the graphical user interface operation method based on edge-cloud multi-model collaboration as described in any one of claims 1 to 8.

Images