A GUI element operation method, system, device and storage medium

Abstract

The application discloses a GUI element operation method, system, device and storage medium. The GUI element operation method comprises the following steps: determining a GUI element to be operated, a scene category of the GUI element to be operated and an operation type of the GUI element to be operated in a GUI element operation command; determining all first GUI elements, a first bounding box and a first element confidence; determining a second bounding box of the GUI element to be operated based on the scene category; calculating an intersection-over-union of each first GUI element based on the first bounding box and the second bounding box; screening out the first GUI element with the intersection-over-union greater than a preset intersection-over-union threshold value as a second GUI element; and determining an environment stability value of the GUI element to be operated, and determining a confidence score based on the operation type, the first element confidence of the second GUI element and the environment stability value, thereby improving the execution success rate of the GUI element operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of GUI element operation, and in particular to a GUI element operation method, system, device and storage medium. BACKGROUND

[0002] At present, GUI (Graphical User Interface) has become the main way for people to interact with various electronic devices, and is widely used in computer operating systems, mobile applications and smart home control panels and many other fields. With the rapid development of technology, users' expectations for GUI interaction experience are increasing.

[0003] At present, AIAgent (Artificial Intelligence Agent) based on MCP (Model Context Protocol) or API interface and command line has made certain progress in automatic task execution. However, when facing office software operations (such as Excel multi-table data association and Word complex layout) and web dynamic interaction (such as form submission with pop-up window verification), it lacks dynamic execution strategy, resulting in a high failure rate of element operation execution. SUMMARY

[0004] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application provides a GUI element operation method, system, device and storage medium, which can improve the success rate of element operation execution.

[0005] In a first aspect of the present application, a GUI element operation method is provided, comprising the following steps:

[0006] In the case of receiving a GUI element operation command, determining the to-be-operated GUI element in the GUI element operation command, the scene category of the to-be-operated GUI element and the operation type of the to-be-operated GUI element;

[0007] Determining all first GUI elements of the interface where the to-be-operated GUI element is located, a first bounding box of each first GUI element and a first element confidence of each first GUI element;

[0008] Determining a second bounding box of the to-be-operated GUI element based on the scene category;

[0009] In the case where the first element confidence of each first GUI element is greater than a preset confidence threshold, calculating the intersection over union of each first GUI element based on the first bounding box and the second bounding box;

[0010] Screening out the first GUI element with an intersection over union greater than a preset intersection over union threshold as a second GUI element;

[0011] In a case where the number of the second GUI elements is one, an environment stability value of the GUI element to be operated is determined, wherein the environment stability value is a constant value used to represent whether a pop-up window appears in a case where the GUI element operation command is executed;

[0012] A confidence score is determined based on the operation type, the first element confidence of the second GUI element, and the environment stability value;

[0013] In a case where the confidence score is greater than a preset confidence score threshold, the GUI element operation command is ended.

[0014] According to the GUI element operation method provided in the embodiments of the present application, at least the following beneficial effects are achieved:

[0015] The method provided in the embodiments of the present application comprises the following steps: in a case where a GUI element operation command is received, determining a GUI element to be operated, a scene category of the GUI element to be operated, and an operation type of the GUI element to be operated; determining all first GUI elements of an interface in which the GUI element to be operated is located, a first bounding box of each first GUI element, and a first element confidence of each first GUI element; determining a second bounding box of the GUI element to be operated based on the scene category; in a case where the first element confidence of each first GUI element is greater than a preset confidence threshold, calculating an intersection-over-union of each first GUI element based on the first bounding box and the second bounding box; screening out a first GUI element with an intersection-over-union greater than a preset intersection-over-union threshold as a second GUI element; in a case where the number of the second GUI elements is one, determining an environment stability value of the GUI element to be operated, wherein the environment stability value is a constant value used to represent whether a pop-up window appears in a case where the GUI element operation command is executed; determining a confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value; and in a case where the confidence score is greater than a preset confidence score threshold, ending the GUI element operation command. The embodiments of the present application can realize positioning across scenes and across operation types by determining the second bounding box of the GUI element to be operated based on the scene category, and improve the positioning accuracy. The embodiments of the present application can further improve the accuracy of element positioning by calculating the intersection-over-union of each first GUI element based on the first bounding box and the second bounding box, and screening out the first GUI element with the intersection-over-union greater than the preset intersection-over-union threshold. The embodiments of the present application can realize a dynamic execution strategy combined with the confidence score by ending the GUI element operation command in a case where the confidence score is greater than the preset confidence score threshold, and improve the execution success rate of the GUI element operation.

[0016] According to some embodiments of the present application, before the determination of all first GUI elements of the interface in which the GUI element to be operated is located, the first bounding box of each first GUI element, and the first element confidence of each first GUI element, the method comprises the following steps:

[0017] screenshot the interface where the GUI element to be operated is located to obtain a screenshot image;

[0018] gray the screenshot image to obtain a gray image;

[0019] edge enhance the gray image to obtain an edge enhanced image;

[0020] The determining of all first GUI elements, a first bounding box of each first GUI element, and a first element confidence of each first GUI element in the interface where the GUI element to be operated is located comprises:

[0021] inputting the edge enhanced image into a trained YOLOv8 model to obtain all first GUI elements, the first bounding box of each first GUI element, and the first element confidence of each first GUI element output by the trained YOLOv8 model.

[0022] According to some embodiments of the present application, the method further comprises:

[0023] in the case where there are several first GUI elements with a first element confidence less than or equal to a preset confidence threshold, regarding the several first GUI elements as second GUI elements;

[0024] enlarging the image area of the first bounding box of all second GUI elements by a preset enlargement ratio value to obtain an enlarged image area of each second GUI element;

[0025] inputting the enlarged image area into the trained YOLOv8 model to obtain a third bounding box and a first updated element confidence of each second GUI element output by the trained YOLOv8 model;

[0026] updating the first bounding box of each second GUI element based on the third bounding box, and updating the first element confidence of each second GUI element based on the first updated element confidence.

[0027] According to some embodiments of the present application, the scene category is at least one of a webpage scene and a desktop application scene, and the determining of the confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value comprises:

[0028] In a case where the scene category is the webpage scene, a node identifier and a coordinate of the GUI element to be operated are acquired by a DOM tree method, and in a case where both the node identifier and the coordinate exist, the second element confidence is a first preset constant value; in a case where the scene category is the desktop application scene, a control state of the GUI element to be operated is acquired by a UI hierarchy collection method, and the second element confidence is determined based on matching of the control state and a preset control state rule, wherein the preset control state rule includes a value of the second element confidence corresponding to each control state, and the control state includes a control type;

[0029] The third element confidence is calculated based on the first element confidence of the second GUI element and the second element confidence.

[0030] An operation complexity value corresponding to the operation type is determined, wherein the operation complexity value is a scalar value representing complexity of the operation type.

[0031] The operation complexity value is multiplied by a first preset weight value to obtain a first product value.

[0032] The third element confidence is multiplied by a second preset weight value to obtain a second product value.

[0033] The environment stability value is multiplied by a third preset weight value to obtain a third product value.

[0034] A sum of the first product value, the second product value and the third product value is calculated.

[0035] The sum is taken as the confidence score.

[0036] According to some embodiments of the present application, the third element confidence is calculated based on the first element confidence of the second GUI element and the second element confidence, including:

[0037] The first element confidence of the second GUI element is multiplied by a first preset confidence weight value to obtain a first confidence product value.

[0038] The second element confidence is multiplied by a second preset confidence weight value to obtain a second confidence product value.

[0039] The first confidence product value and the second confidence product value are added to obtain the third element confidence.

[0040] According to some embodiments of the present application, the second bounding box of the GUI element to be operated is determined based on the scene category, including:

[0041] In a case where the scene category is a webpage scene, a second bounding box of the GUI element to be operated is obtained by a DOM tree method;

[0042] In a case where the scene category is a desktop application scene, the second bounding box of the GUI element to be operated is obtained by a UI hierarchy collection method.

[0043] According to some embodiments of the present application, the method further comprises:

[0044] In a case where the confidence score is less than or equal to the preset confidence score threshold, an execution result of the GUI element operation command is obtained, wherein the execution result is execution success or execution failure;

[0045] In a case where the execution result is execution failure, a failure reason of the GUI element to be operated is determined;

[0046] The failure reason is matched with a preset recovery rule database to obtain a recovery level and a recovery strategy;

[0047] An execution mode corresponding to the recovery level is determined, and the recovery strategy is executed based on the execution mode, wherein the execution mode includes automatic execution and manual execution.

[0048] In a second aspect of the present application, a GUI element operation system is provided, which comprises:

[0049] A data acquisition module is configured to determine a GUI element to be operated, a scene category of the GUI element to be operated, and an operation type of the GUI element to be operated in a case where a GUI element operation command is received;

[0050] A first bounding box and a first element confidence determination module is configured to determine all first GUI elements of an interface in which the GUI element to be operated is located, a first bounding box of each first GUI element, and a first element confidence of each first GUI element;

[0051] A second bounding box determination module is configured to determine a second bounding box of the GUI element to be operated based on the scene category;

[0052] An intersection over union calculation module is configured to calculate an intersection over union of each first GUI element based on the first bounding box and the second bounding box in a case where the first element confidence of each first GUI element is greater than a preset confidence threshold;

[0053] An element screening module is configured to screen out a first GUI element with an intersection over union greater than a preset intersection over union threshold as a second GUI element;

[0054] The environment stability value determination module is configured to determine an environment stability value of the GUI element to be operated when the number of the second GUI elements is one, wherein the environment stability value is a constant value used to represent whether a pop-up window appears when the GUI element operation command is executed.

[0055] The confidence score determination module is configured to determine a confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value.

[0056] The ending module is configured to end the GUI element operation command when the confidence score is greater than a preset confidence score threshold.

[0057] The system determines the GUI element to be operated, the scene category of the GUI element to be operated, and the operation type of the GUI element to be operated in the GUI element operation command when the GUI element operation command is received, determines all first GUI elements of the interface in which the GUI element to be operated is located, the first bounding box of each first GUI element, and the first element confidence of each first GUI element, determines the second bounding box of the GUI element to be operated based on the scene category, calculates the intersection over union of each first GUI element based on the first bounding box and the second bounding box when the first element confidence of each first GUI element is greater than a preset confidence threshold, screens out the first GUI element with the intersection over union greater than a preset intersection over union threshold as the second GUI element, determines the environment stability value of the GUI element to be operated when the number of the second GUI elements is one, wherein the environment stability value is a constant value used to represent whether a pop-up window appears when the GUI element operation command is executed, determines the confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value, and ends the GUI element operation command when the confidence score is greater than a preset confidence score threshold. The application can realize cross-scene and cross-operation type positioning based on the determination of the second bounding box of the GUI element to be operated based on the scene category, improves the positioning accuracy, further improves the element positioning accuracy based on the calculation of the intersection over union of each first GUI element based on the first bounding box and the second bounding box, and ends the GUI element operation command when the confidence score is greater than a preset confidence score threshold, realizes the dynamic execution strategy combined with the confidence score, and improves the execution success rate of the GUI element operation.

[0058] In a third aspect, the application provides a GUI element operation electronic device, which comprises at least one control processor and a memory connected in communication with the at least one control processor. The memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the GUI element operation method described above.

[0059] In a fourth aspect of the present application, a computer readable storage medium is provided, which stores computer executable instructions for causing a computer to execute the GUI element operation method described above.

[0060] It should be noted that the beneficial effects between the second aspect to the fourth aspect of the present application and the prior art are the same as the beneficial effects between the GUI element operation system described above and the prior art, which will not be described here in detail.

[0061] Additional aspects and advantages of the present application will be in part apparent and in part pointed out below in the description of the application. BRIEF DESCRIPTION OF DRAWINGS

[0062] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings, wherein:

[0063] Figure 1 is a flow diagram of an embodiment of the GUI element operation method provided by the present application;

[0064] Figure 2 is a structural diagram of an embodiment of the GUI element operation system provided by the present application;

[0065] Figure 3 is a structural diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0066] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which examples of the embodiments are shown, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0067] In the description of the present application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0068] In the description of the present application, it should be understood that the position description, such as the position or location relationship indicated by up, down, etc., is based on the position or location relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element indicated must have a particular position, be constructed and operated in a particular position, and therefore cannot be understood as a limitation of the present application.

[0069] In the description of the present application, it should be noted that, unless otherwise explicitly defined, the words such as setting, installation, connection, etc. should be understood broadly, and the person skilled in the art can determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0070] Currently, GUI (Graphical User Interface) has become the main way for people to interact with various electronic devices, and is widely used in computer operating systems, mobile applications and smart home control panels, and many other fields. With the rapid development of technology, users' expectations for GUI interaction experience are increasing.

[0071] Currently, AIAgent (Artificial Intelligence Agent) based on MCP (Model Context Protocol) or API interface and command line has made certain progress in automatic task execution. However, when facing office software operations (such as Excel multi-table data association, Word complex layout) and web dynamic interaction (such as form submission with pop-up verification), it lacks dynamic execution strategy, resulting in a high failure rate of element operation execution.

[0072] In order to solve the above technical defects, the embodiments of the present application provide a GUI element operation method, system, device and storage medium.

[0073] Please refer to Figure 1 It is a flowchart of a GUI element operation method provided by the embodiments of the present application, which is applied to an electronic device, which can be a server, etc. As Figure 1 The GUI element operation method comprises:

[0074] Step S101, in the case of receiving a GUI element operation command, determining the to-be-operated GUI element in the GUI element operation command, the scene category of the to-be-operated GUI element and the operation type of the to-be-operated GUI element;

[0075] Step S102, determining all first GUI elements of the interface where the to-be-operated GUI element is located, the first bounding box of each first GUI element and the first element confidence of each first GUI element;

[0076] Step S103, determining the second bounding box of the to-be-operated GUI element based on the scene category;

[0077] Step S104, in the case that the first element confidence of each first GUI element is greater than a preset confidence threshold, calculating the intersection over union of each first GUI element based on the first bounding box and the second bounding box;

[0078] Step S105, screening out the first GUI element with the intersection over union greater than a preset intersection over union threshold as the second GUI element;

[0079] Step S106, in the case where the number of the second GUI elements is one, determining an environment stability value of the GUI element to be operated, wherein the environment stability value is a constant value used to represent whether a pop-up window appears or not in the case where the GUI element operation command is executed;

[0080] Step S107, determining a confidence score based on the operation type, the first element confidence of the second GUI element and the environment stability value.

[0081] Step S108, in the case where the confidence score is greater than a preset confidence score threshold, ending the GUI element operation command.

[0082] The GUI element can include but is not limited to a button, an input box, a file upload control, text, an icon, a pop-up window and a prompt box.

[0083] The scene category can be at least one of a webpage scene and a desktop application scene.

[0084] The operation type can be at least one of clicking, dragging and inputting.

[0085] The preset confidence threshold can be a constant value set in advance according to actual needs, and can be 0.7.

[0086] The preset IoU threshold can be a constant value set in advance according to actual needs.

[0087] The preset confidence score threshold can be a constant value set in advance according to actual needs. It can be 0.8.

[0088] In step S101, in the case where the GUI element operation command is received, the GUI element to be operated, the scene category of the GUI element to be operated and the operation type of the GUI element to be operated in the GUI element operation command can be extracted by natural language technology in the case where the user input GUI element operation command is received.

[0089] In step S104, the IoU of each first GUI element can be calculated based on the first bounding box and the second bounding box. The coordinates of the first bounding box of each first GUI element and the coordinates of the second bounding box of the GUI element to be operated are obtained. The coordinates of the first bounding box are converted into screen absolute coordinates. The IoU of the first bounding box of each first GUI element and the second bounding box of the GUI element to be operated is calculated based on the screen absolute coordinates of the first bounding box of each first GUI element and the coordinates of the second bounding box of the GUI element to be operated, and is taken as the IoU of each first GUI element.

[0090] In the step S106, the environment stability value of the GUI element to be operated can be 0.5 when the GUI element operation command is executed and a pop-up window appears, and can be 1 when the GUI element operation command is executed and no pop-up window appears.

[0091] In some embodiments, when the number of the second GUI elements is not one, a GUI element is selected from the second GUI elements by manual operation for the subsequent calculation.

[0092] The method determines the GUI element to be operated, the scene category of the GUI element to be operated, and the operation type of the GUI element to be operated in the received GUI element operation command; determines all first GUI elements of the interface where the GUI element to be operated is located, the first bounding box of each first GUI element, and the first element confidence of each first GUI element; determines the second bounding box of the GUI element to be operated based on the scene category; calculates the intersection over union of each first GUI element based on the first bounding box and the second bounding box when the first element confidence of each first GUI element is greater than a preset confidence threshold; screens out the first GUI element with the intersection over union greater than a preset intersection over union threshold as the second GUI element; determines the environment stability value of the GUI element to be operated when the number of the second GUI elements is one, wherein the environment stability value is a constant value for representing whether a pop-up window appears when the GUI element operation command is executed; determines the confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value; and ends the GUI element operation command when the confidence score is greater than a preset confidence score threshold. The application can realize cross-scene and cross-operation type positioning based on the determination of the second bounding box of the GUI element to be operated based on the scene category, improve the positioning accuracy, further improve the element positioning accuracy based on the calculation of the intersection over union of each first GUI element based on the first bounding box and the second bounding box, and realize the dynamic execution strategy combined with the confidence score to improve the execution success rate of the GUI element operation when the confidence score is greater than the preset confidence score threshold.

[0093] In some embodiments, before the step S102, the steps S201 to S203 can be included, but are not limited to.

[0094] The step S201 captures the interface where the GUI element to be operated is located to obtain a captured image.

[0095] The step S202 performs grayscale processing on the captured image to obtain a grayscale processed image.

[0096] Step S203, edge enhancement is performed on the gray image to obtain an edge-enhanced image.

[0097] Step S102 can include, but is not limited to, step S204:

[0098] Step S204, inputting the edge-enhanced image into the trained YOLOv8 model to obtain all first GUI elements, a first bounding box of each first GUI element and a first element confidence of each first GUI element output by the trained YOLOv8 model.

[0099] The present application determines all first GUI elements, a first bounding box of each first GUI element and a first element confidence of each first GUI element, which provides data basis for subsequent element positioning and improves the accuracy of element positioning.

[0100] In some embodiments, the method can further include, but is not limited to, steps S301 to S304:

[0101] Step S301, in the case where the first element confidence of the plurality of first GUI elements is less than or equal to a preset confidence threshold, the plurality of first GUI elements are taken as second GUI elements;

[0102] Step S302, magnifying the image area of the first bounding box of all second GUI elements by a preset magnification value to obtain an enlarged image area of each second GUI element;

[0103] Step S303, inputting the enlarged image area into the trained YOLOv8 model to obtain a third bounding box of each second GUI element and a first updated element confidence output by the trained YOLOv8 model;

[0104] Step S304, updating the first bounding box of each second GUI element based on the third bounding box; updating the first element confidence of each second GUI element based on the first updated element confidence.

[0105] In the above step S304, updating the first bounding box of each second GUI element based on the third bounding box; updating the first element confidence of each second GUI element based on the first updated element confidence can replace the third bounding box of each second GUI element with the corresponding first bounding box; and replace the first updated element confidence of each second GUI element with the corresponding first element confidence.

[0106] The present application improves the accuracy of element positioning by re-detecting the first GUI elements with low confidence.

[0107] In some embodiments, step S107 can include but is not limited to steps S401-S408:

[0108] Step S401, in the case of the scene category being a webpage scene, the node identifier and the coordinates of the GUI element to be operated are obtained by the DOM tree method, and in the case that both the node identifier and the coordinates exist, the second element confidence is a first preset constant value; in the case of the scene category being a desktop application scene, the control state of the GUI element to be operated is obtained by the UI hierarchy collection method, and the second element confidence is determined based on the matching of the control state and a preset control state rule, wherein the preset control state rule includes the value of the second element confidence corresponding to each control state, and the control state includes the control type;

[0109] Step S402, calculating a third element confidence based on the first element confidence and the second element confidence of the second GUI element;

[0110] Step S403, determining an operation complexity value corresponding to the operation type, wherein the operation complexity value is a scalar value representing the complexity of the operation type;

[0111] Step S404, multiplying the operation complexity value by a first preset weight value to obtain a first product value;

[0112] Step S405, multiplying the third element confidence by a second preset weight value to obtain a second product value;

[0113] Step S406, multiplying the environment stability value by a third preset weight value to obtain a third product value;

[0114] Step S407, calculating the sum of the first product value, the second product value, and the third product value;

[0115] Step S408, taking the sum as the confidence score.

[0116] The first preset constant value is a constant value set in advance according to actual needs.

[0117] In step S403, the operation complexity value corresponding to the operation type can be 1 when the operation type is clicking, 2 when the operation type is dragging, and 1.5 when the operation type is inputting.

[0118] The first preset weight value is a constant value set in advance according to actual needs, and can be 0.4.

[0119] The second preset weight value is a constant value set in advance according to actual needs, and can be 0.3.

[0120] The third preset weight value is a constant value set in advance according to actual needs, and can be 0.3.

[0121] The sum of the first preset weight value, the second preset weight value and the third preset weight value is 1.

[0122] The application determines the confidence score for the operation type, the element confidence and the environment stability value, and gives different weight values to the operation type, the element confidence and the environment stability value, thereby improving the accuracy of the confidence score calculation and improving the execution success rate of the GUI element operation.

[0123] In some embodiments, step S402 can include but is not limited to steps S501-S503:

[0124] Step S501, multiplying the first element confidence of the second GUI element by the first preset confidence weight value to obtain a first confidence product value;

[0125] Step S502, multiplying the second element confidence by the second preset confidence weight value to obtain a second confidence product value;

[0126] Step S503, adding the first confidence product value and the second confidence product value to obtain a third element confidence.

[0127] The first preset confidence weight value is a constant value set in advance according to actual needs.

[0128] The second preset confidence weight value is a constant value set in advance according to actual needs.

[0129] The sum of the first preset confidence weight value and the second preset confidence weight value is 1.

[0130] The application determines the third element confidence for the first element confidence and the second element confidence, and gives different weight values to the first element confidence and the second element confidence, thereby improving the accuracy of the third element confidence calculation and providing data basis for subsequent confidence score calculation.

[0131] In some embodiments, step S103 can include but is not limited to steps S601-S602:

[0132] Step S601, in the case where the scene category is a webpage scene, obtaining the second bounding box of the GUI element to be operated by a DOM tree method;

[0133] Step S602, in the case where the scene category is a desktop application scene, obtaining the second bounding box of the GUI element to be operated by a UI hierarchy collection method.

[0134] The application improves the accuracy of element positioning by providing different positioning methods for different scene categories.

[0135] In some embodiments, the method can further include, but is not limited to, steps S701 to S704:

[0136] Step S701, obtaining an execution result of a GUI element operation command in a case where the confidence score is less than or equal to a preset confidence score threshold, wherein the execution result is execution success or execution failure;

[0137] Step S702, determining a failure cause of the GUI element to be operated in a case where the execution result is execution failure;

[0138] Step S703, matching the failure cause with a preset recovery rule database to obtain a recovery level and a recovery strategy;

[0139] Step S704, determining an execution mode corresponding to the recovery level, and executing the recovery strategy based on the execution mode, wherein the execution mode includes automatic execution and manual execution.

[0140] The above failure cause can include, but is not limited to, GUI element disablement and network delay.

[0141] The above recovery level can include, but is not limited to, first-level recovery and second-level recovery.

[0142] The above recovery strategy can include, but is not limited to, merging cells and re-parsing element coordinates.

[0143] In the above step S702, the determination of the failure cause of the GUI element to be operated can be determined through system feedback or manual checking.

[0144] In the above step S703, the matching of the failure cause with the preset recovery rule database to obtain the recovery level and the recovery strategy can be inputting the failure cause into the preset recovery rule database for matching to obtain the recovery level and the recovery strategy.

[0145] In the above step S704, the determination of the execution mode corresponding to the recovery level and the execution of the recovery strategy based on the execution mode can be that in a case where the recovery level is first-level recovery, the corresponding execution mode is automatic execution, and the system executes the above recovery strategy through automatic execution; in a case where the recovery level is second-level recovery, the corresponding execution mode is manual execution, and the system executes the above recovery strategy through manual execution.

[0146] The application processes different abnormal operations through different recovery levels and recovery strategies, improving the accuracy and efficiency of abnormal operation processing.

[0147] In addition, with reference to Figure 2 In one embodiment of the present application, a GUI element operation system is provided, which comprises a data acquisition module 1100, a first bounding box and first element confidence determination module 1200, a second bounding box determination module 1300, an intersection over union calculation module 1400, an element screening module 1500, an environment stability value determination module 1600, a confidence score determination module 1700, and an end module 1800, wherein:

[0148] The data acquisition module 1100 is configured to determine a GUI element to be operated, a scene category of the GUI element to be operated, and an operation type of the GUI element to be operated in a GUI element operation command received;

[0149] The first bounding box and first element confidence determination module 1200 is configured to determine all first GUI elements in an interface where the GUI element to be operated is located, a first bounding box of each first GUI element, and a first element confidence of each first GUI element;

[0150] The second bounding box determination module 1300 is configured to determine a second bounding box of the GUI element to be operated based on the scene category;

[0151] The intersection over union calculation module 1400 is configured to calculate an intersection over union of each first GUI element based on the first bounding box and the second bounding box when the first element confidence of each first GUI element is greater than a preset confidence threshold value;

[0152] The element screening module 1500 is configured to screen out, as second GUI elements, the first GUI elements whose intersection over union is greater than a preset intersection over union threshold value;

[0153] The environment stability value determination module 1600 is configured to determine an environment stability value of the GUI element to be operated when the number of the second GUI elements is one, wherein the environment stability value is a constant value used to represent whether a pop-up window appears when the GUI element operation command is executed;

[0154] The confidence score determination module 1700 is configured to determine a confidence score based on the operation type, the first element confidence of the second GUI elements, and the environment stability value;

[0155] The end module 1800 is configured to end the GUI element operation command when the confidence score is greater than a preset confidence score threshold value.

[0156] The system determines the GUI element to be operated, the scene category of the GUI element to be operated, and the operation type of the GUI element to be operated in the GUI element operation command in a case where the GUI element operation command is received; determines all first GUI elements of an interface where the GUI element to be operated is located, a first bounding box of each first GUI element, and a first element confidence of each first GUI element; determines a second bounding box of the GUI element to be operated based on the scene category; calculates an intersection over union of each first GUI element based on the first bounding box and the second bounding box in a case where the first element confidence of each first GUI element is greater than a preset confidence threshold; screens out the first GUI element with the intersection over union greater than a preset intersection over union threshold as a second GUI element; determines an environment stability value of the GUI element to be operated in a case where the number of the second GUI elements is one, wherein the environment stability value is a constant value used to represent whether a pop-up window appears in a case where the GUI element operation command is executed; determines a confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value; ends the GUI element operation command in a case where the confidence score is greater than a preset confidence score threshold. The system can realize positioning across scenes and across operation types based on the determination of the second bounding box of the GUI element to be operated based on the scene category, thereby improving positioning accuracy. The system further calculates the intersection over union of each first GUI element based on the first bounding box and the second bounding box, screens out the first GUI element with the intersection over union greater than the preset intersection over union threshold, and again improves the accuracy of element positioning. Finally, the system ends the GUI element operation command in a case where the confidence score is greater than the preset confidence score threshold, thereby realizing a dynamic execution strategy combined with the confidence score and improving the execution success rate of the GUI element operation.

[0157] It should be noted that the system embodiments and the above-mentioned method embodiments are based on the same inventive concept, and therefore the related content of the above-mentioned method embodiments is also applicable to the system embodiments, which will not be described here again.

[0158] Figure 3 A hardware structure schematic diagram of the GUI element operation provided by the embodiments of the present application is shown.

[0159] The GUI element operation device can include a processor 301 and a memory 302 having computer program instructions stored therein.

[0160] Specifically, the above-mentioned processor 301 can include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC), or can be configured as one or more integrated circuits that implement the embodiments of the present application.

[0161] The memory 302 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 302 can include a Hard Disk Drive (HDD), a floppy disk drive, a flash memory, a Compact Disc (CD) or other optical disk, a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 302 can include removable or non-removable (or fixed) media, where appropriate. Where appropriate, the memory 302 can be internal or external to the integrated gateway disaster recovery appliance. In particular embodiments, the memory 302 is non-volatile, solid-state memory.

[0162] In some embodiments, the memory 302 can include read-only memory (ROM), random-access memory (RAM), a disk storage medium, an optical storage medium, a flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to

[0163] The processor 301 implements the GUI element operation method of any of the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0164] In one example, the GUI element operation device can further include a communication interface 303 and a bus 310. Wherein, as shown in the figure, the processor 301, the memory 302, the communication interface 303 are connected through the bus 310 and complete the communication between each other. Figure 3

[0165] The communication interface 303 is mainly used to realize the communication between each module, device, unit and / or equipment in the embodiments of the application.

[0166] ​Bus 310 includes hardware, software, or both, to couple components of GUI element manipulation device to each other in the system. While Figure 1 provides an example of a bus, it is to be understood that other bus architectures can be used in conjunction with the various embodiments. For example, an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these can be used in the system. Where appropriate, bus 310 can include one or more buses. Although the present embodiments describe and show a particular bus, the present embodiments contemplate any suitable bus or interconnect.

[0167] The GUI element manipulation device can perform the GUI element manipulation method in the present embodiments based on a three-dimensional design model, thereby realizing the GUI element manipulation method and system described in the present embodiments. Figure 1 and Figure 2 the GUI element manipulation method and system described in the present embodiments.

[0168] In addition, in combination with the GUI element manipulation method in the above embodiments, the present embodiments can provide a computer storage medium to realize. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to realize any one of the GUI element manipulation methods in the above embodiments.

[0169] It needs to be clear that the present embodiments are not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present embodiments are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present embodiments.

[0170] The functional blocks shown in the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium, or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0171] It is also important to note that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the steps mentioned above, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0172] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing devices to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing devices to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0173] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A GUI element operation method, characterized by, The GUI element operation method comprises: In the case of receiving a GUI element operation command, determining a GUI element to be operated in the GUI element operation command, a scene category of the GUI element to be operated, and an operation type of the GUI element to be operated; Determining all first GUI elements of an interface in which the GUI element to be operated is located, a first bounding box of each of the first GUI elements, and a first element confidence of each of the first GUI elements; Determining a second bounding box of the GUI element to be operated based on the scene category; In the case that the first element confidence of each of the first GUI elements is greater than a preset confidence threshold, calculating an intersection over union of each of the first GUI elements based on the first bounding box and the second bounding box; Screening out a first GUI element with an intersection over union greater than a preset intersection over union threshold as a second GUI element; In the case that the number of the second GUI elements is one, determining an environment stability value of the GUI element to be operated, wherein the environment stability value is a constant value used to represent whether a pop-up window appears in the case of executing the GUI element operation command; Determining a confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value; In the case that the confidence score is greater than a preset confidence score threshold, ending the GUI element operation command.

2. The method of claim 1, wherein, Before the determination of all first GUI elements of an interface in which the GUI element to be operated is located, a first bounding box of each of the first GUI elements, and a first element confidence of each of the first GUI elements, the method comprises: Taking a screenshot of the interface in which the GUI element to be operated is located to obtain a post-screenshot image; Grayscale processing the post-screenshot image to obtain a post-gray-scale image; Edge enhancement is performed on the post-gray-scale image to obtain a post-edge-enhancement image; The determination of all first GUI elements of an interface in which the GUI element to be operated is located, a first bounding box of each of the first GUI elements, and a first element confidence of each of the first GUI elements comprises: Inputting the post-edge-enhancement image into a trained YOLOv8 model to obtain all the first GUI elements, the first bounding box of each of the first GUI elements, and the first element confidence of each of the first GUI elements output by the trained YOLOv8 model.

3. The method of claim 2, wherein, The method further comprises: In the case that there are a plurality of first GUI elements with a first element confidence less than or equal to a preset confidence threshold, the plurality of first GUI elements are taken as second GUI elements; Magnifying the image area of the first bounding box of all the second GUI elements by a preset magnification value to obtain a post-magnification image area of each of the second GUI elements; Inputting the post-magnification image area into the trained YOLOv8 model to obtain a third bounding box and a first updated element confidence of each of the second GUI elements output by the trained YOLOv8 model. updating the first bounding box of each of the second GUI elements based on the third bounding box; updating a first element confidence of each of the second GUI elements based on the first updated element confidence.

4. The method of claim 1, wherein, The scene category is at least one of a webpage scene and a desktop application scene, and the determining a confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value comprises: In a case where the scene category is the webpage scene, a node identifier and a coordinate of the GUI element to be operated are obtained by a DOM tree method, and in a case where both the node identifier and the coordinate exist, a second element confidence is a first preset constant value; in a case where the scene category is the desktop application scene, a control state of the GUI element to be operated is obtained by a UI hierarchy collection method, and the second element confidence is determined based on matching of the control state and a preset control state rule, wherein the preset control state rule comprises a value of the second element confidence corresponding to each control state, and the control state comprises a control type; calculating a third element confidence based on the first element confidence of the second GUI element and the second element confidence; determining an operation complexity value corresponding to the operation type, wherein the operation complexity value is a scalar value representing complexity of the operation type; multiplying the operation complexity value by a first preset weight value to obtain a first product value; multiplying the third element confidence by a second preset weight value to obtain a second product value; multiplying the environment stability value by a third preset weight value to obtain a third product value; calculating a sum of the first product value, the second product value, and the third product value; taking the sum as the confidence score.

5. The method of claim 4, wherein, The calculating a third element confidence based on the first element confidence of the second GUI element and the second element confidence comprises: multiplying the first element confidence of the second GUI element by a first preset confidence weight value to obtain a first confidence product value; multiplying the second element confidence by a second preset confidence weight value to obtain a second confidence product value; adding the first confidence product value and the second confidence product value to obtain the third element confidence.

6. The method of claim 5, wherein, The determining the second bounding box of the GUI element to be operated based on the scene category comprises: in a case where the scene category is the webpage scene, obtaining the second bounding box of the GUI element to be operated by a DOM tree method; in a case where the scene category is the desktop application scene, obtaining the second bounding box of the GUI element to be operated by a UI hierarchy collection method.

7. The method of claim 1, wherein, The method further comprises: in a case where the confidence score is less than or equal to the preset confidence score threshold, obtaining an execution result of the GUI element operation command, wherein the execution result is execution success or execution failure; in a case where the execution result is execution failure, determining a failure cause of the GUI element to be operated; matching the failure cause with a preset recovery rule database to obtain a recovery level and a recovery strategy; determine an execution manner corresponding to the recovery level, and execute the recovery strategy based on the execution manner, wherein the execution manner comprises automatic execution and manual execution.

8. A GUI element operating system, characterized by The GUI element operation system comprises: a data acquisition module configured to determine, when a GUI element operation command is received, a GUI element to be operated in the GUI element operation command, a scene category of the GUI element to be operated, and an operation type of the GUI element to be operated; a first bounding box and first element confidence determination module configured to determine all first GUI elements in an interface in which the GUI element to be operated is located, a first bounding box of each of the first GUI elements, and a first element confidence of each of the first GUI elements; a second bounding box determination module configured to determine a second bounding box of the GUI element to be operated based on the scene category; an intersection over union calculation module configured to, when the first element confidence of each of the first GUI elements is greater than a preset confidence threshold, calculate an intersection over union of each of the first GUI elements based on the first bounding box and the second bounding box; an element screening module configured to screen, as a second GUI element, a first GUI element whose intersection over union is greater than a preset intersection over union threshold; an environment stability value determination module configured to, when the number of the second GUI elements is one, determine an environment stability value of the GUI element to be operated, wherein the environment stability value is a constant value used to represent whether a pop-up window appears when the GUI element operation command is executed; a confidence score determination module configured to determine a confidence score based on the operation type, the first element confidence of the second GUI element, and the environment stability value; an ending module configured to, when the confidence score is greater than a preset confidence score threshold, end the GUI element operation command.

9. A GUI element operation device characterized by comprising: The computer readable storage medium stores computer executable instructions for causing a computer to execute a GUI element operation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions for causing a computer to execute a GUI element operation method according to any one of claims 1 to 7.

Images