Method, system and equipment for judging and processing non-standard state of GUI (Graphical User Interface) page and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a GUI page non-standard state judgment processing method, system, device and medium, the method comprises the following steps: page waiting processing, which is used for processing a page which can be changed into a standard page after waiting, therefore, information of a page which can be changed into a standard page without waiting is obtained; and the pop-up window processing is used for processing the condition that a pop-up window exists in the page so as to ensure that information of the page without the pop-up window is obtained. According to the method, detection of various non-standard page states can be taken into consideration, and corresponding processing methods can be provided for different page states, so that the method is more comprehensive, more flexible and more adaptive to the requirements of automatic agency.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to a method for determining a non-standard state of a GUI page, and in particular to a method for determining a non-standard state of a GUI page applied to an Android mobile terminal. Background Art

[0002] With the development of large language model technology, automated agents are receiving more and more attention and research. The mechanism of agent operation is to obtain page information at each step (which can be structured text, page screenshots, or a combination of the two), combine the previous action history and the user's task instructions, and make a new action prediction, and repeat this cycle until the task is completed. Since the page information data learned by the agent in the training phase is collected when the page is loaded and there is no pop-up window (called a standard page), the page information obtained during actual reasoning must also be guaranteed to be a standard page. The accuracy of the agent will be greatly reduced on non-standard pages.

[0003] As far as the existing technical solutions are concerned, there is no relatively complete set of judgment and processing methods that can take into account various non-standard pages. The existing similar solution is Chinese patent CN118550824A, which discloses a mobile application GUI pop-up window detection method based only on visual information. The patent is based on the Rico dataset and the target detection model YOLOv8, and trains a visual model to predict the position of pop-up window controls in the page. The limitations of this method are: 1. It only focuses on the non-standard page state of "pop-up window" and does not consider "page loading" and "page jump". 2. Its definition of "pop-up window" is consistent with the "Modal" type control in the Rico dataset, that is, "at the top layer of the page structure, but the size does not cover the entire screen", but the scenarios that it cannot handle include but are not limited to the following two situations: (1) Controls that meet the Modal type in the Rico dataset, but the interactive operations with it are within the scope of the Agent's capabilities; (2) Pop-ups that do not meet the Modal control type in the Rico dataset (such as pop-ups that occupy the full screen and do not appear at the top layer of the page structure, etc.). 3. After identifying the "pop-up window", no further processing solution can be given (such as identifying the location to close the pop-up window, or determining whether the current pop-up window must be handed over to human users for operation, etc.).

[0004] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new method for judging and processing the non-standard state of the GUI page of the Android mobile terminal application. Summary of the invention

[0005] In order to overcome the defects of the prior art, the present invention proposes a method, system, device and medium for judging and processing the non-standard state of a GUI page, which can take into account the detection of multiple non-standard page states and can provide corresponding processing methods for different page states, so as to be more comprehensive, flexible and more adaptable to the needs of automated agents.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for determining a non-standard state of a GUI page, comprising the following steps:

[0008] Page waiting processing, which is used to process pages that need to wait before becoming standard pages, so as to ensure that information of a page that does not need to wait before becoming a standard page is obtained;

[0009] Pop-up window processing is used to handle the situation where there are pop-up windows in the page to ensure that the information of a page without pop-up windows is obtained.

[0010] Preferably, the page waiting process is: cyclically obtaining the XML and screenshot of the GUI page, and judging whether waiting is no longer necessary based on the XML and the screenshot respectively; if the judgment conclusion is that waiting is still required, entering the next loop until the judgment conclusion is that waiting is no longer necessary or the number of loops or the time consumed exceeds a preset threshold.

[0011] Preferably, the page waiting for processing specifically includes:

[0012] 11) Get XML and screenshots of GUI pages;

[0013] 12) Get the XML of the GUI page again;

[0014] 13) Determine whether the difference between the total number of nodes in the XML obtained twice is less than D, and determine whether the position attributes of the first k nodes in the XML obtained twice are completely consistent. If the difference is less than D and the position attributes are completely consistent, the XML obtained again is used as the XML of the page that does not need to wait before it can be changed to the standard page; if the difference is not less than D and / or the position attributes are not completely consistent, determine whether the total time consumed is greater than or equal to T seconds. If it is greater than or equal to T seconds, the XML obtained again is used as the XML of the page that does not need to wait before it can be changed to the standard page. If it is less than T seconds, return to step 12);

[0015] 14) Take a screenshot of the GUI page again;

[0016] 15) Input the screenshot obtained again into the single-image Wait model and input the screenshots obtained twice adjacently into the dual-image Wait model to respectively determine whether the loading is completed. If the output results of the single-image Wait model and the dual-image Wait model are both loading completed, the screenshot obtained again is used as a screenshot of the page that does not need to be in a waiting state before it can be changed to a standard page; if the output result of the single-image Wait model and / or the dual-image Wait model is that the loading is not completed, determine whether the number of cycles is greater than or equal to n times. If it is greater than or equal to n times, the screenshot obtained again is used as a screenshot of the page that does not need to be in a waiting state before it can be changed to a standard page. If it is less than n times, return to step 14).

[0017] Preferably, D is between 10 and 20, k is between 5 and 10, T is between 1.5 and 2.5, and n is between 5 and 10.

[0018] Preferably, the single-image Wait model and the dual-image Wait model are both trained end-to-end prediction models based on convolutional neural networks.

[0019] Preferably, the pop-up window processing specifically includes:

[0020] 21) Get a screenshot of the GUI page;

[0021] 22) Detect whether there is a pop-up window in the screenshot through the pop-up window detection model. If not, take the screenshot as a screenshot of the page without a pop-up window; if so, determine whether it has been cycled once. If it has been cycled once, leave it to the user to operate. If it has not been cycled once, leave it to the pop-up window closing model and the pop-up window closing model will try to close the pop-up window. If the pop-up window closing model can return to the closed position, the pop-up window will be automatically closed according to the closed position and return to step 21). If the pop-up window closing model cannot return to the closed position, leave it to the user to operate.

[0022] Preferably, the pop-up window detection model and the pop-up window closing model are both trained end-to-end prediction models based on convolutional neural networks.

[0023] In addition, the present invention also provides a system for determining and processing a non-standard state of a GUI page, which is characterized by comprising:

[0024] A page waiting processing module is used to process a page that needs to wait before it can become a standard page, so as to ensure that information of a page that does not need to wait before it can become a standard page is obtained;

[0025] The pop-up window processing module is used to process the situation where there are pop-up windows in the page to ensure that the information of a page without pop-up windows is obtained.

[0026] Furthermore, the present invention also provides a GUI page non-standard state judgment processing device, characterized in that it includes:

[0027] one or more processors;

[0028] A memory for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for determining and processing the non-standard state of a GUI page.

[0030] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method for determining and processing the non-standard state of a GUI page as described above are implemented.

[0031] Compared with the prior art, the method, system, device and medium for determining and processing the non-standard state of a GUI page of the present invention have one or more of the following beneficial technical effects:

[0032] 1. The present invention takes into account the detection of various non-standard page states such as "page loading", "page jumping", and "pop-up window", and can provide corresponding processing methods for different non-standard page states.

[0033] 2. The present invention is more comprehensive, flexible, and more adaptable to the needs of Agent technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the method for determining and processing the non-standard state of a GUI page of the present invention.

[0035] Figure 2 It is a flow chart of the page waiting process of the present invention.

[0036] Figure 3 It is a flow chart of the pop-up window processing of the present invention.

[0037] Figure 4 It is a schematic diagram of the structure of the judgment and processing system of the non-standard state of the GUI page of the present invention. DETAILED DESCRIPTION

[0038] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components described below or illustrated in the following figures. The present invention can have other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used here are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and its variations herein is intended to cover the items and their equivalents and additional items displayed below. Unless otherwise specified or limited, the terms "install", "connect", "support" and "couple" and their variations are widely used and cover direct installation and indirect installation, connection, support and connection. In addition, "connect" and "couple" are not limited to physical or mechanical connections or connections.

[0039] Furthermore, on the first aspect, in the disclosure of the present invention, the orientation or positional relationship indicated by terms such as "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second aspect, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0040] The existing technology for parsing and processing Android GUI (Graphical User Interface) pages is based on standard pages (completely loaded and no pop-up windows appear). However, in the process of actual operation of mobile terminals by users, GUI agents, etc., non-standard page states (such as waiting for loading, page jumps, pop-up windows, etc.) are often encountered. The existing technology cannot make judgments and processes these non-standard pages, so it will fail. The problem to be solved by the present invention is to distinguish these non-standard page states from standard page states and provide a method for processing non-standard pages.

[0041] Figure 1 The flowchart of the method for determining the non-standard state of a GUI page of the present invention is shown. Figure 1 As shown, the method for determining the non-standard state of a GUI page of the present invention includes:

[0042] 1. The page is waiting to be processed.

[0043] Page waiting processing is used to process "page loading", "page jumping" and other pages that need to wait for a while before becoming a standard page, so as to ensure that information of a page that does not need to wait before becoming a standard page is obtained.

[0044] Figure 2 The flowchart of the page waiting process of the present invention is shown. This process can be effective at any time when the application is running on the mobile terminal. After this process, the XML and screenshots obtained can be guaranteed not to be in non-standard page states such as "page loading" and "page jumping".

[0045] In the present invention, the page waiting processing is divided into two parallel processing processes: the XML (Extensible Markup Language) side and the screenshot side. The core idea is to cyclically obtain the XML and screenshot of the GUI page, and judge whether it is no longer necessary to wait based on the XML and screenshot respectively. If the judgment conclusion is that waiting is still required, enter the next loop until the judgment conclusion is that waiting is no longer required or the number of loops or the time consumed exceeds a preset threshold.

[0046] Specifically, Figure 2 As shown, the page waiting for processing specifically includes:

[0047] 1. Get the XML and screenshots of the GUI page.

[0048] The method of obtaining the XML and screenshots of the GUI page is not the focus of the present invention, and can be obtained by any existing technology. For example, a reference implementation method is: using the Accessibility Service provided in the Android development tool, you can easily obtain the Screenshot and Accessibility Node Info, and parse the Accessibility Node Info to get the XML structure of the page.

[0049] After obtaining the XML and screenshots of the GUI page, it is divided into two parallel processing processes: the XML (Extensible Markup Language) side and the screenshot side.

[0050] The processing process on the XML side is as follows:

[0051] 2. Get the XML of the GUI page again.

[0052] You can still use the Accessibility Service provided in the Android development tool to conveniently obtain the Accessibility Node Info. You can parse the Accessibility Node Info to get the XML of the GUI page again.

[0053] 3. Determine whether the difference in the total number of nodes (Node class nodes in the present invention) in the XML obtained twice is less than D, and determine whether the position attributes of the first k nodes (Node class nodes in the present invention) in the XML obtained twice are completely consistent. If the difference is less than D and the position attributes are completely consistent, it means that it is not in a non-standard page state such as "page loading" or "page jump", and the XML obtained again can be used as the XML of the page that does not need to wait before it can become a standard page; if the difference is not less than D and / or the position attributes are not completely consistent, determine whether the total time consumed on the XML side is greater than or equal to T seconds. If it is greater than or equal to T seconds, the XML obtained again is used as the XML of the page that does not need to wait before it can become a standard page. If it is less than T seconds, return to step 2, obtain the XML of the GUI page again, and start the next cycle.

[0054] In the present invention, by traversing the obtained XML file tree, the total number of node class nodes and the difference calculation can be realized. The XML file tree is traversed in pre-order to obtain the first k node class nodes and their bounds attributes (the bounds attribute represents the position pixel value of the element on the screen, which is a position attribute). And, if the total time consumed on this side is greater than or equal to T seconds, the loop is jumped out to avoid an infinite loop.

[0055] Preferably, D is between 10 and 20, k is between 5 and 10, and T is between 1.5 and 2.5.

[0056] The processing process on the screenshot side is as follows:

[0057] 4. Take a screenshot of the GUI page again.

[0058] You can still use the Accessibility Service provided in the Android development tools to easily obtain Screenshots.

[0059] 5. Input the screenshot acquired again into the single-image Wait model and input the screenshots acquired twice adjacently into the dual-image Wait model to respectively determine whether the loading is complete. If the output results of the single-image Wait model and the dual-image Wait model are both loading completed, the screenshot acquired again is used as a screenshot of the page that does not need to wait before it can be changed to a standard page; if the output result of the single-image Wait model and / or the dual-image Wait model is that the loading is not complete, determine whether the number of cycles is greater than or equal to n times (preferably, n is between 5 and 10). If it is greater than or equal to n times, the screenshot acquired again is used as a screenshot of the page that does not need to wait before it can be changed to a standard page. If it is less than n times, return to step 4 and start the next cycle.

[0060] Among them, the single-image Wait model mainly focuses on the following situations: (1) there is an obvious prompt on the page that it is in a loading state; (2) the page is obviously in a transition state between two pages. The dual-image Wait model mainly focuses on situations that the single-image wait model cannot handle, including: (1) no page jump occurs, but some changes in the page have not yet ended; (2) a page jump occurs, but due to various reasons (such as mobile phone performance, good network conditions) the screenshots before and after are relatively complete. Therefore, through the single-image Wait model and the dual-image Wait model, it is possible to realize the judgment of various "page loading", "page jump" and other pages that require a short wait before becoming a standard page.

[0061] In the present invention, the single-image Wait model and the dual-image Wait model are both trained end-to-end prediction models based on convolutional neural networks. In specific use, CNN (convolutional neural network) can be used as the single-image Wait model and the dual-image Wait model, and training data can be collected and manually annotated, and the constructed single-image Wait model and the dual-image Wait model can be trained with the annotated training data to obtain the single-image Wait model and the dual-image Wait model.

[0062] The training data of the single-image Wait model are pages that are clearly marked as being in a loading state after being manually annotated, and pages that are clearly in a transitional state of two-page jumps after being manually annotated. The training data of the dual-image Wait model are pages that have not undergone page jumps after being manually annotated, but some changes within the page have not yet been completed, and pages that have undergone page jumps after being manually annotated, but the screenshots before and after are relatively complete due to various reasons.

[0063] 2. Pop-up window processing.

[0064] Pop-up processing is used to handle the situation where there are pop-up windows on the page to ensure that the information of a page without pop-up windows is obtained. In other words, pop-up processing is responsible for handling the situation where there are "pop-up windows" on the page. Here, "pop-up windows" mainly refer to unexpected interactive windows that appear on the operation path. The appearance of this window is uncertain and interferes with and hinders the normal completion of the original task.

[0065] Figure 3 FIG. 4 shows a flowchart of the pop-up window processing of the present invention. Figure 3 As shown, the pop-up window processing specifically includes:

[0066] 1. Take a screenshot of the GUI page.

[0067] You can still use the Accessibility Service provided in the Android development tools to easily obtain Screenshots.

[0068] 2. Detect whether there is a pop-up window in the screenshot through the pop-up window detection model. If not, take the screenshot as a screenshot of the page without a pop-up window; if so, determine whether it has been cycled once (that is, whether it has been handed over to the pop-up window closing model for processing once). If it has been cycled once, it is handed over to the user for operation. If it has not been cycled once, it is handed over to the pop-up window closing model, and the pop-up window closing model tries to close the pop-up window. If the pop-up window closing model can return to the closed position, the pop-up window is automatically closed according to the closed position and returns to step 1. If the pop-up window closing model cannot return to the closed position, it is handed over to the user for operation.

[0069] The pop-up window detection model and the pop-up window closing model are also trained end-to-end prediction models based on convolutional neural networks. The pop-up window detection model is used to determine whether there is a pop-up window in the screenshot, and the pop-up window closing model is used to determine whether there is a click position in the screenshot that can close the pop-up window, and if so, predict its position.

[0070] In specific use, CNN (convolutional neural network) can be used as the pop-up window detection model and pop-up window closing model, and training data can be collected and manually labeled. The constructed pop-up window detection model and pop-up window closing model can be trained with the labeled training data to obtain the pop-up window detection model and pop-up window closing model.

[0071] The training data of the pop-up window detection model are manually annotated pages with pop-up windows and their labels are "pop-up windows exist". The training data of the pop-up window closing model are manually annotated pages with pop-up windows and their labels are "pixel coordinates of the close button in the pop-up window".

[0072] In addition, in the present invention, when the user operates, the user can choose to manually close the pop-up window existing in the page, or manually confirm that there is no pop-up window on the current page, thereby ending the process. In the case of manually confirming that there is no pop-up window on the current page, the screenshot obtained again can be directly used as the screenshot of the page without the pop-up window; in the case of choosing to manually close the pop-up window existing in the page, return to step 1 again to obtain a screenshot again, and use the screenshot obtained again as the screenshot of the page without the pop-up window.

[0073] Figure 4 The schematic diagram of the structure of the judgment and processing system of the non-standard state of the GUI page of the present invention is shown. Figure 4 As shown, the judgment and processing system of the non-standard state of a GUI page of the present invention includes:

[0074] A page waiting processing module is used to process a page that needs to wait before it can become a standard page, so as to ensure that information of a page that does not need to wait before it can become a standard page is obtained;

[0075] The pop-up window processing module is used to process the situation where there are pop-up windows in the page to ensure that the information of a page without pop-up windows is obtained.

[0076] In addition, the present invention also provides a GUI page non-standard state judgment and processing device, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the GUI page non-standard state judgment and processing method as described above.

[0077] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for determining and processing the non-standard state of a GUI page as described above.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for determining a non-standard state of a GUI page, characterized in that: The following steps are involved: Page waiting processing, which is used to process pages that need to wait before becoming standard pages, so as to ensure that information of a page that does not need to wait before becoming a standard page is obtained; Pop-up window processing is used to handle the situation where there are pop-up windows in the page to ensure that the information of a page without pop-up windows is obtained.

2. The method for determining a non-standard state of a GUI page according to claim 1, characterized in that: The page waiting process is: cyclically obtaining the XML and screenshot of the GUI page, and judging whether waiting is no longer necessary based on the XML and the screenshot respectively; if the judgment conclusion is that waiting is still necessary, entering the next loop until the judgment conclusion is that waiting is no longer necessary or the number of loops or the time consumed exceeds a preset threshold.

3. The method for determining a non-standard state of a GUI page according to claim 2, wherein: The page waiting for processing specifically includes: 11) Get XML and screenshots of GUI pages; 12) Get the XML of the GUI page again; 13) Determine whether the difference between the total number of nodes in the XML obtained twice is less than D, and determine whether the position attributes of the first k nodes in the XML obtained twice are completely consistent. If the difference is less than D and the position attributes are completely consistent, the XML obtained again is used as the XML of the page that does not need to wait before it can be changed to the standard page; if the difference is not less than D and / or the position attributes are not completely consistent, determine whether the total time consumed is greater than or equal to T seconds. If it is greater than or equal to T seconds, the XML obtained again is used as the XML of the page that does not need to wait before it can be changed to the standard page. If it is less than T seconds, return to step 12); 14) Take a screenshot of the GUI page again; 15) Input the screenshot obtained again into the single-image Wait model and input the screenshots obtained twice adjacently into the dual-image Wait model to respectively determine whether the loading is completed. If the output results of the single-image Wait model and the dual-image Wait model are both loading completed, the screenshot obtained again is used as a screenshot of the page that does not need to be in a waiting state before it can be changed to a standard page; if the output result of the single-image Wait model and / or the dual-image Wait model is that the loading is not completed, determine whether the number of cycles is greater than or equal to n times. If it is greater than or equal to n times, the screenshot obtained again is used as a screenshot of the page that does not need to be in a waiting state before it can be changed to a standard page. If it is less than n times, return to step 14).

4. The method for determining a non-standard state of a GUI page according to claim 3, wherein: D is between 10 and 20, k is between 5 and 10, T is between 1.5 and 2.5, and n is between 5 and 10.

5. The method for determining a non-standard state of a GUI page according to claim 4, characterized in that: The single-image Wait model and the dual-image Wait model are both trained end-to-end prediction models based on convolutional neural networks.

6. The method for determining a non-standard state of a GUI page according to any one of claims 1 to 5, characterized in that: The pop-up window processing specifically includes: 21) Get a screenshot of the GUI page; 22) Detect whether there is a pop-up window in the screenshot through the pop-up window detection model. If not, take the screenshot as a screenshot of the page without a pop-up window; if so, determine whether it has been cycled once. If it has been cycled once, leave it to the user to operate. If it has not been cycled once, leave it to the pop-up window closing model and the pop-up window closing model will try to close the pop-up window. If the pop-up window closing model can return to the closed position, the pop-up window will be automatically closed according to the closed position and return to step 21). If the pop-up window closing model cannot return to the closed position, leave it to the user to operate.

7. The method for determining a non-standard state of a GUI page according to claim 6, wherein: The pop-up window detection model and the pop-up window closing model are both trained end-to-end prediction models based on convolutional neural networks.

8. A system for determining and processing a non-standard state of a GUI page, characterized in that: include: A page waiting processing module is used to process a page that needs to wait before it can become a standard page, so as to ensure that information of a page that does not need to wait before it can become a standard page is obtained; The pop-up window processing module is used to process the situation where there are pop-up windows in the page to ensure that the information of a page without pop-up windows is obtained.

9. A device for determining a non-standard state of a GUI page, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining a non-standard state of a GUI page as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for determining the non-standard state of a GUI page as described in any one of claims 1 to 7 are implemented.

Images