Popup window control detection method and device, equipment and storage medium

Abstract

The invention provides a popup window control detection method and device, equipment and a storage medium, and the method comprises the steps: obtaining a view tree corresponding to a current interface in an automatic task execution process; determining a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used for indicating whether the current interface comprises a pop-up window control or not; and when the popup window detection result indicates that the current interface comprises the popup window control, closing the popup window control so as to continue to execute the automatic task. By implementing the method, the pop-up window can be detected and processed in time, and the problem that automatic task execution fails due to display of the pop-up window control is effectively solved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method, device, equipment and storage medium for detecting a pop-up control. Background Art

[0002] At present, most applications still rely on touch interaction of the human-computer interface. For some high-frequency application usage scenarios, there will be problems such as too long interaction process and too low interaction efficiency.

[0003] To solve this problem, for high-frequency application usage scenarios, electronic devices will automatically repeat the previous user's operation steps without user intervention. However, many applications will pop up various pop-up windows from time to time during use, resulting in the task encountering inconsistent interfaces and paths during the automatic execution process with the user's manual execution process, and the user's operation process cannot be reproduced, resulting in task execution failure. Summary of the invention

[0004] The present application provides a pop-up control detection method, device, equipment and storage medium, which can realize timely detection and processing of pop-up windows, and effectively solve the problem of failure of automated task execution due to the display of pop-up controls.

[0005] A first aspect of an embodiment of the present application provides a method for detecting a pop-up window control, the method being applied to an electronic device, comprising:

[0006] During the execution of the automated task, obtain the view tree corresponding to the current interface;

[0007] Determine a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control;

[0008] When the pop-up window detection result indicates that the current interface includes the pop-up window control, close the pop-up window control to continue executing the automated task.

[0009] A second aspect of an embodiment of the present application provides a pop-up window control detection device, the device is on an electronic device, and the device includes:

[0010] The acquisition unit is used to obtain the view tree corresponding to the current interface during the execution of the automation task;

[0011] A detection unit, used to determine a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control;

[0012] A processing unit is used to close the pop-up window control when the pop-up window detection result indicates that the current interface includes the pop-up window control, so as to continue to execute the automated task.

[0013] A third aspect of the embodiments of the present application provides an electronic device,

[0014] A memory storing executable program code;

[0015] and a processor coupled to the memory;

[0016] The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method disclosed in the first aspect of the embodiment of the present application.

[0017] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having executable program code stored thereon. When the executable program code is executed by a processor, the method disclosed in the first aspect of the embodiment of the present application is implemented.

[0018] The fifth aspect of the embodiments of the present application discloses a computer program product. When the computer program product runs on a computer, the computer executes the method disclosed in the first aspect of the embodiments of the present application.

[0019] A sixth aspect of an embodiment of the present application discloses an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes the method disclosed in the first aspect of the embodiment of the present application.

[0020] It can be seen from the above technical solutions that the embodiments of the present application have at least the following advantages:

[0021] During the execution of the automated task, the view tree corresponding to the current interface is obtained; the pop-up detection result is determined based on the view tree, and the pop-up detection result is used to indicate whether the current interface includes a pop-up control; when the pop-up detection result indicates that the current interface includes a pop-up control, the pop-up control is closed to continue executing the automated task.

[0022] By implementing this method, during the execution of the automated task, it is possible to detect whether the current interface includes a pop-up control based on the view tree corresponding to the current interface, and if the current interface includes a pop-up control, the pop-up control is closed so that the pop-up control on the current interface disappears, so that the automated task can continue to be executed. This achieves timely detection and processing of pop-ups during the execution of automated tasks, and effectively solves the problem of failure of automated task execution due to the display of pop-up controls. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a method for detecting a pop-up window control disclosed in an embodiment of the present application;

[0024] Figure 2A It is an interface diagram disclosed in the embodiment of the present application;

[0025] Figure 2B This is a schematic diagram of interaction between an electronic device and a server disclosed in an embodiment of the present application;

[0026] Figure 2C is another schematic diagram of interaction between an electronic device and a server disclosed in an embodiment of the present application;

[0027] Figure 2D It is a set of pop-up window schematic diagrams disclosed in the embodiments of the present application;

[0028] Figure 2E is another set of pop-up window schematic diagrams disclosed in the embodiments of the present application;

[0029] Figure 3 It is another flowchart of the method for detecting a pop-up window control disclosed in an embodiment of the present application;

[0030] Figure 4 It is another interface diagram disclosed in the embodiment of the present application;

[0031] Figure 5A It is a structural diagram of the view tree disclosed in the embodiment of the present application;

[0032] Figure 5B It is a flowchart illustrating the processing of icon images by the visual classification recognition model disclosed in the embodiment of the present application;

[0033] Figure 6 It is a flowchart illustrating that an electronic device disclosed in an embodiment of the present application detects and processes a pop-up window based on a pop-up window configuration table;

[0034] Figure 7 It is a structural diagram of a detection device for a pop-up window control disclosed in an embodiment of the present application;

[0035] Figure 8 It is a structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application provides a pop-up control detection method, device, equipment and storage medium, which can realize timely detection and processing of pop-up windows, and effectively solve the problem of failure of automated task execution due to the display of pop-up controls.

[0037] In order to make the technical personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiment of the present application will be described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, they should all fall within the scope of protection of the present application.

[0038] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0039] "At least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or plural.

[0040] With the development of software and hardware of electronic devices such as smartphones, tablets, and smart watches in recent years, the functions of applications we can experience are becoming more and more abundant. Through electronic devices, we can complete most of the daily tasks in life. For example, through social media applications and video call applications, we can keep in touch with family, friends, and colleagues anytime and anywhere. For example, through shopping applications, we can browse and purchase goods. For example, through mobile payment applications, we can complete payments safely and conveniently. For example, through video playback applications, we can watch videos.

[0041] However, in practice, it is found that most applications still rely on touch interaction of the human-computer interface. For some high-frequency application scenarios, there will be problems such as too long interaction process and too low interaction efficiency. For example, if a user wants to watch a video that was not finished last time in a video playback application, first, the user needs to manually open the video playback application, and the electronic device will display the home page interface of the video playback application. Then, the user manually clicks the "Personal Center" button on the home page interface, and the electronic device displays the historical viewing record. Finally, the user clicks the window corresponding to the latest viewing record, and the electronic device plays the video that the user did not finish watching last time.

[0042] To solve this problem, for high-frequency application usage scenarios, electronic devices can record the user's operation steps and corresponding applications and interfaces in the application usage scenario. Subsequent users can activate the application usage scenario through voice or other instructions, and the electronic device will automatically repeat the previous user's operation steps in the application usage scenario, automatically completing the same operation steps and processes without user intervention. For example, if a user wants to watch a video that was not finished last time in a video playback application, the user can voice input "continue playing the last video" to activate the application usage scenario, and the electronic device will automatically open the video playback application to display the home page interface of the video playback application, and automatically trigger the "Personal Center" button to display the historical viewing record, and automatically trigger the window corresponding to the latest viewing record to play the video that the user did not finish watching last time. This entire process is completely automatically executed by the electronic device without user operation.

[0043] However, many applications will pop up various pop-up windows from time to time during use, resulting in the task encountering inconsistent interfaces and paths during automatic execution and during manual execution by the user, making it impossible for the electronic device to reproduce the user's operation process as it is, resulting in task execution failure.

[0044] To solve this problem, the embodiments of the present application disclose a method, device, equipment and storage medium for detecting a pop-up window control, which can realize timely detection and processing of pop-up windows, and effectively solve the problem of failure of automated task execution due to the display of pop-up window controls. The method for detecting a pop-up window control disclosed in the embodiments of the present application may include: during the execution of an automated task, it may be possible to detect whether the current interface includes a pop-up window control according to the view tree corresponding to the current interface, and if the current interface includes a pop-up window control, close the pop-up window control so that the pop-up window control on the current interface disappears, so as to continue to execute the automated task, thereby realizing timely detection and processing of pop-ups during the execution of automated tasks, and effectively solving the problem of failure of automated task execution due to the display of pop-up window controls.

[0045] It should be noted that the execution subject of the pop-up control detection method disclosed in the embodiment of the present application can be an electronic device, which can include a general handheld screen electronic device, such as a mobile phone, a smart phone, a portable terminal, a terminal, a personal digital assistant (PDA), a portable multimedia player (Personal Media Player, PMP) device, a laptop computer, a notebook (Note Pad), a wireless broadband (Wireless Broadband, Wibro) terminal, a tablet computer (Personal Computer, PC) and a smart PC, etc.

[0046] Electronic devices may also include wearable devices. Wearable devices are portable electronic devices that can be worn directly on the user or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but can also achieve powerful intelligent functions through software support, data interaction, and cloud server interaction, such as: computing function, positioning function, and alarm function. They can also be connected to mobile phones and various terminals. Wearable devices may include but are not limited to watches supported by the wrist (such as watches, wrists, etc.), shoes supported by the feet (such as shoes, socks, or other products worn on the legs), glasses supported by the head (such as glasses, helmets, headbands, etc.), as well as smart clothing, school bags, crutches, accessories and other non-mainstream product forms.

[0047] The present application scheme is further introduced below in conjunction with specific embodiments.

[0048] See also Figure 1 , Figure 1 FIG. 1 is a flowchart of a method for detecting a pop-up window control disclosed in an embodiment of the present application. Figure 1 The detection method of the pop-up window control shown may include the following steps:

[0049] 101. During the execution of an automated task, the electronic device obtains a view tree corresponding to the current interface.

[0050] Among them, an automation task refers to a task that an electronic device performs on behalf of a user. The electronic device triggers the execution of the automation task upon receiving a target instruction, and different automation tasks correspond to different target instructions.

[0051] The current interface refers to the interface currently displayed on the display screen of the electronic device. The view tree ViewTree corresponding to the current interface is used to describe the layout structure of the current interface. Each node in the ViewTree represents a control in the current interface. The view tree includes some relevant properties of the control, including text description, interactive properties (such as whether it is clickable, whether text can be entered, whether it is slidable, etc.) and control position.

[0052] The following Figure 2A Take the interface shown in the figure as an example to introduce the view tree. Figure 2A The parent control in the interface represents the root node of the view tree, the pop-up control in the interface represents the child nodes of the root node, and the child controls in the pop-up control represent the child nodes of the child nodes of the root node respectively.

[0053] In the embodiment of the present application, the electronic device may obtain the view tree corresponding to the current interface in the following ways, but is not limited to:

[0054] The electronic device can obtain the view tree corresponding to the current interface through the interface provided by the underlying code of the system;

[0055] or,

[0056] The electronic device obtains the view tree corresponding to the current interface through the AccessibilityService related interface.

[0057] 102. The electronic device determines a pop-up window detection result according to the view tree, where the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control.

[0058] In some embodiments, the electronic device determines the pop-up window detection result according to the view tree, which may include: the electronic device determines the pop-up window detection result according to the view tree through a pop-up window recognition model.

[0059] Optionally, the electronic device determines the pop-up window detection result by using a pop-up window recognition model according to the view tree, which may include:

[0060] Method 1: The electronic device converts the view tree into text, and processes the text through a pop-up window recognition model to obtain a pop-up window detection result. The pop-up window recognition model may be a model for processing linear text.

[0061] Method 2: The electronic device processes the view tree through a pop-up window recognition model to obtain a pop-up window detection. The pop-up window recognition model may be a model for processing tree structure data.

[0062] It is understandable that the electronic device can accurately detect the pop-up window control by determining the pop-up window detection result based on the view tree and the pop-up window recognition model.

[0063] Further optionally, the electronic device determines the pop-up window detection result through a pop-up window recognition model according to the view tree, which may include: the electronic device determines the pop-up window detection result through a pop-up window recognition model according to the view tree and the image of the current interface.

[0064] The image of the current interface may be obtained by the electronic device performing a screenshot operation on the current interface.

[0065] Optionally, the electronic device determines the pop-up window detection result by using a pop-up window recognition model according to the view tree and the image of the current interface, which may include:

[0066] The electronic device determines the first pop-up window detection result through the pop-up window recognition model according to the view tree;

[0067] The electronic device determines the second pop-up window detection result through the pop-up window recognition model according to the image of the current interface;

[0068] The electronic device obtains a pop-up window detection result according to the first pop-up window detection result and the second pop-up window detection result.

[0069] Among them, the electronic device determines the first pop-up window detection result through the pop-up window recognition model according to the view tree, which can be referred to the above introduction and will not be repeated here. It should be noted that the pop-up window recognition model can be a multimodal model, which can not only process linear text and / or tree-structured data, but also image data.

[0070] It is understandable that the electronic device can further improve the detection accuracy of the pop-up window control by determining the pop-up window detection result through a pop-up window recognition model based on the view tree and the image of the current interface.

[0071] Optionally, the pop-up window recognition model can be deployed on an electronic device or on a server. When the pop-up window recognition model is deployed on the server, the electronic device can send a view tree to the server, and the server determines a pop-up window detection result based on the view tree by using the pop-up window recognition model, and sends the pop-up window detection result to the electronic device. Alternatively, the electronic device can send a view tree and an image of the current interface to the server, and the server determines a pop-up window detection result based on the view tree and the image of the current interface by using the pop-up window recognition model, and sends the pop-up window detection result to the electronic device.

[0072] It should be noted that the process of the server determining the pop-up window detection result can refer to the process of the electronic device determining the pop-up window detection result, which will not be repeated here.

[0073] For example, when the pop-up window recognition model is deployed on the server, the interaction diagram between the electronic device and the server can be seen in the following Figure 2B and Figure 2C Among them, Figure 2B As shown, the pop-up window recognition model is deployed on the server 10, and the electronic device 20 sends the view tree to the server 10. The server 10 determines the pop-up window detection result through the pop-up window recognition model according to the view tree, and sends the pop-up window detection result to the electronic device 20. Figure 2C As shown, the pop-up window recognition model is deployed on the server 10, and the electronic device 20 sends the view tree and the image of the current interface to the server 10. The server 10 determines the pop-up window detection result through the pop-up window recognition model based on the view tree and the image of the current interface, and sends the pop-up window detection result to the electronic device 20.

[0074] It is understandable that when the pop-up window recognition model is deployed on an electronic device, the electronic device can quickly identify whether the current interface includes a pop-up window, which is conducive to improving the detection efficiency of pop-up windows. When the pop-up window recognition model is deployed on a server, the electronic device only needs to send the view tree, or the view tree and the image of the current interface to the server, and the server detects the pop-up window control through the pop-up window recognition model, which is conducive to reducing the device power consumption of the electronic device.

[0075] In other embodiments, the electronic device may determine the pop-up window detection result according to the size information corresponding to the target control in the current interface. The size information corresponding to the target control may include the proportional relationship between the size of the target control and the size of the display screen of the electronic device. For a specific description of determining the pop-up window detection result according to the size information corresponding to the target control, please refer to the relevant description below, which will not be repeated here.

[0076] 103. When the pop-up window detection result indicates that the current interface includes a pop-up window control, the electronic device closes the pop-up window control to continue executing the automated task.

[0077] It is understandable that after the electronic device closes the pop-up control, the pop-up control is no longer displayed on the current interface.

[0078] In some embodiments, when the pop-up detection result indicates that the current interface includes a pop-up control, the electronic device closes the pop-up control for processing, which may include: when the pop-up detection result indicates that the current interface includes a pop-up control, the electronic device outputs a prompt message, the prompt message is used to instruct the user to close the pop-up control; and, in response to an operation input by the user, closes the pop-up control on the current interface.

[0079] It can be understood that when the pop-up detection result indicates that the current interface includes a pop-up control, the electronic device can promptly prompt the user to close the pop-up control on the current interface by outputting a prompt message, and close the pop-up control on the current interface in response to the user input, which can effectively avoid long-term interruption of the automated task.

[0080] Optionally, the electronic device outputting the prompt information may include: the electronic device outputting the prompt information through a speaker, or the electronic device displaying the prompt information on the current interface.

[0081] Exemplarily, when the electronic device displays a prompt message on the current interface, the prompt message may be text such as “A pop-up window appears, please process it” or “A pop-up window appears”.

[0082] Optionally, when the pop-up window detection result is determined by a pop-up window recognition model, if the pop-up window detection result indicates that the current interface includes a pop-up window control, the pop-up window detection result may also include a pop-up window type and / or a pop-up window processing rule.

[0083] Among them, the pop-up window types may include but are not limited to any of the following: user terms pop-up windows, permission request pop-up windows, application upgrade pop-up windows, application login pop-up windows, advertisement pop-up windows, and application usage guide pop-up windows. For example, the user terms pop-up windows, permission request pop-up windows, and application upgrade pop-up windows can be seen in the following figure. Figure 2D ,in, Figure 2DThe three interfaces shown include, from left to right, a pop-up window for user terms, a pop-up window for permission request, and a pop-up window for application upgrade. For example, the illustrations of the pop-up window for application login, the pop-up window for advertisement, and the pop-up window for application usage guide can be found below. Figure 2E ,in, Figure 2E The three interfaces shown include, from left to right, an application login pop-up window, an advertisement pop-up window, and an application usage guide pop-up window.

[0084] The pop-up window processing rule refers to a strategy for the electronic device to automatically close the pop-up window control. When the pop-up window detection result includes the pop-up window processing rule corresponding to the pop-up window control, the electronic device closing the pop-up window control may include: the electronic device closes the pop-up window control in the current interface according to the instruction of the pop-up window processing rule corresponding to the pop-up window control.

[0085] It can be understood that the above-mentioned pop-up window recognition model can not only identify whether the current interface includes a pop-up window control, but also identify the pop-up window processing rules corresponding to the pop-up window control when the current interface includes a pop-up window control. The electronic device automatically closes the pop-up window control based on the pop-up window processing rules corresponding to the pop-up window control without user intervention, thereby greatly improving the processing efficiency of the pop-up window control.

[0086] By implementing Figure 1 The method shown can, during the execution of the automated task, detect whether the current interface includes a pop-up control based on the view tree corresponding to the current interface, and if the current interface includes a pop-up control, close the pop-up control so that the pop-up control on the current interface disappears to continue executing the automated task. This achieves timely detection and processing of pop-ups during the execution of the automated task, and effectively solves the problem of failure of automated task execution due to the display of pop-up controls.

[0087] See also Figure 3 , Figure 3 FIG. 1 is another flowchart of the pop-up window control detection method disclosed in the embodiment of the present application. Figure 3 The detection method of the pop-up window control shown may include the following steps:

[0088] 301. During the execution of an automated task, the electronic device obtains a view tree corresponding to the current interface.

[0089] It should be noted that, for the introduction of step 301, reference may be made to the above description of step 101, which will not be repeated here.

[0090] 302. The electronic device obtains size information corresponding to a target control in the current interface according to the view tree.

[0091] The target control may be any node except the root node in the view tree. The size information corresponding to the target control may include the proportional relationship between the size of the target control and the size of the display screen of the electronic device.

[0092] Optionally, the size information corresponding to the target control may include a ratio of an area of ​​the target control to an area of ​​the display screen, and / or a ratio of a width of the target control to a width of the display screen.

[0093] 303. The electronic device determines whether the target control is a pop-up control according to size information corresponding to the target control.

[0094] In some embodiments, when the target control is a child node of the root node in the view tree, the electronic device determines whether the target control is a pop-up window control according to size information corresponding to the target control, which may include:

[0095] When the ratio of the area of ​​the target control to the area of ​​the display screen is less than or equal to a first threshold, and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a second threshold, the electronic device determines that the target control is a pop-up control.

[0096] It should be noted that when the ratio of the area of ​​the target control to the area of ​​the display screen is greater than a first threshold, and / or the ratio of the width of the target control to the width of the display screen is less than a second threshold, the electronic device determines that the target control is not a pop-up control.

[0097] In the embodiment of the present application, when the target control is a child node of the root node in the view tree, the target control can be an independent application window.

[0098] The first threshold and the second threshold may be determined based on the corresponding size information of an existing pop-up window control (such as a permission pop-up window) belonging to an independent application window. For example, the first threshold may be 70% and the second threshold may be 60%.

[0099] It can be understood that when the target control is a child node of the root node in the view tree, if the target control is a pop-up control, then the target control is a pop-up control belonging to an independent application window. By implementing this method, the detection of pop-up controls belonging to an independent application window is achieved.

[0100] It should be noted that there are relatively few pop-up controls that belong to independent application windows. Most pop-up controls do not belong to independent application windows, such as user terms pop-ups, application upgrade pop-ups, application login pop-ups, advertising pop-ups, and application usage guide pop-ups. The detection of these pop-up controls is to detect nodes in the view tree that are not child nodes of the root node of the view tree. The specific detection method can be as follows:

[0101] In some embodiments, when the target control is not a child node of the root node of the view tree, the electronic device determines whether the target control is a pop-up window control according to size information corresponding to the target control, which may include:

[0102] When the ratio of the area of ​​the target control to the area of ​​the display screen, and / or the ratio of the width of the target control to the width of the display screen, is greater than or equal to a third threshold, the electronic device determines the coverage ratio of the sub-control of the target control to the target control;

[0103] When the coverage ratio is less than or equal to the ratio threshold, the electronic device determines that the target control is a pop-up window control;

[0104] When the coverage ratio is greater than the ratio threshold, the electronic device determines that the target control is not a pop-up control.

[0105] It should be noted that when the ratio of the area of ​​the target control to the area of ​​the display screen, and / or the ratio of the width of the target control to the width of the display screen, is less than a third threshold, the electronic device determines that the target control is not a pop-up control.

[0106] The third threshold may include a threshold corresponding to the ratio of the area of ​​the target control to the area of ​​the display screen, and / or a threshold corresponding to the ratio of the width of the target control to the width of the display screen.

[0107] It can be understood that when the target control is not a pop-up control, the electronic device uses the sub-control of the target control as the new target control, and performs an operation of detecting whether the ratio of the area of ​​the target control to the area of ​​the display screen, and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a third threshold, until the ratio of the area of ​​the new target control to the area of ​​the display screen, and / or the ratio of the width to the width of the display screen, is greater than or equal to the third threshold.

[0108] In an embodiment of the present application, the electronic device determines the coverage ratio of the target control's sub-controls to the target control, which may include: the electronic device obtains the coverage ratio of each sub-control of the target control to the target control from the view tree, and sums the coverage ratio of each sub-control to the target control to obtain the coverage ratio of all sub-controls of the target control to the target control.

[0109] Among them, the third threshold value can be determined based on the size information corresponding to the existing pop-up window that does not belong to the independent application window. Exemplarily, the threshold corresponding to the area ratio can be 80%, and the threshold corresponding to the width ratio can be 60%. The ratio threshold value can be obtained based on the coverage ratio relationship between the existing pop-up window control that does not belong to the independent application window and its sub-controls. Exemplarily, the ratio threshold value can be 70%.

[0110] It should be noted that, when the target control has a parent control and its parent control is not a child node of the root node, the ratio of the area and / or width of the parent control to the display screen is greater than or equal to the third threshold, and the coverage ratio of the child controls of the parent control to the parent control is greater than the ratio threshold.

[0111] For example, the layout relationship between the target control and its corresponding sub-controls can be seen in the following Figure 4 .like Figure 4 In the interface shown, the target control includes 4 sub-controls, and the coverage range of the target control by these 4 sub-controls is the target area.

[0112] It can be understood that when the target control is not a child node of the root node of the view tree, if the target control is a pop-up control, then the target control is a pop-up control that does not belong to an independent application window. By implementing this method, detection of pop-up controls that do not belong to an independent application window is achieved.

[0113] Optionally, when the target control is a child node of a child node of a root node in the view tree, the above pop-up window detection process may occur when the child node of the root node in the view tree is not a pop-up window control. Figure 5A The view tree shown introduces the detection process of the pop-up control.

[0114] Among them, Figure 5A The view tree shown includes a root node 0, child nodes 1 and 2 of the root node, child nodes 3 and 4 of child node 1, and child nodes 5 and 6 of child node 3. The process of detecting a pop-up window control by an electronic device may be that the electronic device first detects whether the ratio of the area of ​​child nodes 1 and child nodes 2 to the display screen is less than or equal to a first threshold, and / or whether the ratio to the width of the display screen is greater than or equal to a second threshold. If both child nodes 1 and 2 do not meet the requirements, then detect whether the ratio of the area and / or width of child node 3 to the display screen is greater than or equal to a third threshold. If so, continue to detect child nodes 5 and child nodes 6 to see whether the coverage ratio of child node 3 is less than the ratio threshold. If so, determine that child node 3 is a pop-up window control. Otherwise, determine that child node 3 is not a pop-up window control, and the electronic device determines that there is no pop-up window control on the current interface.

[0115] Since pop-up controls usually cover other controls, after the electronic device determines that the target control is a pop-up control, the electronic device can also determine whether the target control covers other controls. If so, the electronic device determines that the target control is a pop-up control; otherwise, it is considered that the target control is not a pop-up control, which can further improve the detection accuracy of the pop-up control.

[0116] 304. When the target control is a pop-up control, the electronic device closes the pop-up control to continue executing the automation task.

[0117] In some embodiments, the electronic device closes the pop-up window control for processing, which may include: the electronic device identifies the pop-up window type of the pop-up window control, and closes the pop-up window control on the current interface according to the pop-up window type of the pop-up window control.

[0118] It is understandable that the electronic device can achieve differentiated management and control of pop-up controls by closing the pop-up controls on the current interface according to the pop-up type of the pop-up controls.

[0119] Optionally, the electronic device closes a pop-up control on the current interface according to the pop-up type of the pop-up control, which may include: the electronic device determines a preset processing rule corresponding to the pop-up type of the pop-up control, and closes the pop-up control on the current interface according to the preset processing rule corresponding to the pop-up type.

[0120] It is understandable that the electronic device is pre-set with corresponding preset processing rules for each type of pop-up control. After identifying the pop-up type of the pop-up control, the electronic device can directly use the preset processing rules corresponding to the pop-up type to automatically close the pop-up control without manual intervention by the user, thereby greatly improving the processing efficiency of the pop-up control.

[0121] Optionally, the electronic device closes a pop-up control on the current interface based on the pop-up type of the pop-up control, which may include: when the pop-up type is a specified type, the electronic device outputs a prompt message, the prompt message is used to prompt the user to close the pop-up control; and, in response to an operation input by the user, closes the pop-up control on the current interface; and, when the pop-up type is not a specified type, the electronic device closes the pop-up control on the current interface according to a preset processing rule corresponding to the pop-up type.

[0122] The designated type may be a pop-up window type involving key information settings that is preset by the user. For example, the designated type may be "user terms type", "authority request type", "application upgrade type" and "application login type".

[0123] It can be understood that when the pop-up type is of a specified type, the electronic device outputs a prompt message, and then based on the user's input operation, closes the pop-up control on the current interface, which can prevent the user from missing the setting of some key information. When the pop-up control is not of a specified type, the electronic device automatically closes the pop-up control from the current interface according to the preset processing rules corresponding to the pop-up type, and the processing efficiency is faster. It can be seen that this pop-up control processing method can ensure processing efficiency while preventing users from missing the setting of some key information, thereby realizing reasonable and effective processing of pop-up controls.

[0124] In an embodiment of the present application, the electronic device closes the pop-up control on the current interface according to the preset processing rule corresponding to the pop-up type, which may include: the electronic device determines the target button indicated by the preset processing rule corresponding to the pop-up type; and triggers a touch event for the target button to close the pop-up control on the current interface.

[0125] Among them, the touch event triggered by the electronic device on the target button may be visible to the user or may not be visible, which is not limited in the embodiments of the present application.

[0126] In some embodiments, the electronic device identifying the pop-up type of a pop-up control may include: the electronic device identifying the pop-up type of the pop-up control based on the text description and / or icon image of the sub-control of the pop-up control, thereby achieving accurate identification of the pop-up type.

[0127] The electronic device can obtain the text description of the sub-control of the pop-up control from the view tree.

[0128] The electronic device can obtain the icon image of the sub-control of the pop-up control from the view tree and / or the image of the current interface.

[0129] The electronic device identifies the pop-up type of a pop-up control based on the text description of the sub-control of the pop-up control, which may include: the electronic device matches the text description of the sub-control of the pop-up control with a text feature expression corresponding to a preset pop-up type to obtain the pop-up type of the pop-up control.

[0130] The electronic device identifies the pop-up type of a pop-up control based on the icon image of the sub-control of the pop-up control, which may include: the electronic device determines the icon type of the sub-control of the pop-up control based on the icon image of the sub-control of the pop-up control, and obtains the pop-up type of the pop-up control based on the icon type of the sub-control of the pop-up control.

[0131] It should be noted that the sub-controls that require icon type identification mainly refer to sub-controls that lack text descriptions.

[0132] Optionally, the electronic device may identify the icon image of the sub-control of the pop-up window control through a visual classification recognition model to obtain the icon type of the sub-control of the pop-up window control.

[0133] Further optionally, the pixel points of the icon image input into the visual classification recognition model may be lower than the pixel point threshold, which is beneficial to reduce the amount of calculation and improve calculation efficiency.

[0134] In an embodiment of the present application, the training data of the visual classification recognition model can be obtained and annotated from the screenshot data of the graphical user interface (refer to the RICO data set), and after the training is completed, the model is converted into a tflite model. Among them, the visual classification recognition model can be a lightweight neural network model, and exemplarily, the visual classification recognition model can be MobileNetV3.

[0135] For example, the flowchart of the visual classification recognition model processing icon images can be found in the following Figure 5B .like Figure 5B As shown in the figure, the visual classification recognition model first processes the icon image (including grayscale, normalization, translation and whitening), and then inputs the processed image into the convolution layer, and the icon type is obtained after the convolution layer processing. It should be noted that Figure 5B The number of convolutional layers in the visual classification recognition model shown is only exemplary and not limiting.

[0136] In some embodiments, the electronic device may be preset with a pop-up window configuration table, which may include multiple preset pop-up window types, and the identification method and processing method corresponding to each preset pop-up window type, and the electronic device may implement type identification and processing of the pop-up window control based on the pop-up window configuration table. Among them, the processing method in the pop-up window configuration table is used to indicate that the pop-up window is closed by user intervention or automatically closed by the electronic device.

[0137] It should be noted that the pop-up configuration table can be continuously updated based on user needs.

[0138] In the embodiment of the present application, the process of the electronic device detecting and processing the pop-up window based on the pop-up window configuration table can be seen as follows Figure 6 ,in, Figure 6 The following steps are involved:

[0139] 610. The electronic device obtains a view tree corresponding to the current interface.

[0140] 611. The electronic device obtains size information corresponding to a target control in the current interface according to the view tree, and determines whether the target control is a pop-up control according to the size information corresponding to the target control.

[0141] If the target control is not a pop-up window control, step 612 is executed; if the target control is a pop-up window control, step 613 is executed.

[0142] 612. The electronic device prompts a pop-up window indicating that detection failed.

[0143] 613. The electronic device sequentially reads configuration items from the pop-up configuration table.

[0144] 614. The electronic device calls the corresponding identification interface according to the pop-up window identification method in the current configuration item to identify the target control;

[0145] If the recognition fails, continue to execute step 613, and if the recognition succeeds, execute step 615.

[0146] 615. The electronic device calls the corresponding processing interface according to the pop-up window processing method in the current configuration item to process the target control.

[0147] If the processing is successful, execute step 616; if the processing fails, execute step 617.

[0148] 616. The electronic device prompts that the pop-up control has been processed successfully.

[0149] 617. The electronic device prompts that the pop-up window processing failed.

[0150] It should be noted that if all configuration items are not successfully identified, step 618 is executed.

[0151] 618. The electronic device prompts that the pop-up control recognition failed.

[0152] For example, the pop-up window configuration table is shown in Table 1:

[0153] Table 1

[0154]

[0155] Among them, "-" means not configured, indicating that this configuration is not effective; the recognition method of the pop-up window Match means extracting the text description of the sub-control of the pop-up window control from the interface view tree, and matching the given text feature expression from the text description of each sub-control; "[]" means one or more given feature expressions, and multiple expressions are separated by ","; multiple expressions need to be matched one by one to be considered a match; in the expression, "^" means matching the beginning of the text, and "$" means matching the end of the text; "\S" means matching any non-blank character; "*" means matching the previous character 0 times or infinite times; "|" means matching any expression fragment on the left and right; the recognition method of the pop-up window Recognize means extracting the icon image of the sub-control of the pop-up window control based on the interface view tree and the screenshot of the current interface, and identifying the type of the icon through the image classification model; "×" (such as Figure 2D The pop-up window in the middle shows a visual close icon. The Alert in the pop-up window processing method indicates that the user is prompted to process the window, and the parameter indicates the corresponding window type. Tap indicates that the control button corresponding to the parameter is automatically clicked to realize automatic processing of the window.

[0156] By implementing Figure 3The method shown can determine the size information corresponding to the target control according to the view tree corresponding to the current interface during the execution of the automation task, and then determine whether the target control is a pop-up control based on the size information corresponding to the target control, thereby realizing simple and efficient detection of pop-up controls, and closing the pop-up control in time when the target control is a pop-up control, so that the pop-up control on the current interface disappears, so as to continue to execute the automation task, effectively solving the problem of failure of automation task execution due to the display of pop-up controls.

[0157] See also Figure 7 , Figure 7 is a structural diagram of a detection device for a pop-up window control disclosed in an embodiment of the present application. Figure 7 The detection device of the pop-up window control shown may include an acquisition unit 701, a detection unit 702 and a processing unit 703; wherein:

[0158] The acquisition unit 701 is used to acquire the view tree corresponding to the current interface during the execution of the automation task;

[0159] A detection unit 702, configured to determine a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control;

[0160] The processing unit 703 is used to close the pop-up window control when the pop-up window detection result indicates that the current interface includes the pop-up window control, so as to continue to execute the automation task.

[0161] In some embodiments, the detection unit 702 is used to determine the pop-up detection result based on the view tree, which may specifically include: the detection unit 702 is used to obtain the size information corresponding to the target control in the current interface according to the view tree; and determine whether the target control is a pop-up control based on the size information corresponding to the target control.

[0162] In some embodiments, the size information corresponding to the target control includes: a proportional relationship between the size of the target control and the size of the display screen of the electronic device.

[0163] In some embodiments, the target control is a child node of the root node of the view tree, and the detection unit 702 is used to determine whether the target control is a pop-up control based on size information corresponding to the target control. Specifically, the detection unit 702 is used to determine that the target control is a pop-up control when the ratio of the area of ​​the target control to the area of ​​the display screen is less than or equal to a first threshold, and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a second threshold.

[0164] In some embodiments, the target control is not a child node of the root node of the view tree, and the detection unit 702 is used to determine whether the target control is a pop-up control based on the size information corresponding to the target control, which may specifically include: the detection unit 702 is used to determine the coverage ratio of the target control's sub-controls to the target control when the ratio of the area of ​​the target control to the area of ​​the display screen and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a third threshold; and, when the coverage ratio is less than or equal to a ratio threshold, determine that the target control is a pop-up control.

[0165] In some embodiments, the processing unit 703 is configured to close the pop-up window control in a manner that, when the pop-up window detection result indicates that the current interface includes the pop-up window control, may specifically include:

[0166] Processing unit 703 is used to output prompt information when the pop-up window detection result indicates that the current interface includes the pop-up window control, and the prompt information is used to prompt the user to close the pop-up window control; and, in response to the operation input by the user, close the pop-up window control on the current interface.

[0167] In some embodiments, the processing unit 703 is configured to close the pop-up window control in a manner that, when the pop-up window detection result indicates that the current interface includes the pop-up window control, may specifically include:

[0168] The processing unit 703 is used to identify the pop-up type of the pop-up control when the pop-up detection result indicates that the current interface includes the pop-up control; and, according to the pop-up type, close the pop-up control on the current interface.

[0169] In some embodiments, the processing unit 703 is used to close the pop-up control on the current interface according to the pop-up type, which may specifically include: the processing unit 703 is used to determine the preset processing rule corresponding to the pop-up type of the pop-up control; and, according to the preset processing rule corresponding to the pop-up type, closing the pop-up control on the current interface.

[0170] In some embodiments, the processing unit 703 is used to close the pop-up control on the current interface according to the pop-up type, which may specifically include: the processing unit 703 is used to output a prompt message when the pop-up type is a specified type, the prompt message is used to prompt the user to close the pop-up control, and close the pop-up control on the current interface in response to the operation input by the user; and, when the pop-up type is not a specified type, close the pop-up control on the current interface according to a preset processing rule corresponding to the pop-up type.

[0171] In some embodiments, the method in which the processing unit 703 is used to identify the pop-up type of the pop-up control may specifically include: the processing unit 703 is used to identify the pop-up type of the pop-up control based on the text description of the sub-control of the pop-up control and / or the icon image of the sub-control of the pop-up control.

[0172] In some embodiments, the processing unit 703 is used to close the pop-up control on the current interface according to the preset processing rules corresponding to the pop-up type. The specific method may include: the processing unit 703 is used to determine the target button indicated by the preset processing rules corresponding to the pop-up type; trigger a touch event for the target button to close the pop-up control on the current interface.

[0173] In some embodiments, the detection unit 702 is used to determine the pop-up window detection result according to the view tree, which may specifically include: the detection unit 702 is used to determine the pop-up window detection result according to the view tree through a pop-up window recognition model.

[0174] In some embodiments, the detection unit 702 is used to determine the pop-up window detection result through a pop-up window recognition model based on the view tree, and the method may specifically include: the detection unit 702 is used to determine the pop-up window detection result through a pop-up window recognition model based on the view tree and the image of the current interface.

[0175] See also Figure 8 , Figure 8 FIG. 1 is a structural diagram of an electronic device disclosed in an embodiment of the present application. Figure 8 The electronic device shown may include components such as a processor 801 , a memory 802 , a display unit 803 , an input unit 804 , a sensor 805 , and an audio circuit 806 .

[0176] The processor 801 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 802, and calling data stored in the memory 802, so as to monitor the electronic device as a whole. Optionally, the processor 801 may include one or more processing units; optionally, the processor 801 may integrate an application processor, which mainly processes operating devices, user interfaces, and application programs, etc. Of course, other processors may also be included, which are not listed here one by one.

[0177] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 802. The memory 802 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating devices, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0178] The display unit 803 can be used to display information input by the user or information provided to the user and various menus of the electronic device. The display unit 803 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 801 to determine the type of touch event. Then, the processor 801 provides a corresponding visual output on the display panel according to the type of touch event. Among them, the touch panel and the display panel are not in contact with each other. Figure 8 The touch panel and the display panel can be used as two independent components to realize the input and output functions of the electronic device, or the touch panel and the display panel can be integrated to realize the input and output functions of the electronic device.

[0179] The input unit 804 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the input unit 804 may include a touch panel and other input devices. A touch panel, also referred to as a touch screen, can collect user touch operations on or near it (such as operations performed by a user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel), and drive the corresponding connection device according to a pre-set program. In addition, a variety of types such as resistive, capacitive, infrared, and surface acoustic waves can be used to implement the touch panel. In addition to the touch panel, the input unit 804 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of function keys (such as volume control keys, switch keys, etc.), trackballs, joysticks, etc.

[0180] The electronic device may also include at least one sensor 805, such as a magnetometer, a gyroscope sensor, a motion sensor, and other sensors. Specifically, the magnetometer is used to determine the orientation of the electronic device, and the gyroscope sensor can be used to determine the motion posture of the electronic device. It can be used for anti-shake shooting, and can also be used for navigation and somatosensory game scenes. As a type of motion sensor, the acceleration sensor can detect the magnitude of acceleration in all directions, and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the electronic device, such as horizontal and vertical screen switching, related games, magnetometer posture calibration, etc.; as for other sensors that the electronic device can also be configured with, such as pressure gauges, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0181] The audio circuit 806 may include a speaker and a microphone, and may provide an audio interface between a user and an electronic device. The audio circuit 806 may transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 806 and converted into audio data, and then the audio data is output to the processor 801 for processing, and then sent to another device through the video circuit, or the audio data is output to the memory 802 for further processing.

[0182] Although not shown, the electronic device may further include a power supply and a camera. Optionally, the camera may be located at the front or rear of the electronic device, which is not limited in the present embodiment.

[0183] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0184] In the embodiment of the present application, the processor 801 also has the following functions:

[0185] During the execution of the automated task, obtain the view tree corresponding to the current interface;

[0186] Determine a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control;

[0187] When the pop-up window detection result indicates that the current interface includes the pop-up window control, close the pop-up window control to continue executing the automated task.

[0188] In the embodiment of the present application, the processor 801 also has the following functions:

[0189] According to the view tree, obtaining size information corresponding to the target control in the current interface;

[0190] Determine whether the target control is a pop-up window control according to the size information corresponding to the target control.

[0191] In the embodiment of the present application, the size information corresponding to the target control includes: a proportional relationship between the size of the target control and the size of the display screen of the electronic device.

[0192] In the embodiment of the present application, the target control is a child node of the root node of the view tree, and the processor 801 also has the following functions:

[0193] When the ratio of the area of ​​the target control to the area of ​​the display screen is less than or equal to a first threshold, and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a second threshold, the target control is determined to be a pop-up control.

[0194] In the embodiment of the present application, the target control is not a child node of the root node of the view tree, and the processor 801 also has the following functions:

[0195] When the ratio of the area of ​​the target control to the area of ​​the display screen, and / or the ratio of the width of the target control to the width of the display screen, is greater than or equal to a third threshold, the coverage ratio of the sub-control of the target control to the target control is determined; when the coverage ratio is less than or equal to the ratio threshold, the target control is determined to be a pop-up control.

[0196] In the embodiment of the present application, the processor 801 also has the following functions:

[0197] When the pop-up window detection result indicates that the current interface includes the pop-up window control, outputting prompt information, wherein the prompt information is used to prompt the user to close the pop-up window control;

[0198] In response to the operation input by the user, the pop-up window control is closed on the current interface.

[0199] In the embodiment of the present application, the processor 801 also has the following functions:

[0200] When the pop-up window detection result indicates that the current interface includes the pop-up window control, identifying the pop-up window type of the pop-up window control;

[0201] According to the pop-up window type, close the pop-up window control on the current interface.

[0202] In the embodiment of the present application, the processor 801 also has the following functions:

[0203] Determine a preset processing rule corresponding to the pop-up window type of the pop-up window control;

[0204] According to a preset processing rule corresponding to the pop-up window type, the pop-up window control is closed on the current interface.

[0205] In the embodiment of the present application, the processor 801 also has the following functions:

[0206] When the pop-up window type is a specified type, outputting prompt information, the prompt information is used to prompt the user to close the pop-up window control, and in response to the operation input by the user, closing the pop-up window control on the current interface;

[0207] When the pop-up window type is not a specified type, the pop-up window control is closed on the current interface according to a preset processing rule corresponding to the pop-up window type.

[0208] In the embodiment of the present application, the processor 801 also has the following functions:

[0209] The pop-up window type of the pop-up window control is identified according to the text description of the sub-control of the pop-up window control and / or the icon image of the sub-control of the pop-up window control.

[0210] In the embodiment of the present application, the processor 801 also has the following functions:

[0211] Determine the target button indicated by the preset processing rule corresponding to the pop-up window type;

[0212] A touch event for the target button is triggered to close the pop-up window control on the current interface.

[0213] In the embodiment of the present application, the processor 801 also has the following functions:

[0214] According to the view tree, a pop-up window detection result is determined by a pop-up window recognition model.

[0215] In the embodiment of the present application, the processor 801 also has the following functions:

[0216] According to the view tree and the image of the current interface, a pop-up window detection result is determined through a pop-up window recognition model.

[0217] An embodiment of the present application discloses a computer-readable storage medium having executable program code stored thereon. When the executable program code is executed by a processor, the method executed by the electronic device in the embodiment of the present application is implemented.

[0218] The embodiment of the present application discloses a computer program product. When the computer program product is executed on a computer, the computer implements the method executed by the electronic device in the embodiment of the present application.

[0219] An embodiment of the present application discloses an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer implements the method executed by the electronic device in the embodiment of the present application.

[0220] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0221] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0222] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0223] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0225] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0226] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0227] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0228] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0229] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0230] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0231] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0232] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting a pop-up window control, characterized in that: The method is applied to an electronic device, comprising: During the execution of the automated task, obtain the view tree corresponding to the current interface; Determine a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control; When the pop-up window detection result indicates that the current interface includes the pop-up window control, close the pop-up window control to continue executing the automated task.

2. The method according to claim 1, characterized in that The determining the pop-up window detection result according to the view tree includes: According to the view tree, obtaining size information corresponding to the target control in the current interface; Determine whether the target control is a pop-up window control according to the size information corresponding to the target control.

3. The method according to claim 2, characterized in that The size information corresponding to the target control includes: a proportional relationship between the size of the target control and the size of the display screen of the electronic device.

4. The method according to claim 3, characterized in that The target control is a child node of the root node of the view tree, and determining whether the target control is a pop-up control according to size information corresponding to the target control includes: When the ratio of the area of ​​the target control to the area of ​​the display screen is less than or equal to a first threshold, and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a second threshold, the target control is determined to be a pop-up control.

5. The method according to claim 3, characterized in that: The target control is not a child node of the root node of the view tree, and determining whether the target control is a pop-up control according to size information corresponding to the target control includes: When the ratio of the area of ​​the target control to the area of ​​the display screen and / or the ratio of the width of the target control to the width of the display screen is greater than or equal to a third threshold, determining the coverage ratio of the sub-control of the target control to the target control; When the coverage ratio is less than or equal to the ratio threshold, it is determined that the target control is a pop-up window control.

6. The method according to claim 1, characterized in that When the pop-up window detection result indicates that the current interface includes the pop-up window control, closing the pop-up window control: When the pop-up window detection result indicates that the current interface includes the pop-up window control, outputting prompt information, wherein the prompt information is used to prompt the user to close the pop-up window control; In response to the operation input by the user, the pop-up window control is closed on the current interface.

7. The method according to claim 1, characterized in that When the pop-up window detection result indicates that the current interface includes the pop-up window control, closing the pop-up window control includes: When the pop-up window detection result indicates that the current interface includes the pop-up window control, identifying the pop-up window type of the pop-up window control; According to the pop-up window type, close the pop-up window control on the current interface.

8. The method according to claim 7, characterized in that Closing the pop-up window control on the current interface according to the pop-up window type includes: Determine a preset processing rule corresponding to the pop-up window type of the pop-up window control; According to a preset processing rule corresponding to the pop-up window type, the pop-up window control is closed on the current interface.

9. The method according to claim 7, characterized in that: Closing the pop-up window control on the current interface according to the pop-up window type includes: When the pop-up window type is a specified type, outputting prompt information, the prompt information is used to prompt the user to close the pop-up window control, and in response to the operation input by the user, closing the pop-up window control on the current interface; When the pop-up window type is not a specified type, the pop-up window control is closed on the current interface according to a preset processing rule corresponding to the pop-up window type.

10. The method according to any one of claims 7 to 9, characterized in that: The identifying the pop-up window type of the pop-up window control includes: The pop-up window type of the pop-up window control is identified according to the text description of the sub-control of the pop-up window control and / or the icon image of the sub-control of the pop-up window control.

11. The method according to claim 8 or 9, characterized in that: The step of closing the pop-up window control on the current interface according to a preset processing rule corresponding to the pop-up window type includes: Determine the target button indicated by the preset processing rule corresponding to the pop-up window type; A touch event for the target button is triggered to close the pop-up window control on the current interface.

12. The method according to claim 1, characterized in that The determining the pop-up window detection result according to the view tree includes: According to the view tree, a pop-up window detection result is determined by a pop-up window recognition model.

13. The method according to claim 12, characterized in that Determining the pop-up window detection result by using a pop-up window recognition model according to the view tree includes: According to the view tree and the image of the current interface, a pop-up window detection result is determined through a pop-up window recognition model.

14. A detection device for a pop-up window control, characterized in that: The device is on an electronic device, and comprises: The acquisition unit is used to obtain the view tree corresponding to the current interface during the execution of the automation task; A detection unit, used to determine a pop-up window detection result according to the view tree, wherein the pop-up window detection result is used to indicate whether the current interface includes a pop-up window control; A processing unit is used to close the pop-up window control when the pop-up window detection result indicates that the current interface includes the pop-up window control, so as to continue to execute the automated task.

15. An electronic device, characterized in that: include: A memory storing executable program code; and a processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having executable program code stored thereon, characterized in that: When the executable program code is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images