Popup window processing method and device and storage medium

Abstract

The embodiment of the invention provides a pop-up window processing method and device and a storage medium. In the embodiment of the invention, based on a collaborative architecture of two AI models, pop-up window processing is divided into two stages of potential pop-up window area analysis and pop-up window feature recognition, a potential pop-up window area in a screen image is analyzed by using a first AI model, and a sub-image of the potential pop-up window area is cut out from the screen image; and performing multi-dimensional feature analysis and popup window feature identification based on the sub-images by using the second AI model, positioning position information of closing options according to the multi-dimensional feature analysis and the popup window feature identification, executing popup window closing operation based on the position information, cutting out the sub-images of potential popup window areas, converting global analysis into local analysis, reducing calculation overhead of irrelevant areas, and on the basis, improving the popup window closing efficiency. Model reasoning is used for analyzing the cut sub-images, the efficiency and accuracy of pop-up window detection and closing are effectively improved, then the probability that automatic testing is interrupted due to pop-up window interference is reduced, and the testing stability and continuity are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a pop-up window handling method, device and storage medium. Background Technology

[0002] During automated testing of an application, test scripts are executed to simulate user actions such as clicking, typing, and swiping, thereby verifying the application's various functions. While executing the test scripts, the operating system or other applications may pop up various windows (collectively referred to as "pop-ups"), interrupting the execution of the test scripts. These pop-ups need to be manually closed, which can severely impact the stability and efficiency of automated testing.

[0003] With the development of AI technology, AI models can detect and close pop-ups during testing. However, current AI-based pop-up recognition involves full-domain analysis of the entire screenshot. Pop-ups are usually located in specific areas of the screen in terms of user interaction logic. Analyzing the entire screenshot generates a lot of unnecessary computational overhead and processing delay. Especially in scenarios with limited computing resources, such as mobile devices, the large computational overhead and processing delay can seriously reduce the efficiency of automated testing and even cause automated testing to be interrupted. Summary of the Invention

[0004] This application provides a pop-up window processing method, device, and storage medium to improve the efficiency of pop-up window detection and closing, and reduce the probability of automated testing being interrupted due to pop-up window interference.

[0005] This application provides a pop-up window handling method, comprising: during the operation of an automated testing tool on a target device, if a pop-up window appears on the screen of the target device, capturing an image of the target device's screen; based on the screen image and the screen parameter information of the target device, calling a first AI model for pop-up window region analysis to analyze the potential pop-up window region of the screen image; based on the potential pop-up window region, cropping a sub-image corresponding to the pop-up window from the screen image; based on the sub-image and the pop-up window's visual features, calling a second AI model for pop-up window feature recognition to identify pop-up window feature information based on the sub-image, and locating the position information of the close option in the pop-up window based on the pop-up window feature information; and based on the potential pop-up window region, the position information, and the screen parameter information, calling the automated testing tool to operate the close option to close the pop-up window.

[0006] This application also provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor is coupled to the memory to execute the computer program to implement the steps in the various methods provided in this application.

[0007] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps in the methods described above.

[0008] This application also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, enable the processor to implement the steps described in the above method embodiments.

[0009] In this embodiment, based on a collaborative architecture of two AI models, pop-up processing is divided into two stages: potential pop-up region analysis and pop-up feature recognition. The first AI model analyzes the potential pop-up region in the screen image and crops out a sub-image of the potential pop-up region from the screen image. The second AI model performs multi-dimensional feature analysis based on the sub-image to identify pop-up features and locate the position information of the close option accordingly. Based on the position information, the pop-up is closed. By cropping out the sub-image of the potential pop-up region, global analysis is transformed into local analysis, reducing the computational overhead of irrelevant areas. On this basis, model inference is used to analyze the cropped sub-image, which effectively improves the efficiency and accuracy of pop-up detection and closing, thereby reducing the probability of automated testing being interrupted due to pop-up interference and improving the stability and continuity of testing. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a pop-up window handling method provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating a pop-up window handling method provided as another exemplary embodiment of this application; Figure 3a A schematic diagram of the structure of a screen image provided as an exemplary embodiment of this application; Figure 3b A schematic diagram of the structure of a sub-image provided for an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0013] Additionally, it should be noted that when user interaction operations or triggering operations are involved in the embodiments of this application, these operations include, but are not limited to, various interaction methods such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations. Touch operations include, but are not limited to, click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight-line swipes and curved-line swipes.

[0014] Furthermore, it should be noted that, in the cases where the embodiments of this application involve jumping between the first interface and the second interface, the jumping methods involved in the embodiments of this application include, but are not limited to: jumping directly from the first interface to the second interface, or jumping from the first interface to the task interface and completing the corresponding task operation on the task interface before jumping to the second interface; completing the corresponding task operation on the task interface includes, but is not limited to: completing the game operation on the game interface when the task interface is implemented as a game interface; completing identity authentication on the identity authentication interface when the task interface is implemented as an identity authentication interface; completing the recharge operation on the recharge interface when the task interface is implemented as a recharge interface; and so on.

[0015] During automated testing of mobile applications, system-level or third-party applications frequently display various system-level pop-ups. These pop-ups are independent of the application under test, interrupting the current application interface and requiring a response from the user (or test script). This severely impacts the stability and efficiency of automated testing. Therefore, to reduce the interference of pop-ups on testing, the industry mainly adopts the following two solutions: Solution A1: Environment Control and Configuration: Manually disable notification permissions for all third-party applications that may interfere with the test on the test device, such as the option for automatic system updates, to prevent most pop-ups.

[0016] Solution A2: UI Element Detection and Closing in Automated Testing Frameworks: Some mobile automation frameworks provide the ability to find and manipulate UI elements. Therefore, by finding characteristic elements of a known pop-up (such as its title, button text, or XPath), you can simulate clicking buttons like "OK," "Allow," or "Will Remind You Later" to close the pop-up.

[0017] The above solutions have certain drawbacks. For example, while solution A1 can block some pop-ups, it cannot block all of them, and some pop-ups will still need to be responded to during the testing process. Solution A2 uses the UI element detection capabilities of the automated testing framework to identify pop-ups, but this requires pre-analysis of potential pop-up characteristics and embedding detection code into the test script to achieve timed detection and handling of pop-ups. However, the appearance of pop-ups is random, and embedding detection code into the test script code can complicate the code logic. In addition, timed polling detection can also affect the performance of the application itself.

[0018] To address the aforementioned technical issues, this application embodiment utilizes a collaborative architecture of two AI models to divide pop-up processing into two stages: potential pop-up region analysis and pop-up recognition. The first AI model analyzes potential pop-up regions in the screen image, cropping out sub-images of these regions. The second AI model performs multi-dimensional feature analysis on these sub-images to identify pop-up features and locate the position of the close option. Based on this position information, the pop-up is closed. By cropping out sub-images of potential pop-up regions, global analysis is transformed into local analysis, reducing computational overhead in irrelevant areas. Furthermore, model inference is used to analyze the cropped sub-images, effectively improving the efficiency and accuracy of pop-up detection and closure. This reduces the probability of automated testing being interrupted by pop-up interference, enhancing test stability and continuity.

[0019] Furthermore, the first AI model focuses on analyzing potential pop-up window regions, providing high-quality image input for the second AI model. Through this design, the filtering mechanism of the first AI model and the pop-up window feature recognition of the second AI model form the first layer of synergy. The first AI model can effectively filter out information irrelevant to the pop-up window, providing a more focused input for the second stage and improving the overall performance of pop-up window recognition. This is particularly beneficial in resource-constrained environments such as mobile devices, significantly optimizing the efficiency of computing resource utilization.

[0020] Furthermore, in an optional embodiment of this application, when the second AI model performs pop-up feature recognition, it actively queries an external prior knowledge base to assist in the recognition and location of the pop-up. This process forms a second layer of collaboration, namely, cooperation between the second AI model and the prior knowledge base. By working closely with the prior knowledge base, the accuracy and reliability of pop-up recognition are further improved.

[0021] Furthermore, in an optional embodiment of this application, if the second AI model fails to successfully identify the pop-up window or the click operation fails, a retry mechanism will be triggered. During this process, the mechanism will call the first AI model again, and the first AI model will dynamically adjust the boundary range of the potential pop-up window area to obtain a new potential pop-up window area. Then, the adjusted potential pop-up window area will be re-cropped to serve as a new input for the second AI model. Through the combination of the dynamic adjustment of the first AI model and the retry mechanism, when the pop-up window recognition fails or closing fails, the adaptive adjustment of the potential area can be automatically executed, enabling the recognition process to have an adaptive adjustment capability driven by the feedback information of the second AI model. This forms a third layer of collaboration between the two-stage models, ensuring that the pop-up window recognition can cover all possible areas and minimizing missed detections.

[0022] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a pop-up window processing method provided for an exemplary embodiment of this application. Figure 1 As shown, the method includes: S101: If a pop-up window appears on the screen of the target device during the operation of the automated testing tool on the target device, capture the screen image of the target device. S102: Based on the screen image and the screen parameter information of the target device, call the first AI model for potential pop-up area analysis to analyze the potential pop-up area of ​​the screen image; S103: Based on the potential pop-up window area, crop the sub-image corresponding to the pop-up window from the screen image; S104: Based on the visual features of the sub-image and the pop-up window, call the second AI model for pop-up window feature recognition, identify pop-up window feature information based on the sub-image, and locate the position information of the close option in the pop-up window based on the pop-up window feature information; S105: Based on the potential pop-up area, location information, and screen parameter information, invoke the automated testing tool to operate the close option to close the pop-up.

[0024] In this embodiment, the target device can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, and there is no limitation thereto.

[0025] In this embodiment, the screen parameter information of the target device is obtained in advance. The screen parameter information includes, but is not limited to, the physical pixel height and width of the screen, the screen orientation (landscape or portrait), and the resolution.

[0026] Furthermore, if a pop-up window appears on the target device's screen while the automated testing tool is running, a screenshot of the target device's screen is captured. In this embodiment, the automated testing tool refers to a set of software tools that support automated testing for specific test types.

[0027] In this embodiment, the method of detecting a pop-up window on the screen is not limited. For example, it can be triggered based on the return value during the test process, such as when an assertion fails, a timeout occurs, there is no interactive response, or an exception code is returned during the test, then a pop-up window is considered to have been detected on the screen. Another example is... Figure 2 As shown, the device management tool can automatically capture screen images of the target device at preset time intervals. This device management tool can be implemented as adb (Android Debug Bridge, a command-line tool for Android devices) on the Android platform (a Linux-based mobile device operating system), or as a third-party tool such as tidevice for the iOS platform (Apple's mobile device operating system). Screen images are captured by calling the APIs (application program interfaces) provided by these tools or executing specific commands. In this case, the device management tool does not detect whether a pop-up window appears; it simply automatically captures the screen image when the preset time arrives, thus continuing with the subsequent steps of this embodiment. The preset time interval is a configurable parameter and can be flexibly set according to different test scenarios.

[0028] In this embodiment, an AI model is introduced to identify and process pop-up windows. The AI ​​model refers to a multimodal large-scale model based on deep learning, possessing a large number of parameters and powerful computing capabilities. This multimodal large-scale model combines a visual encoder and a large language model, enabling efficient recognition, analysis, and representation of image and video content. It can have tens of millions or even billions of parameters, primarily based on neural networks, especially the Transformer architecture. This model can process multimodal data, such as combinations of images and text, achieving more advanced understanding and generation capabilities. Optionally, the multimodal large-scale model can be, but is not limited to, a pre-trained large-scale visual model (such as a Transformer-based model), enabling these pre-trained models to recognize pop-up content in screen images for specific personalized needs. The core of this technology lies in enabling the pre-trained model to learn to focus on specific user-defined objects or scenes; in this embodiment, this mainly refers to pop-up feature recognition.

[0029] Furthermore, such as Figure 2 As shown, based on the screen image and the screen parameter information of the target device, a first AI model for potential pop-up window region analysis is invoked to analyze the potential pop-up window regions of the screen image. The first AI model can be used to analyze regions from different areas of the screen image where the probability of pop-up windows appearing meets preset conditions. These preset conditions could be, for example, the highest probability of pop-up windows appearing, but are not limited to this.

[0030] Furthermore, such as Figure 2 As shown, a sub-image corresponding to the pop-up window is cropped from the screen image based on the potential pop-up window region. Optionally, the sub-image cropping can be achieved by calling the APIs provided by these tools or executing specific commands. Another optional example utilizes image processing techniques to crop the sub-image of the potential pop-up window region. By cropping the screen image, the amount of data that needs to be processed for subsequent pop-up window recognition is greatly reduced, interference from irrelevant background information is decreased, and recognition efficiency is improved.

[0031] Furthermore, such as Figure 2 As shown, after obtaining the cropped sub-image, a second AI model for pop-up recognition is invoked based on the visual features of the sub-image and the pop-up. This second AI model extracts pop-up-related feature information from the sub-image, also known as pop-up feature information, to distinguish pop-ups from ordinary interface elements.

[0032] Among them, the visual features of pop-ups are the visual characteristics that distinguish pop-ups from ordinary interface elements, including but not limited to: layout and structural features, text content features, and contextual differences. These features are described in detail below.

[0033] Layout and structural features: Pop-ups usually have relatively independent bounding boxes, which include areas such as title area, prompt content area and interactive area. The spatial distribution and hierarchical relationship of these areas have identifiable regularity.

[0034] Text content characteristics: Pop-ups often contain title bar text, prompt text, and interactive option text. The interactive option text includes phrases such as "OK", "Cancel", "Allow", or "Will remind you later".

[0035] Contextual differences: Pop-ups are overlaid on the content displayed on the screen image. Their display characteristics are reflected in the differences in the hierarchical relationship and the content occlusion relationship between them and the content displayed on the screen image.

[0036] Optionally, the second AI model learns multi-dimensional features to represent the visual characteristics of pop-ups by pre-training on a large amount of labeled data (including images of pop-ups and non-pop-ups). This pre-training process enables the model to quickly identify common structures and layouts of pop-ups, significantly improving its recognition speed and accuracy. In another optional embodiment, when the second AI model is invoked, the visual features of the pop-ups are provided as prompts. These prompts can be dynamically adjusted according to the actual task, giving the model greater flexibility, especially when facing unlabeled data or new types of pop-ups. By adjusting the prompts in real time, the model can quickly adapt to changes and accurately identify specific pop-ups.

[0037] In this embodiment, the second AI model locates the position of the close option within the pop-up window based on pop-up feature information. Pop-up feature information reflects the pop-up window's layout structure, text features, and contextual differences. Based on this multi-dimensional pop-up feature information, the second AI model can infer the semantics and function of the pop-up window's interactive options, identify the close option, and locate its position.

[0038] Furthermore, such as Figure 2 As shown, based on the potential pop-up area, the location information of the close option, and screen parameter information, the automated testing tool is invoked to operate the close option to close the pop-up. In this embodiment, the location information of the potential pop-up area and the close button is provided to the automated testing tool, and combined with the screen parameter information, the close option is accurately mapped, thereby executing the close operation in a timely and accurate manner, realizing the automatic closure of the pop-up, reducing interference in the automated testing process, and improving the stability and efficiency of automated testing.

[0039] In this embodiment, based on a collaborative architecture of two AI models, pop-up processing is divided into two stages: potential pop-up region analysis and pop-up feature recognition. The first AI model analyzes the potential pop-up region in the screen image and crops out a sub-image of the potential pop-up region from the screen image. The second AI model performs multi-dimensional feature analysis based on the sub-image to identify pop-up features and locate the position information of the close option accordingly. Based on the position information, the pop-up is closed. By cropping out the sub-image of the potential pop-up region, global analysis is transformed into local analysis, reducing the computational overhead of irrelevant areas. On this basis, model inference is used to analyze the cropped sub-image, which effectively improves the efficiency and accuracy of pop-up detection and closing, thereby reducing the probability of automated testing being interrupted due to pop-up interference and improving the stability and continuity of testing.

[0040] Pop-ups are typically located in the central area of ​​the screen in terms of user interaction logic. When processing pop-ups, performing full-area analysis on the entire screen image directly will generate a lot of unnecessary computational overhead, especially in scenarios with limited computing resources, such as mobile devices.

[0041] To address this, this application employs an efficient filtering mechanism for the first AI model, utilizing a heuristic filtering algorithm. This algorithm analyzes screen parameters such as screen size, resolution, and orientation to automatically identify and focus on the central area of ​​the screen, designating it as the potential pop-up window region. A sub-image of this region is then cropped and used as input for the subsequent pop-up feature recognition by the second AI model. This reduces the area processed by the second AI model, avoiding the significant overhead of full-image analysis. Through this design, the filtering mechanism of the first AI model and the pop-up feature recognition of the second AI model form a first layer of synergy. The first AI model can effectively filter out information irrelevant to the pop-up, providing a more focused input for the second stage and improving the overall performance of pop-up recognition. This is particularly beneficial in resource-constrained environments such as mobile devices, significantly optimizing the efficiency of computational resource utilization. This first layer of synergy will be described in detail below.

[0042] In one optional embodiment, the first AI model presets a screen centering prior condition, which is used to constrain potential pop-up areas to be screen centering areas first.

[0043] In this embodiment, the screen centering prior condition can be provided to the first AI model as a prompt word; or, it can be learned during the training process of the first AI model and transformed into a part of the first AI model, thus being embedded inside the first AI model, eliminating the need to combine the prompt word from the outside each time it is called, thereby improving the convenience and efficiency of calling the model.

[0044] Furthermore, in this optional embodiment, a preset prompt word is embedded in the first AI model as an example, but it is not limited to this.

[0045] In this optional embodiment, when calling the first AI model to analyze the potential pop-up window area of ​​the screen image based on the screen image and the screen parameter information of the target device, the process includes: calling the first AI model based on the screen image and the screen parameter information of the target device, calculating the geometric center of the screen image based on the screen parameter information under the guidance of the screen centering prior condition, and determining the boundary range of the screen centering area based on the calculated geometric center to obtain the potential pop-up window area.

[0046] In this embodiment, the physical pixel height and width, screen orientation (landscape or portrait), resolution, and other information of the target device's screen are first obtained. Then, guided by the screen centering prior condition, the first AI model calculates the geometric center point of the screen image based on the screen size information, and uses the geometric center point as a reference to determine the boundary range of the centering area.

[0047] Optionally, the first AI model learns the distribution characteristics of a large number of pop-up window samples during the model training phase. Therefore, during the inference phase, it can evaluate the saliency of the central region based on the geometric center, combined with screen image features (such as borders, alignment, local textures, etc.), and dynamically determine the boundary range accordingly. In this way, the potential pop-up window region is determined to be closer to the position of the pop-up window in the actual screen image, providing a data foundation for subsequent accurate cropping.

[0048] In this embodiment, saliency assessment refers to: guided by the prior condition of screen centering, the first AI model, combined with the distribution characteristics learned during the training phase, estimates the probability that the geometric center region contains a pop-up window. When the probability is high, the first AI model maintains the determined boundary range unchanged; when the probability is low, the first AI model can dynamically adjust the boundary range, for example, appropriately expanding or shrinking the boundary range of the screen centering region. Through this method, the potential pop-up window region can adaptively change under different screen parameter information and application interface layouts, thereby improving the accuracy of potential pop-up window region determination.

[0049] In an alternative embodiment, such as Figure 2As shown, when the probability of saliency assessment is low, if it is determined that there is no pop-up within the potential pop-up area, the boundary range of the central area of ​​the screen can be expanded according to a preset expansion coefficient to obtain an expanded potential pop-up area. After expansion, it will include more screen area to cover potentially ignored pop-up content. Then, based on the expanded potential pop-up area, a new sub-image is cropped from the screen image. Based on the expanded potential pop-up area, the operation of cropping the sub-image corresponding to the pop-up from the screen image is performed, and the new sub-image is sent to the second AI model for subsequent operations. Optionally, to prevent unnecessary computational overhead and performance degradation caused by invalid expansion, a preset number of expansion operations is set. Specifically, each time expansion is performed, the current expansion count can be incremented by 1 until the preset number of expansions is reached. If no pop-up is detected, expansion stops to avoid excessive expansion and waste of computational resources.

[0050] Further optionally, the first AI model includes: a first feature extraction layer, a central region calculation layer, and a boundary determination layer. In this optional embodiment, when the first AI model calculates the geometric center of the screen image based on screen parameter information under the guidance of the screen centering prior condition, and determines the boundary range of the screen centering region based on the geometric center to obtain the potential pop-up window region, the process includes: inputting the screen image and screen parameter information into the first feature extraction layer, partitioning the screen image, extracting features from each region to obtain image features of multiple regions, and extracting target attributes related to screen size, resolution, and orientation from the screen parameter information; inputting the image features and target attributes of multiple regions into the central region calculation layer, and determining the target region corresponding to the geometric center based on the target attributes under the guidance of the screen centering prior condition; inputting the target region and target attributes into the boundary determination layer, determining the boundary range of the screen centering region based on the target region and target attributes, and generating the potential pop-up window region based on the boundary range.

[0051] In this embodiment, feature extraction is performed on each region, the screen image is partitioned, and vectorization is applied to each region to obtain image features for multiple regions. The purpose of partition extraction is to transform the overall information of the screen image into finer-grained local features, providing a finer-grained data foundation for the subsequent determination of the central region.

[0052] In this embodiment, the central region computation layer receives image features and target attributes from multiple regions. Guided by the prior condition of screen centering, it calculates the geometric center of the screen image, which serves as a position reference for potential pop-ups. Further, the central region computation layer combines the target attributes to determine the area surrounding the geometric center, thereby forming the target region for screen centering. Optionally, when determining the target region corresponding to the geometric center based on the target attributes, using the geometric center as a reference, the central region computation layer can analyze the screen image region containing the geometric center based on the target attributes and combined with pre-learned visual features of pop-up and non-pop-up regions to determine whether a pop-up is included. If a pop-up is included, the region containing the geometric center is determined as the target region.

[0053] like Figure 3a The image shown is an example of a screen image that includes pop-up and non-pop-up areas.

[0054] Furthermore, in this embodiment, the boundary determination layer receives the target region output by the central region calculation layer. Based on this, the boundary range of the target region is determined. The boundary determination layer, based on information such as the size, position, and shape of the target region, and combined with target attributes, determines a more precise boundary range. This boundary range can cover potential pop-up window areas while effectively reducing the introduction of non-pop-up window areas (also known as background areas), providing more accurate input for subsequent pop-up window recognition and reducing the amount of data that needs to be processed for recognition.

[0055] Furthermore, based on the potential pop-up window area, the sub-image corresponding to the pop-up window is cropped from the screen image. The cropping method can be referred to the above, and will not be repeated here.

[0056] like Figure 3b The image shown is an example of a sub-image, in which the sub-image includes a non-popup area, but this does not constitute a limitation on this embodiment.

[0057] Given a sub-image of the potential pop-up area, the second AI model is invoked based on the sub-image and the visual features of the pop-up. The pop-up feature information is then identified based on the sub-image, and the position of the close option within the pop-up is located based on the pop-up feature information.

[0058] Optionally, based on the visual features of the sub-image and the pop-up window, a second AI model is invoked to identify the layout structure features, text features, and contextual difference features between the pop-up window area and the non-pop-up window area in the sub-image based on the pop-up window visual features; and the location information of the close option is located based on the layout structure features, text features, and contextual difference features.

[0059] In this embodiment, the second AI model receives a sub-image as input and, combined with the pop-up visual features learned during the training phase, performs multi-dimensional feature analysis on the sub-image. During this multi-dimensional feature analysis, the second AI model can identify the pop-up's layout structure features, text features, and contextual difference features. Layout structure features include determining whether a title bar, prompt content area, and interactive area exist within the sub-image, and identifying the hierarchical relationship between these areas. Secondly, it extracts the text features from the pop-up, including the text features of each area within the pop-up area. Furthermore, the second AI model can also identify the contextual difference features between the pop-up and its background, also known as the non-pop-up area, which in this case refers to the non-pop-up area within the sub-image. For example, pop-ups are often overlaid on the sub-image as a floating layer, superimposed on the main interface content. The model can learn to recognize this hierarchical relationship and content occlusion features, thereby determining the pop-up area and non-pop-up area within the sub-image.

[0060] In this embodiment, after extracting layout structure features, text features, and contextual difference features, the second AI model further locates the close option based on these features. Specifically, the second AI model first uses layout structure features to determine the area of ​​the interaction zone; within this area, it combines text features to identify interactive controls with close semantics, such as buttons containing text content like "Cancel," "Close," or "×." One example of an interactive option is a button, but it is not limited to this. Simultaneously, it determines whether the button belongs to a pop-up window area based on contextual difference features, thus eliminating interfering options outside of pop-up window areas. Through this method, the second AI model can output the location information of the close option on the screen, providing precise positioning for subsequent automated testing tools to perform interactive operations.

[0061] Optionally, the second AI model includes: a second feature extraction layer, a layout structure analysis layer, a text feature analysis layer, and a contextual difference analysis layer. Based on the visual features of the sub-image and the pop-up window, the second AI model is invoked to identify the layout structure features, text features, and contextual difference features between the pop-up window area and the non-pop-up window area in the sub-image, based on the pop-up window visual features. This includes: inputting the sub-image into the second feature extraction layer to extract the visual features of the sub-image; inputting the visual features into the layout structure analysis layer, performing local feature analysis on the visual features based on the pop-up window visual features to obtain the layout features of each area of ​​the pop-up window and the structural hierarchy relationship of different areas, which are then used as layout structure features. The pop-up window areas include the title area, the prompt content area, and the interaction area; inputting the layout structure features and visual features into the text feature analysis layer, analyzing the text content in the title area, the prompt content area, and the interaction area based on the layout structure features of each area to obtain text features, including the text features corresponding to each area; and inputting the visual features and layout structure features into the contextual difference analysis layer to identify the occlusion and overlay relationship between the pop-up window area and the non-pop-up window area in the sub-image, thereby obtaining contextual difference features.

[0062] In this embodiment, the sub-image is input to the second feature extraction layer. This layer performs hierarchical feature extraction on the sub-image to obtain visual features comprising multiple levels. Optionally, these visual features can be visual features extracted layer by layer through convolutional layers. The number of convolutional layers is positively correlated with the level of the visual features; the more convolutional layers, the deeper the level of the extracted visual features, the richer the semantic information expressed, and the more advanced the visual features. Different levels of visual features refer to image features of different depths. Through hierarchical feature extraction, information at different levels of abstraction is captured.

[0063] Subsequently, the visual features are input into the layout structure analysis layer. Guided by the pop-up's visual features, the layout structure analysis layer performs regionalization processing on the input visual features, identifying the title area, prompt content area, and interaction area of ​​the pop-up in the sub-image, and determining the hierarchical relationship between these areas. Thus, the layout structure features not only reflect the regional distribution of the pop-up, but also characterize the hierarchical relationship of the pop-up's regional division.

[0064] Next, the layout structure features and visual features are input into the text feature analysis layer. Based on the layout structure features of each region, the text feature analysis layer detects and identifies the text content in the title area, prompt content area, and interactive area. During the identification process, not only is the semantic information of the text content extracted, but its position and functional attributes in different regions are also combined to generate corresponding text features. These text features are feature representations of the text content in each region, reflecting the semantic information and spatial distribution patterns of the text content across different regions, providing a data foundation for subsequent semantic recognition targeting specific regions.

[0065] Furthermore, visual features and layout structure features are input into the contextual difference analysis layer. This layer analyzes the contextual differences between pop-up and non-pop-up regions within a sub-image, thereby assisting the model in distinguishing between pop-ups and the background. Specifically, the contextual difference analysis layer identifies the hierarchical and occlusion relationships of different regions based on the input visual features. For example, pop-ups typically overlay the main interface as independent views, thus creating a clearly defined occlusion relationship with the main interface. Since the contextual difference analysis layer learns from a large number of samples containing both pop-ups and non-pop-ups during the training phase, it can identify contextual differences in sub-images based on the input visual and layout structure features during the inference phase and extract contextual difference features accordingly.

[0066] Optionally, the second AI model further includes an interaction option analysis layer. When locating the position information of the close option in the pop-up window based on the pop-up feature information, the second AI model includes: inputting layout structure features and text features into the interaction option analysis layer; performing semantic recognition on the text features in the title area, prompt content area, and interaction area respectively to obtain the corresponding title text content, prompt text content, and text content of the interaction option in the interaction area; determining the close option based on the layout structure features and the text content of the interaction option; and locating the positioning information of the close option based on the layout structure features and the text content of the close option, wherein the positioning information includes at least one of the text positioning information and coordinate position information corresponding to the close option.

[0067] In this embodiment, the interaction option analysis layer of the second AI model, upon receiving layout structure features and text features, performs semantic recognition on the text features corresponding to different areas of the pop-up window. Through semantic recognition, the text content corresponding to each area is obtained, such as the title text content in the title area, the prompt text content in the prompt content area, and the interaction option text content in the interaction area. The text content of the interaction options includes, but is not limited to, "OK", "Cancel", and "Will remind you later".

[0068] Building upon this, the interaction option analysis layer combines layout structure features with the text content within the interaction area to determine the interaction option corresponding to the close operation, i.e., the close option. Further, based on the layout structure features and the text content of the close option, the positioning information of the close option is located. This positioning information includes at least text positioning information and / or coordinate position information.

[0069] Text location information refers to the correspondence between the text content of the close option and its corresponding UI element, used to uniquely identify the UI element corresponding to the close option. For example, text content such as "OK" or "Cancel" can be used as text location information. Based on the fact that this text content can correspond to specific UI controls (one example of UI elements), it provides location support for the close operation.

[0070] The coordinate position information refers to the position information of the close option in the screen coordinate system of the target device. It can be represented by pixel coordinates or relative coordinates and is used to locate the position of the close option on the screen.

[0071] In an optional embodiment of this application, the second AI model does not operate in isolation. When the second AI model performs pop-up feature recognition, it actively queries an external prior knowledge base to assist in recognizing and locating the pop-up. This process forms a second layer of collaboration, namely, cooperation between the second AI model and the prior knowledge base. By working closely with the prior knowledge base, the accuracy and robustness of pop-up recognition are further improved. The second layer of collaboration is described in detail below.

[0072] Optionally, the prior knowledge base is a multilingual prior knowledge base. This knowledge base maintains the text content of common closing options and their corresponding positional distribution patterns, and has the ability to supplement knowledge across languages ​​and scenarios.

[0073] The knowledge base maintains the text content of common closing options in different languages. For example, common closing options in Chinese include "OK", "Cancel", and "Close", while common closing options in English include "OK", "Cancel", and "Close". Other language environments also have corresponding common interactive options. At the same time, the knowledge base also maintains the positional distribution pattern of these interactive options in the interaction area, such as being located at the bottom center, bottom right corner, or top right corner of the interaction area.

[0074] Based on this, when invoking the second AI model, it can perform semantic matching based on the language category of the text content of the "close" option and by calling a prior knowledge base. By comparing it with the text of common interaction options maintained in the knowledge base, at least one candidate interaction option is obtained. Then, the second AI model performs a consistency check between the positional distribution pattern corresponding to at least one candidate interaction option and the layout structure features in the sub-image, and selects the candidate interaction option that passes the consistency check as the "close" option.

[0075] The consistency check involves matching the positional distribution pattern of at least one candidate interactive option maintained in the prior knowledge base with the actual layout structure features identified by the layout structure analysis layer in the sub-image to determine whether at least one candidate interactive option conforms to the common distribution patterns of pop-ups. The positional distribution pattern refers to the typical layout of interactive options in a pop-up window, pre-stored in the knowledge base, such as a close button located in the upper right corner of the title bar, or an "OK" button located below the interaction area.

[0076] In some cases, when new styles of pop-ups or "rogue pop-ups" not covered by the training set (such as a close button disguised as a normal element) appear, the model's generalization ability is insufficient, leading to recognition failure.

[0077] To address the aforementioned issues, in some embodiments, when faced with a new style pop-up, if it is not matched in the prior knowledge base, the text content of the candidate interaction option is stored as new text content in the prior knowledge base, thus achieving semantic expansion. Simultaneously, the positional distribution pattern corresponding to the candidate interaction option can also be stored, so that when a new style pop-up with a similar distribution appears later, the positional distribution model in the knowledge base can be queried for comparison, improving the model's generalization ability and recognition accuracy in unknown scenarios.

[0078] Furthermore, in some embodiments, when faced with "rogue pop-ups," in this embodiment, the second AI model performs a consistency check between the positional distribution pattern corresponding to the candidate interaction option and the positional distribution pattern maintained in the prior knowledge base. When there is a significant deviation between the two positional distribution patterns, the candidate interaction option is determined to be a "close" option disguised as a normal element and is removed from the candidate interaction options.

[0079] In one optional embodiment, after the first AI model adjusts the boundary range based on the saliency assessment, the second AI model performs pop-up feature recognition based on the new cropped area. For example... Figure 2As shown, if the second AI model fails to recognize the pop-up window or the click operation fails, a retry mechanism will be triggered. Specifically, when the second AI model fails to click the close button, it returns a pop-up window closing failure message to the first AI model. Then, upon receiving the pop-up window closing failure message, the first AI model dynamically adjusts the boundary range of the potential pop-up window area. For example, it can adaptively correct the already determined potential area. This correction may include adjusting the expansion coefficient or direction of the potential area based on feedback information to obtain a new potential pop-up window area, and then re-cropping the new potential pop-up window area as new input for the second AI model to execute the subsequent operations of this embodiment. By combining the dynamic adjustment of the first AI model with the retry mechanism, not only is the accuracy of the pop-up window area improved in the initial recognition, but it also allows for readjustment and retrying when subsequent recognition fails, thus forming a third layer of collaboration between the two-stage models, improving the accuracy of pop-up window recognition and optimizing the efficiency of computing resource utilization. Optionally, a preset number of retry attempts is set for the retry mechanism, and retries stop when the preset number of retry attempts is reached.

[0080] In an optional scenario embodiment, when the target device is implemented as a mobile device, the above method is applied to a computer device, but this does not limit the execution subject of this embodiment. In this embodiment, the device management tool is deployed on the computer device and interconnected with the mobile device. One computer device can support connecting multiple mobile devices to form a group control system. The group control system refers to achieving automated testing by using multiple mobile devices or simulating multiple mobile devices and installing script software on the mobile devices to control applications on the phone. The pop-up handling method for any mobile device can be implemented with reference to the above embodiment, and will not be repeated here.

[0081] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0082] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0083] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0084] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device includes a memory 44 and a processor 45.

[0085] Memory 44 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0086] The processor 45, coupled to the memory 44, is used to execute a computer program in the memory 44 for: capturing a screen image of the target device if a pop-up appears on the screen of the target device during the operation of an automated testing tool on the target device; calling a first AI model for potential pop-up region analysis based on the screen image and the screen parameter information of the target device to analyze the potential pop-up region of the screen image; cropping a sub-image corresponding to the pop-up from the screen image based on the potential pop-up region; calling a second AI model for pop-up feature recognition based on the sub-image and the visual features of the pop-up, recognizing pop-up feature information based on the sub-image, and locating the position information of the close option in the pop-up based on the pop-up feature information; and calling the automated testing tool to operate the close option to close the pop-up based on the potential pop-up region, the position information, and the screen parameter information.

[0087] In an optional embodiment, the first AI model presets a screen centering prior condition, which is used to constrain the potential pop-up window area to be preferentially a screen centering area. When the processor 45 calls the first AI model to analyze the potential pop-up window area of ​​the screen image based on the screen image and the screen parameter information of the target device, it specifically performs the following: based on the screen image and the screen parameter information of the target device, it calls the first AI model to calculate the geometric center of the screen image based on the screen parameter information under the guidance of the screen centering prior condition, and determines the boundary range of the screen centering area based on the geometric center to obtain the potential pop-up window area.

[0088] In an optional embodiment, the first AI model includes: a first feature extraction layer, a central region calculation layer, and a boundary determination layer; when the processor 45 calls the first AI model based on the screen image and the screen parameter information of the target device, and calculates the geometric center of the screen image based on the screen parameter information under the guidance of the screen centering prior condition, and determines the boundary range of the screen centering region based on the geometric center to obtain the potential pop-up window region, it is specifically used to: input the screen image and the screen parameter information into the first feature extraction layer, partition the screen image, extract features from each region to obtain image features of multiple regions, and extract target attributes related to screen size, resolution, and orientation from the screen parameter information; input the image features of the multiple regions and the target attributes into the central region calculation layer, and determine the target region corresponding to the geometric center based on the target attributes under the guidance of the screen centering prior condition; input the target region and the target attributes into the boundary determination layer, determine the boundary range of the screen centering region based on the target region and the target attributes, and generate the potential pop-up window region based on the boundary range.

[0089] In an optional embodiment, when the processor 45 invokes the second AI model based on the sub-image and pop-up visual features, identifies pop-up feature information based on the sub-image, and locates the position information of the close option in the pop-up based on the pop-up feature information, the processor 45 includes: invoking the second AI model based on the sub-image and pop-up visual features, identifying the layout structure features, text features, and contextual difference features between the pop-up area and non-pop-up area of ​​the sub-image based on the pop-up visual features; and locating the position information of the close option based on the layout structure features, the text features, and the contextual difference features.

[0090] In an optional embodiment, the second AI model includes: a second feature extraction layer, a layout structure analysis layer, a text feature analysis layer, and a context difference analysis layer; when the processor 45 calls the second AI model based on the sub-image and pop-up visual features to identify the layout structure features, text features, and context difference features between the pop-up area and the non-pop-up area in the sub-image based on the pop-up visual features, it is specifically configured to: input the sub-image into the second feature extraction layer to extract the visual features of the sub-image; input the visual features into the layout structure analysis layer, and perform local feature analysis on the visual features based on the pop-up visual features to obtain the pop-up... The layout structure features of each area of ​​the window and the hierarchical relationship of different areas are used as the layout structure features. The pop-up window area includes a title area, a prompt content area, and an interaction area. The layout structure features and the visual features are input into the text feature analysis layer. Based on the layout features of each area, the text content in the title area, the prompt content area, and the interaction area are analyzed to obtain the text features. The text features include the text features corresponding to each area. The visual features and the layout structure features are input into the context difference analysis layer to identify the occlusion and superposition relationship between the pop-up window area and the non-pop-up window area in the sub-image to obtain the context difference features.

[0091] In an optional embodiment, the second AI model further includes an interaction option analysis layer. When the processor 45 calls the second AI model to locate the position information of the close option in the pop-up based on the pop-up feature information, it is specifically configured to: input the layout structure features and the text features into the interaction option analysis layer; perform semantic recognition on the text features in the title area, the prompt content area, and the interaction area respectively to obtain the corresponding title text content, prompt text content, and text content of the interaction option in the interaction area; determine the close option based on the layout structure features and the text content of the interaction option; and locate the positioning information of the close option based on the layout structure features and the text content of the close option, wherein the positioning information includes at least one of the text positioning information and coordinate position information corresponding to the close option.

[0092] In one optional embodiment, the second AI model is associated with a multilingual prior knowledge base, which maintains the text content and positional distribution patterns of common interaction options. When the processor 45 determines the target interaction option within the interaction area based on the layout structure features and the text content of the interaction options, it specifically performs the following: according to the language category of the text content of the close option, it calls the prior knowledge base to perform semantic matching between the text content of the close option and the text content of the common interaction options maintained by the prior knowledge base, so as to obtain at least one candidate interaction option; it performs a consistency check between the positional distribution pattern of the at least one candidate interaction option and the layout structure features, and uses the candidate interaction option that passes the consistency check as the close option.

[0093] In an optional embodiment, the processor 45 is further configured to: if the pop-up window does not exist in the potential pop-up window area, expand the boundary range of the screen center area according to a preset expansion coefficient to obtain an expanded potential pop-up window area; and based on the expanded potential pop-up window area, perform the operation of cropping the sub-image corresponding to the pop-up window from the screen image and subsequent operations.

[0094] Furthermore, such as Figure 4 As shown, the electronic device also includes other components such as a communication component 46, a display 47, a power supply component 48, and an audio component 49. Figure 4 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 4 The components shown. Additionally... Figure 4 The components within the dashed box are optional, not mandatory, and their specific configuration depends on the form factor of the electronic device. The electronic device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 4 The components within the dashed box; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., it may be omitted. Figure 4 The component within the dashed box.

[0095] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0096] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0097] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0098] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0099] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0100] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0101] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0102] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A pop-up window handling method, characterized in that, include: If a pop-up window appears on the screen of the target device during the operation of the automated testing tool, the screen image of the target device will be captured. Based on the screen image and the screen parameter information of the target device, a first AI model for potential pop-up region analysis is invoked to analyze the potential pop-up region of the screen image; Based on the potential pop-up window area, crop the sub-image corresponding to the pop-up window from the screen image; Based on the sub-image and the visual features of the pop-up window, a second AI model for pop-up window feature recognition is invoked to identify pop-up window feature information based on the sub-image, and to locate the position information of the close option in the pop-up window based on the pop-up window feature information; Based on the potential pop-up area, the location information, and the screen parameter information, the automated testing tool is invoked to operate the close option to close the pop-up.

2. The method according to claim 1, characterized in that, The first AI model presets a screen centering prior condition, which constrains the potential pop-up window area to be preferentially centered on the screen. Then, based on the screen image and the screen parameter information of the target device, the first AI model is invoked to analyze the potential pop-up window area of ​​the screen image, including: Based on the screen image and the screen parameter information of the target device, the first AI model is invoked. Guided by the screen centering prior condition, the geometric center of the screen image is calculated based on the screen parameter information, and the boundary range of the screen centering area is determined based on the geometric center to obtain the potential pop-up window area.

3. The method according to claim 2, characterized in that, The first AI model includes: a first feature extraction layer, a central region calculation layer, and a boundary determination layer; based on the screen image and the screen parameter information of the target device, the first AI model is invoked, and under the guidance of the screen centering prior condition, the geometric center of the screen image is calculated based on the screen parameter information, and the boundary range of the screen centering region is determined based on the geometric center to obtain the potential pop-up window region, including: The screen image and the screen parameter information are input into the first feature extraction layer. The screen image is divided into regions, and feature extraction is performed on each region to obtain image features of multiple regions. Target attributes related to screen size, resolution, and orientation are extracted from the screen parameter information. The image features of the multiple regions and the target attributes are input into the central region calculation layer. Guided by the screen centering prior condition, the target region corresponding to the geometric center is determined based on the target attributes. The target region and the target attribute are input into the boundary determination layer. Based on the target region and the target attribute, the boundary range of the screen center area is determined, and the potential pop-up window area is generated based on the boundary range.

4. The method according to claim 1, characterized in that, Based on the sub-image and the visual features of the pop-up window, the second AI model is invoked to identify pop-up window feature information based on the sub-image, and to locate the position information of the close option in the pop-up window based on the pop-up window feature information, including: Based on the visual features of the sub-image and the pop-up window, the second AI model is invoked to identify the layout structure features, text features, and contextual difference features between the pop-up window area and the non-pop-up window area in the sub-image based on the pop-up window visual features; based on the layout structure features, the text features, and the contextual difference features, the location information of the close option is located.

5. The method according to claim 4, characterized in that, The second AI model includes: a second feature extraction layer, a layout structure analysis layer, a text feature analysis layer, and a contextual difference analysis layer. Based on the visual features of the sub-image and the pop-up window, the second AI model is invoked to identify the layout structure features, text features, and contextual difference features between the pop-up window region and the non-pop-up window region of the sub-image, including: The sub-image is input into the second feature extraction layer to extract the visual features of the sub-image; The visual features are input into the layout structure analysis layer. Based on the visual features of the pop-up window, local feature analysis is performed on the visual features to obtain the layout structure features of each area of ​​the pop-up window and the structural hierarchy relationship of different areas, which are used as the layout structure features. The areas of the pop-up window include the title area, the prompt content area, and the interaction area. The layout structure features and the visual features are input into the text feature analysis layer. Based on the layout features of each region, the text content in the title area, prompt content area and interaction area are analyzed to obtain the text features, which include the text features corresponding to each region. The visual features and layout structure features are input into the context difference analysis layer to identify the occlusion and superposition relationship between the pop-up area in the sub-image and the non-pop-up area in the sub-image, so as to obtain the context difference features.

6. The method according to claim 5, characterized in that, The second AI model further includes: an interaction option analysis layer, which, when locating the position information of the close option in the pop-up window based on the pop-up window feature information, includes: The layout structure features and the text features are input into the interaction option analysis layer, and semantic recognition is performed on the text features in the title area, the prompt content area and the interaction area respectively to obtain the corresponding title text content, prompt text content and the text content of the interaction options in the interaction area; The close option is determined based on the layout structure features and the text content of the interaction options; Based on the layout structure features and the text content of the close option, the location information of the close option is located, and the location information includes at least one of the text location information and coordinate position information corresponding to the close option.

7. The method according to claim 6, characterized in that, The second AI model is associated with a multilingual prior knowledge base, which maintains the text content and positional distribution patterns of common interaction options; Based on the layout structure features and the text content of the interaction options, the target interaction options within the interaction area are determined, including: Based on the language category of the text content of the shutdown option, the prior knowledge base is invoked to perform semantic matching between the text content of the shutdown option and the text content of the common interaction options maintained by the prior knowledge base, so as to obtain at least one candidate interaction option. The positional distribution pattern of the at least one candidate interaction option is checked for consistency with the layout structure features, and the candidate interaction option that passes the consistency check is used as the closed option.

8. The method according to claim 2, characterized in that, The method further includes: If the pop-up window does not exist within the potential pop-up window area, the boundary range of the screen center area is expanded according to a preset expansion coefficient to obtain the expanded potential pop-up window area. Based on the expanded potential pop-up area, perform the operation of cropping the sub-image corresponding to the pop-up from the screen image and subsequent operations.

9. An electronic device, characterized in that, include: A memory and a processor, the memory for storing a computer program, the processor coupled to the memory for executing the computer program to implement the steps of the method of any one of claims 1-8.

10. A computer-readable storage medium storing a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the processor is enabled to perform the steps of the method according to any one of claims 1-8.

Images