Automatic task page popup window automatic closing method and device, equipment and medium

Abstract

The invention relates to the field of computer vision, and discloses an automatic task page pop-up window closing method and device, equipment and a medium. The method comprises the steps that firstly, a current loading webpage is jumped, and a current webpage screenshot image is obtained; obtaining a popup window positioning area and a popup window type identification result corresponding to the current webpage screenshot image based on a popup window identification model; if it is determined that the pop-up window type identification result is a preset pop-up window type in a preset pop-up window type set, performing corresponding processing on a pop-up window page element corresponding to the pop-up window positioning area based on a pop-up window processing strategy; if it is determined that the popup window page element in the currently loaded webpage is in the processed state, processing the currently loaded webpage by the automatic task to obtain a current webpage content processing result, and storing the current webpage content processing result after the current webpage content processing result is successfully matched with an expected result. According to the embodiment of the invention, the popup window page elements detected in the currently loaded webpage in the execution process of the automatic task are correspondingly and quickly processed by combining a computer vision technology, so that the continuity and reliability of the execution of the automatic task are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology of artificial intelligence, and in particular to a method, device, equipment and medium for automatically closing pop-up windows on automated task pages. Background Art

[0002] In the field of modern information technology, Robotic Process Automation (RPA, full name Robotic Process Automation, is a technology that uses software robots or artificial intelligence assistants to simulate and execute repetitive, rule-defined and large-scale business process operations of humans in a digital environment) has been widely used in various business processes to improve efficiency and reduce operating costs. RPA usually involves the task of automatically collecting information from web pages. However, in actual operation, RPA systems often encounter interference factors such as web page ad pop-ups, which may cause the target elements to be blocked, resulting in click failures and affecting the accuracy and efficiency of the task.

[0003] Currently, existing technologies mainly try to ignore ad pop-ups through simple preset rules or whitelist filtering, but the above methods often do not work well in actual complex scenarios. This is because current web ads are often diverse and dynamic, and it is difficult to fully cover and identify them through static rules. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and medium for automatically closing pop-up windows on automated task pages, aiming to solve the problem in the prior art that when a robotic process automation system encounters interference factors such as web page advertisement pop-ups when automatically executing to obtain information from a web page, it uses preset rules or whitelist filtering to ignore advertisement pop-ups. It is difficult to cover the diversity and dynamics of web page advertisements through static rules, resulting in the inability to quickly close web page advertisement pop-ups.

[0005] In a first aspect, an embodiment of the present invention provides a method for automatically closing a pop-up window on an automated task page, comprising:

[0006] In response to a webpage loading instruction generated in the automation task, jump to a currently loaded webpage corresponding to the webpage loading instruction, and obtain a current webpage screenshot image corresponding to the currently loaded webpage;

[0007] Obtaining a pop-up window location area and a pop-up window type recognition result corresponding to the current webpage screenshot image based on a pre-trained pop-up window recognition model;

[0008] If it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, then based on a preset pop-up window processing strategy, a pop-up window page element corresponding to the pop-up window positioning area is processed accordingly to update the currently loaded web page;

[0009] If it is determined that the pop-up page element in the currently loaded web page is in a processed state, then performing corresponding automated data processing on the currently loaded web page based on the automated task to obtain a current web page content processing result; and

[0010] If it is determined that the current webpage content processing result matches the pre-stored expected result, the current webpage content processing result is stored in a preset storage space.

[0011] In a second aspect, an embodiment of the present invention further provides a device for automatically closing a pop-up window on an automated task page, comprising:

[0012] A webpage loading and screenshot unit, configured to respond to a webpage loading instruction generated in an automated task, jump to a currently loaded webpage corresponding to the webpage loading instruction, and obtain a current webpage screenshot image corresponding to the currently loaded webpage;

[0013] A pop-up window positioning and recognition unit, used to obtain a pop-up window positioning area and a pop-up window type recognition result corresponding to the current webpage screenshot image based on a pre-trained pop-up window recognition model;

[0014] A pop-up window processing unit, configured to, if it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, perform corresponding processing on the pop-up window page element corresponding to the pop-up window positioning area based on a preset pop-up window processing strategy to update the currently loaded web page;

[0015] an automated data processing unit, configured to, if it is determined that the pop-up page element in the currently loaded web page is in a processed state, perform corresponding automated data processing on the currently loaded web page based on the automated task to obtain a current web page content processing result; and

[0016] The data storage unit is used to store the current web page content processing result in a preset storage space if it is determined that the current web page content processing result matches the pre-stored expected result.

[0017] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0019] The embodiment of the present invention provides a method, device, equipment and medium for automatically closing pop-up windows of automated task pages, the method comprising: responding to a web page loading instruction generated in an automated task, jumping to a currently loaded web page corresponding to the web page loading instruction, and obtaining a current web page screenshot image corresponding to the currently loaded web page; obtaining a pop-up window positioning area and a pop-up window type recognition result corresponding to the current web page screenshot image based on a pre-trained pop-up window recognition model; if it is determined that the pop-up window type recognition result is one of the preset pop-up window types in the preset pop-up window type set, then based on a preset pop-up window processing strategy, the pop-up window page element corresponding to the pop-up window positioning area is processed accordingly to update the currently loaded web page; if it is determined that the pop-up window page element in the currently loaded web page is in a processed state, then based on the automated task, the currently loaded web page is processed accordingly to obtain the current web page content processing result; if it is determined that the current web page content processing result matches the pre-stored expected result, then the current web page content processing result is stored in a preset storage space. In the embodiment of the present invention, computer vision technology is combined to perform corresponding and rapid processing on the pop-up window page elements detected on the currently loaded web page during the execution of the automated task, so as to ensure the continuity and reliability of the execution of the automated task. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an application scenario of the method for automatically closing pop-up windows on an automated task page provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a process for automatically closing a pop-up window on an automated task page provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a sub-process of a method for automatically closing a pop-up window on an automated task page provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of another sub-flow of the method for automatically closing pop-up windows on an automated task page provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of another sub-process of the method for automatically closing a pop-up window on an automated task page provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of another sub-flow of the method for automatically closing pop-up windows on an automated task page provided by an embodiment of the present invention;

[0027] Figure 7 A schematic block diagram of a device for automatically closing pop-up windows on an automated task page provided by an embodiment of the present invention;

[0028] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] Please also see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of a scenario of a method for automatically closing pop-up windows on an automated task page according to an embodiment of the present invention. Figure 2 1 is a flow chart of a method for automatically closing a pop-up window on an automated task page provided by an embodiment of the present invention. The method is applied to a server 10 or a user terminal 20, and the server 10 can be connected to the user terminal 20 for communication. After the configuration of the automated task is completed in the server 10 or the user terminal 20, the automated task can be executed locally on the server 10 or the user terminal 20, or the server 10 or the user terminal 20 can send the automated task to other servers or other user terminals for automated task execution.

[0034] like Figure 2As shown, the method includes the following steps S110-S150.

[0035] S110 . In response to a webpage loading instruction generated in an automated task, jump to a currently loaded webpage corresponding to the webpage loading instruction, and obtain a current webpage screenshot image corresponding to the currently loaded webpage.

[0036] In this embodiment, the technical solution is described with either the server or the user terminal as the execution subject. For example, taking the server as the execution subject, an automated task pre-configured based on the task requirements is deployed on the server. More specifically, the automated task includes at least a web page access subtask, a pop-up window recognition and processing subtask, a web page information collection subtask, a web page information verification subtask, and the like. If the server determines that the web page access subtask has been executed when executing the automated task, a web page loading instruction can be generated based on the web page access subtask, and the page can be jumped to the currently loaded web page corresponding to the web page loading instruction. Of course, there will generally be a certain delay when jumping to the currently loaded web page, and the currently loaded web page needs to be automatically screenshotted after the currently loaded web page is completely loaded successfully to obtain a screenshot image of the current web page. Among them, the screenshot image of the current web page obtained is used to determine whether there is a pop-up window element in the currently loaded web page.

[0037] In one embodiment, if Figure 3 As shown, step S110 includes:

[0038] S111, the automation task obtains a current target web page address corresponding to the web page loading instruction, and jumps to the currently loaded web page based on the current target web page address;

[0039] S112: The automation task obtains an automatic screenshot instruction, and executes the automatic screenshot instruction to take a screenshot of the currently loaded web page to obtain a screenshot image of the current web page.

[0040] In this embodiment, the loading of the currently loaded web page can be realized specifically through the web page access subtask in the automation task, that is, the current target web page address corresponding to the web page loading instruction is obtained from the web page access subtask, and the current loaded web page is jumped to based on the current target web page address. Before obtaining the current target web page address, it is necessary to start a local browser, and input the current target web page address into the URL input box of the browser based on the web page access subtask, so as to realize the fast and automatic loading of the currently loaded web page.

[0041] After the successful jump and page loading of the currently loaded web page is completed, the corresponding screenshot tool is started based on the automatic screenshot instruction pre-configured in the web page access subtask to screenshot the complete web page of the currently loaded web page, thereby quickly obtaining the screenshot image of the current web page.

[0042] S120. Obtaining a pop-up window location area and a pop-up window type recognition result corresponding to the current webpage screenshot image based on a pre-trained pop-up window recognition model.

[0043] In this embodiment, after obtaining the current webpage screenshot image, in order to quickly detect whether there is a preset pop-up window type in it, the pop-up window recognition model can be used to first locate whether there is a pop-up window area in the current webpage screenshot image. After completing the pop-up window area location, for example, it is determined that there are several pop-up window location areas in the current webpage screenshot image, and then the specific content of each pop-up window location area is obtained and the type is determined, so as to obtain the pop-up window type recognition results corresponding to the several pop-up window location areas.

[0044] In one embodiment, if Figure 4 As shown, step S120 includes:

[0045] S121, locating a pop-up window area in the current webpage screenshot image based on a target detection model in the pop-up window recognition model to obtain the pop-up window locating area;

[0046] S122, recognizing the text in the pop-up window positioning area based on the text recognition model in the pop-up window recognition model to obtain a text recognition result;

[0047] S123. Identify and classify the text recognition result based on the classification model in the pop-up window recognition model to obtain the pop-up window type recognition result.

[0048] In this embodiment, the specific pop-up window recognition model used at least includes a target detection model (such as using any one of the models YOLOV1-YOLOV8, etc.), a text recognition model (such as using OCR, an optical character recognition technology) and a classification model (such as using naive Bayes, support vector machine, decision tree, random forest, deep learning model TextCNN, FastText, LSTM, BERT, etc.). First, the pop-up window area in the current web page screenshot image is located by the target detection model to obtain the pop-up window positioning area. Generally, the number of pop-up window positioning areas obtained is greater than or equal to 0. Afterwards, if the number of the pop-up window positioning area is 1 and the pop-up window positioning area is a rectangular area, at this time, the text recognition model can be combined to perform text recognition on the target area image corresponding to the pop-up window positioning area in the current web page screenshot image to obtain the text recognition result. Finally, when the text recognition results are identified and classified based on the classification model, the text recognition results can be first extracted by a machine learning model (such as a naive Bayes classifier), and sentence extraction can also be performed. After obtaining the keywords and sentence classification results of the text recognition results, they are input into the classification model (such as a naive Bayes classification model) to obtain the pop-up type recognition results. It can be seen that the above-mentioned technology combined with computer vision in artificial intelligence can quickly identify the pop-up location area and pop-up type recognition results in the currently loaded web page.

[0049] S130: If it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, the pop-up window page element corresponding to the pop-up window positioning area is processed accordingly based on a preset pop-up window processing strategy to update the currently loaded web page.

[0050] In this embodiment, for example, the preset pop-up window type set includes at least an advertisement pop-up window type. If the pop-up window type identification result is determined to be one of the preset pop-up window types in the preset pop-up window type set, it means that there is at least one pop-up window page element in the currently loaded web page, and the pop-up window page element corresponding to the pop-up window positioning area needs to be processed in a timely manner. Moreover, the pop-up window processing strategy adopted can close or avoid the pop-up window page element corresponding to the pop-up window positioning area, thereby eliminating the impact of the pop-up window positioning area on the execution of subsequent automated tasks.

[0051] In one embodiment, if Figure 5 As shown in the first embodiment of step S130, step S130 includes:

[0052] S131A, obtaining a pop-up window page closing strategy in the pop-up window processing strategy;

[0053] S132A, identifying a close button of the pop-up window positioning area in the currently loaded web page based on the pop-up window page closing strategy, and obtaining a close button positioning area;

[0054] S133A: automatically click the close button positioning area based on the automated task, close the pop-up page element corresponding to the pop-up positioning area, and update the currently loaded web page.

[0055] In this embodiment, if the pop-up window processing strategy specifically includes a pop-up window page closing strategy as an example, after obtaining the pop-up window positioning area in the currently loaded web page, it is necessary to identify the image area corresponding to the pop-up window positioning area in the current web page screenshot image, that is, the pop-up window processing strategy can also include the target detection model used in the above pop-up window recognition model, and based on the target detection model, the close button of the image area corresponding to the pop-up window positioning area in the current web page screenshot image can be located (it can also be replaced by the recognition ignore button, "×" button, etc. in the specific implementation), thereby obtaining the close button positioning area (for example, corresponding to a rectangular area, and the coordinates of the four vertices of the rectangular area are also known). Afterwards, based on the pop-up window recognition processing subtask in the automation task, a simulated user clicks the close button operation (which can be understood as automatically moving the mouse icon in the currently loaded web page to the close button positioning area and clicking it) is generated to close the pop-up window page element corresponding to the pop-up window positioning area, and finally update the currently loaded web page after completing the closing of the pop-up window page element. It can be seen that, on the one hand, after the pop-up window positioning area is identified, the close button positioning area of ​​the pop-up window positioning area is identified by computer vision technology and the close button is automatically clicked, realizing the automatic identification and closing of the pop-up window page elements, avoiding the continued existence of the pop-up window page elements and the impact on the execution of the automated task. On the other hand, the pop-up window page elements are automatically processed through the pop-up window processing strategy, reducing the need for manual intervention and improving the degree of task automation.

[0056] In one embodiment, if Figure 6 As shown in the second embodiment of step S130, step S130 includes:

[0057] S131B, obtaining a pop-up window page avoidance strategy in the pop-up window processing strategy;

[0058] S132B, obtaining a current operation path for the currently loaded web page in the automation task;

[0059] S133B, based on the pop-up window page avoidance strategy and the pop-up window positioning area, adjusting the path in the current operation path that passes through the pop-up window positioning area to a path that bypasses the pop-up window positioning area, so as to update the current operation path;

[0060] S134B. Setting a click-forbidden attribute for the pop-up page element corresponding to the pop-up window positioning area based on the pop-up page avoidance strategy to update the currently loaded web page.

[0061] In this embodiment, after detecting the pop-up window area in the currently loaded web page, in addition to automatically closing it based on the pop-up window page closing strategy in the pop-up window processing strategy, the current operation path for the currently loaded web page in the automated task can also be adjusted based on the pop-up window page avoidance strategy in the pop-up window processing strategy so that it avoids the pop-up window positioning area. Specifically, the path passing through the pop-up window positioning area in the current operation path is adjusted to a path that bypasses the pop-up window positioning area. For example, there is a section of the candidate operation path in the current operation path that is within the pop-up window positioning area. At this time, the candidate operation path (such as its starting point is point A and the end point is point B, and point A and point B are both points on the boundary of the pop-up window positioning area) can be adjusted to the area that bypasses along the boundary of the pop-up window positioning area, that is, starting from point A along the boundary of the pop-up window positioning area and always walking along the outer boundary until point B, thereby realizing the update of the candidate operation path, so that the latest candidate operation path is along the boundary of the pop-up window positioning area. After completing the adjustment of the current operation path for the currently loaded web page in the automated task, the pop-up page element corresponding to the pop-up positioning area can also be set to a prohibit-click attribute based on the pop-up page avoidance strategy, so that the click task simulated in the automated task will not be opened after clicking the pop-up page element corresponding to the pop-up positioning area, thereby avoiding affecting subsequent automated tasks.

[0062] S140: If it is determined that the pop-up page element in the currently loaded web page is in a processed state, then based on the automation task, corresponding automated data processing is performed on the currently loaded web page to obtain a current web page content processing result.

[0063] In this embodiment, if it is determined that the pop-up page element in the currently loaded web page is in a processed state, it means that the pop-up page element in the currently loaded web page will no longer affect the automatic execution of the automated task, and at this time, the currently loaded web page is continuously subjected to corresponding automated data processing based on the automated task to obtain the current web page content processing result. For example, the web page information collection subtask, web page information verification subtask, etc. in the automated task are continuously executed to obtain the current web page content processing result.

[0064] Among them, when determining whether the pop-up window page element in the currently loaded web page is in a processed state, the processing process of steps S120-S130 can be referred to, that is, in step S130, after the pop-up window page element corresponding to the pop-up window positioning area has been processed accordingly and the currently loaded web page has been updated, the current web page screenshot image corresponding to the currently loaded web page is obtained again, and the pop-up window positioning area and pop-up window type recognition result corresponding to the current web page screenshot image are obtained based on the pre-trained pop-up window recognition model. When the pop-up window positioning area and the pop-up window type recognition results are both null values, it means that the pop-up window page element in the currently loaded web page is in a processed state; when the pop-up window positioning area or the pop-up window type recognition result is not a null value, it means that the pop-up window page element in the currently loaded web page is in an unprocessed state. It can be seen that the above-mentioned process of verifying whether the pop-up window page element has been processed can further determine whether the automated task can continue to be executed. Moreover, through computer vision technology, real-time monitoring and dynamic adjustment of the web page environment can be achieved to ensure the continuity and reliability of the execution of automated tasks.

[0065] In one embodiment, step S140 includes:

[0066] A data collection task in the automation task is obtained, and the data collection task is executed to collect information in the currently loaded web page to obtain a processing result of the current web page content.

[0067] In this embodiment, if the data collection task in the automated task is continued to be executed at this time, the data collection task can be regarded as the web page information collection subtask in the above example. Through the automatic execution of the data collection task, the information in the currently loaded web page can be collected. Moreover, in the above information collection process, the pop-up page elements have been automatically identified and processed. The automated task can perform the information collection task more accurately, reduce click failures caused by pop-ups, and improve the efficiency of information collection and overall task execution.

[0068] In one embodiment, after step S130, the method further includes:

[0069] If it is determined that the pop-up page element in the currently loaded web page is in an unprocessed state, obtain the current web page screenshot image corresponding to the currently loaded web page, and return to execute the step of obtaining the pop-up location area and pop-up type identification result corresponding to the current web page screenshot image based on the pre-trained pop-up recognition model.

[0070] In this embodiment, if it is determined that the pop-up page element in the currently loaded web page is in an unprocessed state, it means that the pop-up page element in the currently loaded web page has not been successfully processed, and it will also affect the automatic execution of the automated task. At this time, the current web page screenshot image corresponding to the currently loaded web page can be obtained again, and the step of obtaining the pop-up window positioning area and pop-up window type recognition result corresponding to the current web page screenshot image based on the pre-trained pop-up window recognition model is returned. That is, the above process is repeated at least once to process the pop-up page element in the currently loaded web page again. Moreover, when it is detected that the pop-up page element in the currently loaded web page is in an unprocessed state, log data corresponding to the state that the pop-up page element has not been processed is generated, and the log data is saved as an exception log. When the total number of log data included in the exception log exceeds the preset number of logs within a certain time interval (such as a month, a week, etc.), it means that the recognition accuracy of the pop-up window recognition model cannot meet the use requirements, and an update prompt information of a pop-up window recognition model can be automatically generated based on a preset exception handling mechanism, and the update prompt information is sent by the server to the designated receiving terminal. After receiving the update prompt information, the user of the receiving terminal can make timely adjustments to the pop-up window recognition model in the server.

[0071] S150: If it is determined that the current webpage content processing result matches the pre-stored expected result, the current webpage content processing result is stored in a preset storage space.

[0072] In this embodiment, after obtaining the current web page content processing result by executing the automated task, it can also be compared with the pre-stored expected result to ensure the correctness of the collected data. After determining that the current web page content processing result matches the pre-stored expected result, the current web page content processing result can be stored in a preset storage space (such as a database of a server).

[0073] It can be seen that the embodiment of the method can combine computer vision technology to quickly process the pop-up page elements detected on the currently loaded web page during the execution of the automated task, thereby ensuring the continuity and reliability of the execution of the automated task.

[0074] Figure 7 1 is a schematic block diagram of an automatic closing device for an automated task page pop-up window provided by an embodiment of the present invention. Figure 7As shown, corresponding to the above automated task page pop-up window automatic closing method, the present invention also provides an automated task page pop-up window automatic closing device 100. The automated task page pop-up window automatic closing device comprises: a webpage loading screenshot unit 110, a pop-up window positioning and identification unit 120, a pop-up window processing unit 130, an automated data processing unit 140 and a data storage unit 150.

[0075] The webpage loading screenshot unit 110 is used to respond to the webpage loading instruction generated in the automation task, jump to the current loading webpage corresponding to the webpage loading instruction, and obtain the current webpage screenshot image corresponding to the current loading webpage.

[0076] In this embodiment, the technical solution is described with either the server or the user terminal as the execution subject. For example, taking the server as the execution subject, an automated task pre-configured based on the task requirements is deployed on the server. More specifically, the automated task includes at least a web page access subtask, a pop-up window recognition and processing subtask, a web page information collection subtask, a web page information verification subtask, and the like. If the server determines that the web page access subtask has been executed when executing the automated task, a web page loading instruction can be generated based on the web page access subtask, and the page can be jumped to the currently loaded web page corresponding to the web page loading instruction. Of course, there will generally be a certain delay when jumping to the currently loaded web page, and the currently loaded web page needs to be automatically screenshotted after the currently loaded web page is completely loaded successfully to obtain a screenshot image of the current web page. Among them, the screenshot image of the current web page obtained is used to determine whether there is a pop-up window element in the currently loaded web page.

[0077] In one embodiment, the webpage loading screenshot unit 110 is specifically used for:

[0078] The automation task obtains a current target web page address corresponding to the web page loading instruction, and jumps to the currently loaded web page based on the current target web page address;

[0079] The automation task obtains an automatic screenshot instruction, and executes the automatic screenshot instruction to take a screenshot of the currently loaded web page to obtain a screenshot image of the current web page.

[0080] In this embodiment, the loading of the currently loaded web page can be realized specifically through the web page access subtask in the automation task, that is, the current target web page address corresponding to the web page loading instruction is obtained from the web page access subtask, and the current loaded web page is jumped to based on the current target web page address. Before obtaining the current target web page address, it is necessary to start a local browser, and input the current target web page address into the URL input box of the browser based on the web page access subtask, so as to realize the fast and automatic loading of the currently loaded web page.

[0081] After the successful jump and page loading of the currently loaded web page is completed, the corresponding screenshot tool is started based on the automatic screenshot instruction pre-configured in the web page access subtask to screenshot the complete web page of the currently loaded web page, thereby quickly obtaining the screenshot image of the current web page.

[0082] The pop-up window positioning and identification unit 120 is used to obtain a pop-up window positioning area and a pop-up window type identification result corresponding to the current webpage screenshot image based on a pre-trained pop-up window identification model.

[0083] In this embodiment, after obtaining the current webpage screenshot image, in order to quickly detect whether there is a preset pop-up window type in it, the pop-up window recognition model can be used to first locate whether there is a pop-up window area in the current webpage screenshot image. After completing the pop-up window area location, for example, it is determined that there are several pop-up window location areas in the current webpage screenshot image, and then the specific content of each pop-up window location area is obtained and the type is determined, so as to obtain the pop-up window type recognition results corresponding to the several pop-up window location areas.

[0084] In one embodiment, the pop-up window location identification unit 120 is specifically used for:

[0085] Locating the pop-up window area in the current webpage screenshot image based on the target detection model in the pop-up window recognition model to obtain the pop-up window positioning area;

[0086] Recognize the text in the pop-up window positioning area based on the text recognition model in the pop-up window recognition model to obtain a text recognition result;

[0087] The text recognition result is identified and classified based on the classification model in the pop-up window recognition model to obtain the pop-up window type recognition result.

[0088] In this embodiment, the specific pop-up window recognition model used at least includes a target detection model (such as using any one of the models YOLOV1-YOLOV8, etc.), a text recognition model (such as using OCR, an optical character recognition technology) and a classification model (such as using naive Bayes, support vector machine, decision tree, random forest, deep learning model TextCNN, FastText, LSTM, BERT, etc.). First, the pop-up window area in the current web page screenshot image is located by the target detection model to obtain the pop-up window positioning area. Generally, the number of pop-up window positioning areas obtained is greater than or equal to 0. Afterwards, if the number of the pop-up window positioning area is 1 and the pop-up window positioning area is a rectangular area, at this time, the text recognition model can be combined to perform text recognition on the target area image corresponding to the pop-up window positioning area in the current web page screenshot image to obtain the text recognition result. Finally, when the text recognition results are identified and classified based on the classification model, the text recognition results can be first extracted by a machine learning model (such as a naive Bayes classifier), and sentence extraction can also be performed. After obtaining the keywords and sentence classification results of the text recognition results, they are input into the classification model (such as a naive Bayes classification model) to obtain the pop-up type recognition results. It can be seen that the above-mentioned technology combined with computer vision in artificial intelligence can quickly identify the pop-up location area and pop-up type recognition results in the currently loaded web page.

[0089] The pop-up window processing unit 130 is used to perform corresponding processing on the pop-up window page element corresponding to the pop-up window positioning area based on a preset pop-up window processing strategy if it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, so as to update the currently loaded web page.

[0090] In this embodiment, for example, the preset pop-up window type set includes at least an advertisement pop-up window type. If the pop-up window type identification result is determined to be one of the preset pop-up window types in the preset pop-up window type set, it means that there is at least one pop-up window page element in the currently loaded web page, and the pop-up window page element corresponding to the pop-up window positioning area needs to be processed in a timely manner. Moreover, the pop-up window processing strategy adopted can close or avoid the pop-up window page element corresponding to the pop-up window positioning area, thereby eliminating the impact of the pop-up window positioning area on the execution of subsequent automated tasks.

[0091] In one embodiment, as a first embodiment of the pop-up window processing unit 130, the pop-up window processing unit 130 is specifically used for:

[0092] Obtain a pop-up window page closing strategy in the pop-up window processing strategy;

[0093] Based on the pop-up window page closing strategy, a close button of the pop-up window positioning area in the currently loaded web page is identified to obtain a close button positioning area;

[0094] The close button positioning area is automatically clicked based on the automated task to close the pop-up window page element corresponding to the pop-up window positioning area, and the currently loaded web page is updated.

[0095] In this embodiment, if the pop-up window processing strategy specifically includes a pop-up window page closing strategy as an example, after obtaining the pop-up window positioning area in the currently loaded web page, it is necessary to identify the image area corresponding to the pop-up window positioning area in the current web page screenshot image, that is, the pop-up window processing strategy can also include the target detection model used in the above pop-up window recognition model, and based on the target detection model, the close button of the image area corresponding to the pop-up window positioning area in the current web page screenshot image can be located (it can also be replaced by the recognition ignore button, "×" button, etc. in the specific implementation), thereby obtaining the close button positioning area (for example, corresponding to a rectangular area, and the coordinates of the four vertices of the rectangular area are also known). Afterwards, based on the pop-up window recognition processing subtask in the automation task, a simulated user clicks the close button operation (which can be understood as automatically moving the mouse icon in the currently loaded web page to the close button positioning area and clicking it) is generated to close the pop-up window page element corresponding to the pop-up window positioning area, and finally update the currently loaded web page after completing the closing of the pop-up window page element. It can be seen that, on the one hand, after the pop-up window positioning area is identified, the close button positioning area of ​​the pop-up window positioning area is identified by computer vision technology and the close button is automatically clicked, realizing the automatic identification and closing of the pop-up window page elements, avoiding the continued existence of the pop-up window page elements and the impact on the execution of the automated task. On the other hand, the pop-up window page elements are automatically processed through the pop-up window processing strategy, reducing the need for manual intervention and improving the degree of task automation.

[0096] In one embodiment, as a second embodiment of the pop-up window processing unit 130, the pop-up window processing unit 130 is specifically used for:

[0097] Obtain a pop-up page avoidance strategy in the pop-up window processing strategy;

[0098] Obtaining a current operation path for the currently loaded web page in the automation task;

[0099] Based on the pop-up window page avoidance strategy and the pop-up window positioning area, adjusting the path in the current operation path that passes through the pop-up window positioning area to a path that bypasses the pop-up window positioning area, so as to update the current operation path;

[0100] Based on the pop-up page avoidance strategy, a click-forbidden attribute is set for the pop-up page element corresponding to the pop-up positioning area to update the currently loaded web page.

[0101] In this embodiment, after detecting the pop-up window area in the currently loaded web page, in addition to automatically closing it based on the pop-up window page closing strategy in the pop-up window processing strategy, the current operation path for the currently loaded web page in the automated task can also be adjusted based on the pop-up window page avoidance strategy in the pop-up window processing strategy so that it avoids the pop-up window positioning area. Specifically, the path passing through the pop-up window positioning area in the current operation path is adjusted to a path that bypasses the pop-up window positioning area. For example, there is a section of the candidate operation path in the current operation path that is within the pop-up window positioning area. At this time, the candidate operation path (such as its starting point is point A and the end point is point B, and point A and point B are both points on the boundary of the pop-up window positioning area) can be adjusted to the area that bypasses along the boundary of the pop-up window positioning area, that is, starting from point A along the boundary of the pop-up window positioning area and always walking along the outer boundary until point B, thereby realizing the update of the candidate operation path, so that the latest candidate operation path is along the boundary of the pop-up window positioning area. After completing the adjustment of the current operation path for the currently loaded web page in the automated task, the pop-up page element corresponding to the pop-up positioning area can also be set to a prohibit-click attribute based on the pop-up page avoidance strategy, so that the click task simulated in the automated task will not be opened after clicking the pop-up page element corresponding to the pop-up positioning area, thereby avoiding affecting subsequent automated tasks.

[0102] The automated data processing unit 140 is used to perform corresponding automated data processing on the currently loaded web page based on the automated task to obtain a current web page content processing result if it is determined that the pop-up page element in the currently loaded web page is in a processed state.

[0103] In this embodiment, if it is determined that the pop-up page element in the currently loaded web page is in a processed state, it means that the pop-up page element in the currently loaded web page will no longer affect the automatic execution of the automated task, and at this time, the currently loaded web page is continuously subjected to corresponding automated data processing based on the automated task to obtain the current web page content processing result. For example, the web page information collection subtask, web page information verification subtask, etc. in the automated task are continuously executed to obtain the current web page content processing result.

[0104] Among them, when determining whether the pop-up window page element in the currently loaded web page is in a processed state, the processing process of steps S120-S130 can be referred to, that is, in step S130, after the pop-up window page element corresponding to the pop-up window positioning area has been processed accordingly and the currently loaded web page has been updated, the current web page screenshot image corresponding to the currently loaded web page is obtained again, and the pop-up window positioning area and pop-up window type recognition result corresponding to the current web page screenshot image are obtained based on the pre-trained pop-up window recognition model. When the pop-up window positioning area and the pop-up window type recognition results are both null values, it means that the pop-up window page element in the currently loaded web page is in a processed state; when the pop-up window positioning area or the pop-up window type recognition result is not a null value, it means that the pop-up window page element in the currently loaded web page is in an unprocessed state. It can be seen that the above-mentioned process of verifying whether the pop-up window page element has been processed can further determine whether the automated task can continue to be executed. Moreover, through computer vision technology, real-time monitoring and dynamic adjustment of the web page environment can be achieved to ensure the continuity and reliability of the execution of automated tasks.

[0105] In one embodiment, the automated data processing unit 140 is specifically used for:

[0106] A data collection task in the automation task is obtained, and the data collection task is executed to collect information in the currently loaded web page to obtain a processing result of the current web page content.

[0107] In this embodiment, if the data collection task in the automated task is continued to be executed at this time, the data collection task can be regarded as the web page information collection subtask in the above example. Through the automatic execution of the data collection task, the information in the currently loaded web page can be collected. Moreover, in the above information collection process, the pop-up page elements have been automatically identified and processed. The automated task can perform the information collection task more accurately, reduce click failures caused by pop-ups, and improve the efficiency of information collection and overall task execution.

[0108] In one embodiment, the automatic closing device 100 for the pop-up window of the automated task page further includes:

[0109] The repositioning processing unit is used to obtain the current web page screenshot image corresponding to the currently loaded web page if it is determined that the pop-up page element in the currently loaded web page is in an unprocessed state, and return to execute the step of obtaining the pop-up positioning area and pop-up type identification result corresponding to the current web page screenshot image based on the pre-trained pop-up recognition model.

[0110] In this embodiment, if it is determined that the pop-up page element in the currently loaded web page is in an unprocessed state, it means that the pop-up page element in the currently loaded web page has not been successfully processed, and it will also affect the automatic execution of the automated task. At this time, the current web page screenshot image corresponding to the currently loaded web page can be obtained again, and the step of obtaining the pop-up window positioning area and pop-up window type recognition result corresponding to the current web page screenshot image based on the pre-trained pop-up window recognition model is returned. That is, the above process is repeated at least once to process the pop-up page element in the currently loaded web page again. Moreover, when it is detected that the pop-up page element in the currently loaded web page is in an unprocessed state, log data corresponding to the state that the pop-up page element has not been processed is generated, and the log data is saved as an exception log. When the total number of log data included in the exception log exceeds the preset number of logs within a certain time interval (such as a month, a week, etc.), it means that the recognition accuracy of the pop-up window recognition model cannot meet the use requirements, and an update prompt information of a pop-up window recognition model can be automatically generated based on a preset exception handling mechanism, and the update prompt information is sent by the server to the designated receiving terminal. After receiving the update prompt information, the user of the receiving terminal can make timely adjustments to the pop-up window recognition model in the server.

[0111] The data storage unit 150 is configured to store the current web page content processing result in a preset storage space if it is determined that the current web page content processing result matches the pre-stored expected result.

[0112] In this embodiment, after obtaining the current web page content processing result by executing the automated task, it can also be compared with the pre-stored expected result to ensure the correctness of the collected data. After determining that the current web page content processing result matches the pre-stored expected result, the current web page content processing result can be stored in a preset storage space (such as a database of a server).

[0113] It can be seen that the implementation of the device can combine computer vision technology to quickly process the pop-up page elements detected on the currently loaded web page during the execution of the automated task, thereby ensuring the continuity and reliability of the execution of the automated task.

[0114] The above-mentioned automatic closing device for the pop-up window of the automated task page can be implemented in the form of a computer program. The computer program can be used in the following manner: Figure 8 Runs on the computer device shown.

[0115] See also Figure 8 , Figure 8The present invention is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device integrates any automatic task page pop-up window automatic closing device provided by an embodiment of the present invention.

[0116] See also Figure 8 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .

[0117] The storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute the above-mentioned method for automatically closing the pop-up window of the automated task page.

[0118] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0119] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned method for automatically closing the pop-up window of the automated task page.

[0120] The network interface 405 is used to communicate with other devices over the network. Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] The processor 402 is used to run the computer program 4032 stored in the memory to implement the above-mentioned method for automatically closing the pop-up window of the automated task page.

[0122] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0123] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0124] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the above-mentioned method for automatically closing a pop-up window on an automated task page.

[0125] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0127] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0128] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0130] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for automatically closing pop-up windows on automated task pages, characterized in that: include: In response to a webpage loading instruction generated in the automation task, jump to a currently loaded webpage corresponding to the webpage loading instruction, and obtain a current webpage screenshot image corresponding to the currently loaded webpage; Obtaining a pop-up window location area and a pop-up window type recognition result corresponding to the current webpage screenshot image based on a pre-trained pop-up window recognition model; If it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, then based on a preset pop-up window processing strategy, a pop-up window page element corresponding to the pop-up window positioning area is processed accordingly to update the currently loaded web page; If it is determined that the pop-up page element in the currently loaded web page is in a processed state, then performing corresponding automated data processing on the currently loaded web page based on the automated task to obtain a current web page content processing result; as well as If it is determined that the current webpage content processing result matches the pre-stored expected result, the current webpage content processing result is stored in a preset storage space.

2. The method according to claim 1, characterized in that The jumping to the currently loaded web page corresponding to the web page loading instruction and obtaining a current web page screenshot image corresponding to the currently loaded web page includes: The automation task obtains a current target web page address corresponding to the web page loading instruction, and jumps to the currently loaded web page based on the current target web page address; The automation task obtains an automatic screenshot instruction, and executes the automatic screenshot instruction to take a screenshot of the currently loaded web page to obtain a screenshot image of the current web page.

3. The method according to claim 1, characterized in that The obtaining of a pop-up window location area and a pop-up window type recognition result corresponding to the current webpage screenshot image based on a pre-trained pop-up window recognition model includes: Locating the pop-up window area in the current webpage screenshot image based on the target detection model in the pop-up window recognition model to obtain the pop-up window positioning area; Recognize the text in the pop-up window positioning area based on the text recognition model in the pop-up window recognition model to obtain a text recognition result; The text recognition result is identified and classified based on the classification model in the pop-up window recognition model to obtain the pop-up window type recognition result.

4. The method according to claim 1, characterized in that: The step of processing the pop-up window page element corresponding to the pop-up window positioning area based on a preset pop-up window processing strategy to update the currently loaded web page includes: Obtain a pop-up window page closing strategy in the pop-up window processing strategy; Based on the pop-up window page closing strategy, a close button of the pop-up window positioning area in the currently loaded web page is identified to obtain a close button positioning area; The close button positioning area is automatically clicked based on the automated task to close the pop-up window page element corresponding to the pop-up window positioning area, and the currently loaded web page is updated.

5. The method according to claim 1, characterized in that The step of processing the pop-up window page element corresponding to the pop-up window positioning area based on a preset pop-up window processing strategy to update the currently loaded web page includes: Obtain a pop-up page avoidance strategy in the pop-up window processing strategy; Obtaining a current operation path for the currently loaded web page in the automation task; Based on the pop-up window page avoidance strategy and the pop-up window positioning area, adjusting the path in the current operation path that passes through the pop-up window positioning area to a path that bypasses the pop-up window positioning area, so as to update the current operation path; Based on the pop-up page avoidance strategy, a click-forbidden attribute is set for the pop-up page element corresponding to the pop-up positioning area to update the currently loaded web page.

6. The method according to claim 1, characterized in that The performing corresponding automated data processing on the currently loaded web page based on the automated task to obtain a current web page content processing result includes: A data collection task in the automation task is obtained, and the data collection task is executed to collect information in the currently loaded web page to obtain a processing result of the current web page content.

7. The method according to claim 1, characterized in that After the step of, if it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, processing the pop-up window page element corresponding to the pop-up window positioning area based on a preset pop-up window processing strategy to update the currently loaded web page, the method further includes: If it is determined that the pop-up page element in the currently loaded web page is in an unprocessed state, obtain the current web page screenshot image corresponding to the currently loaded web page, and return to execute the step of obtaining the pop-up location area and pop-up type identification result corresponding to the current web page screenshot image based on the pre-trained pop-up recognition model.

8. An automatic closing device for pop-up windows on automated task pages, characterized in that: include: A webpage loading and screenshot unit, configured to respond to a webpage loading instruction generated in an automated task, jump to a currently loaded webpage corresponding to the webpage loading instruction, and obtain a current webpage screenshot image corresponding to the currently loaded webpage; A pop-up window positioning and recognition unit, used to obtain a pop-up window positioning area and a pop-up window type recognition result corresponding to the current webpage screenshot image based on a pre-trained pop-up window recognition model; A pop-up window processing unit, configured to, if it is determined that the pop-up window type identification result is one of the preset pop-up window types in the preset pop-up window type set, perform corresponding processing on the pop-up window page element corresponding to the pop-up window positioning area based on a preset pop-up window processing strategy to update the currently loaded web page; an automated data processing unit, configured to, if it is determined that the pop-up page element in the currently loaded web page is in a processed state, perform corresponding automated data processing on the currently loaded web page based on the automated task to obtain a current web page content processing result; as well as The data storage unit is used to store the current web page content processing result in a preset storage space if it is determined that the current web page content processing result matches the pre-stored expected result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for automatically closing the pop-up window of an automated task page according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the method for automatically closing an automated task page pop-up window as described in any one of claims 1 to 7 can be implemented.

Images