A webpage automatic processing method and device

Abstract

The application discloses a webpage automation processing method and device, and relates to the technical field of data processing.The method comprises the following steps: determining a schedulable set according to the health state of each browser container in a container pool; determining a target container from the schedulable set in response to a webpage processing task; inputting target session data into the browser context of the target container; driving the browser in the target container to execute webpage automation processing in the browser context of the target container; when an abnormality is detected, marking the target container as an abnormal state and removing the target container from the schedulable set; starting a remote graphical access interface of the target container to enable manual takeover processing, and rechecking the target container after the takeover processing is completed; when the rechecking is passed, marking the target container as a health state and adding the target container to the schedulable set. Thus, the application can reduce the consumption of time and computing resources when an abnormality occurs in a processing browser instance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for automated web page processing. Background Technology

[0002] With the deepening development of internet applications, web page automation technology has been widely used in scenarios such as data collection, business process automation, and monitoring. During long-term, large-scale execution of web page automation tasks, browser instances inevitably encounter various anomalies, such as page access verification and request timeouts. The common approach to handling these anomalies is to directly discard the browser instance that encountered the anomaly and then create a completely new browser instance to continue executing the task.

[0003] However, on the one hand, abandoning a browser instance means that the computing resources it previously occupied, such as memory, CPU resources, and network connections, are wasted, and the valid session state, login credentials, and page browsing context previously established by the browser instance are also lost. On the other hand, creating a completely new browser instance requires going through the entire process of image startup, browser process initialization, desktop environment readiness, and network link establishment again, which consumes a lot of time and computing resources again.

[0004] Therefore, how to reduce the consumption of time and computing resources when handling abnormal situations in browser instances has become an urgent technical problem to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and apparatus for automated webpage processing, which can reduce the consumption of time and computing resources when handling abnormal situations that occur in browser instances.

[0006] The embodiments of this application disclose the following technical solutions: In a first aspect, this application discloses a method for automated webpage processing, the method comprising: The schedulable set is determined based on the health status of each browser container in the container pool; the container pool consists of multiple continuously running browser containers. In response to a web page processing task, a target container is determined from the schedulable set; Acquire target session data and input the target session data into the browser context of the target container; the target session data is pre-generated by the local real browser; Drive the browser within the target container to perform automated webpage processing within the browser context of the target container; When a processing anomaly is detected, the target container is marked as an anomaly and removed from the schedulable set; Enable the remote graphical access interface of the target container so that manual takeover processing can be performed in the browser context and session context of the target container, and the target container can be re-inspected after the takeover processing is completed. When the re-inspection passes, the target container is marked as healthy and added to the schedulable set.

[0007] Optionally, determining the schedulable set based on the health status of each browser container in the container pool includes: Get the health status and task interface access status of each browser container in the container pool; Browser containers whose health status indicates normal and whose task interface access status indicates connectable are identified as the schedulable set.

[0008] Optionally, the processing exception includes at least one of the following: the page content returned by the automated webpage processing includes a preset exception title string; the task application programming interface (API) call of the target container exceeds a preset timeout period; the network request of the target container returns a connection timeout or connection failure error; the browser process of the target container exits abnormally.

[0009] Optionally, the re-inspection of the target container includes: The target container is re-inspected by either driving the browser of the target container to access a preset health check page or by driving the browser of the target container to re-execute automated web page processing in the browser context of the target container.

[0010] Optionally, the method further includes: Receive a network switching instruction for the target container; During the switching action of the target container, the target container is removed from the schedulable set; When the target container has been switched over and passed the re-inspection, the target container is added to the schedulable set.

[0011] Secondly, this application discloses a webpage automated processing device, the device comprising: a first determining module, a second determining module, a data acquisition module, a processing driving module, a container removal module, a container re-inspection module, and a container addition module; The first determining module is used to determine a schedulable set based on the health status of each browser container in the container pool; the container pool consists of multiple continuously running browser containers. The second determining module is configured to determine a target container from the schedulable set in response to a web page processing task; The data acquisition module is used to acquire target session data and input the target session data into the browser context of the target container; the target session data is pre-generated by the local real browser; The processing driver module is used to drive the browser within the target container to perform automated webpage processing in the browser context of the target container. The container removal module is used to mark the target container as an abnormal state and remove the target container from the schedulable set when a processing abnormality is detected. The container re-inspection module is used to enable the remote graphical access interface of the target container, so that a person can take over the target container in the browser context and session context, and re-inspect the target container after the takeover is completed. The container re-inspection module is used to mark the target container as healthy when the re-inspection passes, and to add the target container to the schedulable set.

[0012] Optionally, the first determining module is specifically used to: obtain the health status and task interface access status of each browser container in the container pool; and determine the browser containers whose health status indicates normal and whose task interface access status indicates connectable as a schedulable set.

[0013] Optionally, the processing exception includes at least one of the following: the page content returned by the automated webpage processing includes a preset exception title string; the task application programming interface (API) call of the target container exceeds a preset timeout period; the network request of the target container returns a connection timeout or connection failure error; the browser process of the target container exits abnormally.

[0014] Optionally, the container re-inspection module is specifically used to: re-inspect the target container by driving the browser of the target container to access a preset health check page, or by driving the browser of the target container to re-execute webpage automation processing in the browser context of the target container.

[0015] Optionally, the device further includes: a network switching module; The network switching module is configured to receive a network switching instruction for the target container; remove the target container from the schedulable set during the switching operation of the target container; and add the target container to the schedulable set when the switching of the target container is completed and the re-inspection is passed.

[0016] Compared with the prior art, this application has the following beneficial effects: This application discloses a method and apparatus for automated webpage processing. The method includes: determining a schedulable set based on the health status of each browser container in a container pool; the container pool consists of multiple continuously running browser containers; in response to a webpage processing task, determining a target container from the schedulable set; acquiring target session data and inputting the target session data into the browser context of the target container; the target session data is pre-generated by a local real browser; driving the browser within the target container to perform automated webpage processing in the browser context of the target container; when a processing anomaly is detected, marking the target container as an abnormal state and removing the target container from the schedulable set; enabling a remote graphical access interface for the target container, allowing manual takeover processing in the browser context and session context of the target container, and performing a re-inspection of the target container after the takeover processing is completed; when the re-inspection passes, marking the target container as healthy and adding the target container to the schedulable set. Therefore, this application reduces the time cost and computational resource consumption of exception handling by maintaining a container pool consisting of multiple continuously running browser containers, removing the target container from the schedulable set of the container pool when an anomaly is detected, having it manually taken over and handled in situ in the target container, and adding it back to the schedulable set after the handling is completed and passed the re-inspection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a webpage automation processing method provided in this application embodiment; Figure 2 This is a schematic diagram of a web page automation processing device provided in an embodiment of this application. Detailed Implementation

[0019] As described earlier, on the one hand, abandoning a browser instance means that the computing resources it previously occupied, such as memory, CPU resources, and network connections, are wasted, and the valid session state, login credentials, and page browsing context previously established by the browser instance are also lost. On the other hand, creating a completely new browser instance requires going through the entire process of image startup, browser process initialization, desktop environment readiness, and network link establishment again, which consumes a lot of time and computing resources again.

[0020] Through research, the inventors have provided a method and apparatus for automated webpage processing. This application maintains a container pool consisting of multiple continuously running browser containers, and when an anomaly is detected in a target container, the target container is removed from the schedulable set of the container pool and manually taken over the processing in situ within the target container. After processing is completed and a re-inspection is passed, the target container is added back to the schedulable set, thereby reducing the time cost and computational resource consumption of anomaly handling.

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0022] See Figure 1 The figure is a flowchart of a webpage automation processing method provided in an embodiment of this application. The method includes: S101: Determine the schedulable set based on the health status of each browser container in the container pool; the container pool consists of multiple continuously running browser containers.

[0023] The container pool consists of multiple continuously running browser containers, each existing as an independently schedulable web page processing execution unit. Each browser container can not only independently execute web page processing tasks, but also be isolated, taken over, and recovered individually in the event of an exception.

[0024] In the automated webpage processing method provided in this application embodiment, firstly, the health status and task interface access status of each browser container are determined from the instance information maintained by each browser container in the container pool. The instance information includes container identifier, running status, health status, task interface access status, graphics access port, and network egress parameters. Specifically, the health status is used to comprehensively assess whether the browser container can safely execute tasks, and the task interface access status is used to characterize whether the browser container can normally receive webpage processing tasks issued externally. Subsequently, based on the health status and task interface access status of each browser container, browser containers with a normal health status and a connectable task interface access status are determined as the schedulable set. In other words, this application embodiment moves the determination of whether a browser container can receive tasks to the scheduling stage, thereby avoiding sending webpage processing tasks to browser containers that are already abnormal or temporarily unavailable.

[0025] In one specific implementation, the health status can be obtained by combining multiple detection conditions. For example, the health status indicates normal only when the browser container meets N (N is a positive integer less than or equal to 5) of the following five conditions: Condition 1: The browser container is running and not marked as abnormal; Condition 2: The application programming interface (API) within the browser container is accessible normally; Condition 3: The browser process within the browser container is executable; Condition 4: The graphics access service within the browser container is available; Condition 5: The network exit point of the browser container is available. It should be noted that this application does not limit the specific detection conditions.

[0026] Therefore, before task scheduling, this application first filters the container pool according to the health status of the browser containers, and only retains the browser containers that meet the execution conditions as the schedulable set. This can avoid assigning web page processing tasks to browser containers that are abnormal, waiting to be recovered, or temporarily unavailable, reduce the task failure rate, and improve the overall task execution efficiency and operational stability of the container pool.

[0027] S102: In response to a web page processing task, determine the target container from the schedulable set.

[0028] Understandably, since all browser containers in the schedulable set are in good health and have accessible task interfaces, the selected target container can directly take over the web page processing task and execute subsequent automated web page processing procedures.

[0029] Web page processing tasks can be initiated by external business systems, scheduling services, or upper-layer applications. A web page processing task includes at least one of the following task characteristic information: target web page address, processing action type, task timeout parameters, target site identifier, session requirement information, and network environment requirement information.

[0030] In the webpage automated processing method provided in this application embodiment, after receiving the webpage processing task, the target container is determined from the schedulable set by reading the task feature information and based on a preset scheduling strategy.

[0031] In one specific implementation, in response to a web page processing task, a target container for executing the web page processing task is determined from the schedulable set based on a preset scheduling strategy, and the web page processing task is assigned to that target container for execution. In another specific implementation, in response to multiple concurrent web page processing tasks, multiple target containers for executing web page processing tasks are determined from the schedulable set based on a preset scheduling strategy, and the web page processing tasks are assigned to these multiple target containers for execution. This application does not limit the specific implementation method.

[0032] In one specific implementation, the preset scheduling strategy may include at least one of the following: a load-based scheduling strategy (for prioritizing browser containers with lower current task load as target containers), a round-robin scheduling strategy (for selecting browser containers as target containers sequentially according to a preset order), a priority-based scheduling strategy (for selecting browser containers as target containers after sorting them based on their historical success rate, stability, or response performance), a network egress-based scheduling strategy (for selecting browser containers with matching network egress parameters as target containers based on the network environment requirements of the target webpage processing task), and a session-matching scheduling strategy (for prioritizing browser containers that have established a corresponding site session context or can carry the corresponding site session data as target containers). Therefore, while ensuring the availability of browser containers, the stability and processing efficiency of task allocation can be further improved. It should be noted that this application does not limit the specific preset scheduling strategy.

[0033] S103: Obtain the target session data and input the target session data into the browser context of the target container; the target session data is pre-formed by the local real browser.

[0034] The target session data is not a temporarily constructed fake session data, but rather the actual session data formed after the local real browser has accessed the target site in advance.

[0035] In the webpage automation processing method provided in this application embodiment, obtaining target session data includes the following steps: First, obtaining the original session data of the local real browser. The original session data refers to the target container-related session data file generated and saved by the local real browser in its local configuration directory. Second, extracting the session record corresponding to the target container from the original session data. Then, obtaining the key material required for decryption based on the local operating system environment. Finally, using the key material, decrypting the encrypted session fields in the session record to obtain the decrypted target session data. The decrypted target session data includes, in addition to the cookie value itself, at least one of the following: domain name, path, security attributes, HttpOnly attribute, SameSite attribute, and expiration information.

[0036] After obtaining the decrypted target session data, it is converted into a session write structure recognizable by the browser automation engine and written to the browser context of the target container. It should be noted that during the conversion process, at least one of the following must be retained: the domain name, path, security attributes, HttpOnly attribute, and SameSite attribute. This ensures that the written target session data has the same scope and security constraints in the target container's browser context as it does in the local browser.

[0037] In one specific implementation, after writing the target session data into the browser context of the target container, it is possible to verify whether the target session data was written successfully, to ensure that the browser within the target container retains the authorization state formed by the local real browser when accessing the target webpage. For example, verifying whether the target session data was written successfully includes checking whether the preset cookie was written successfully, whether a logged-in status is displayed after accessing the target site, and whether at least one of the following is not displayed: a re-login page or an unauthorized prompt page. It should be noted that this application does not limit the specific method for verifying whether the target session data was written successfully.

[0038] Therefore, through the above method, the browser within the target container can directly reuse the authorization state, login state, or session state pre-formed by the local real browser during subsequent page loading, request sending, and page interaction, without needing to re-execute the login process or reconstruct the session within the target container. This improves session continuity during automated web page processing and reduces the time and resource consumption caused by repeated logins, verifications, or browsing state initializations. Furthermore, the target session data, after being written to the browser context of the target container, is not only used for a single page access but can be continuously reused in subsequent automated processing, anomaly detection, manual intervention, and review within the target container. In other words, from the start of automated task execution to manual intervention after an anomaly, and then to container review after intervention, all can be performed continuously within the same target container, the same browser instance, and the same session context, thereby improving session continuity and reducing the time and resource overhead of repeated initialization operations.

[0039] S104: Drives the browser within the target container to perform automated web page processing within the browser context of the target container.

[0040] It is understandable that since the target session data has been pre-written into the browser context in step S103, the automated web page processing in step S104 can be performed while maintaining the authorization status and the continuity of the browsing context.

[0041] In the webpage automation processing method provided in this application embodiment, the automation execution module of the target container controls the browser in the target container to perform webpage automation processing in the browser context of the target container by calling the browser automation engine.

[0042] The browser automation engine can be chromedp, Selenium, Playwright, or other automation control components that can drive a real browser. Web page automation processing includes at least one of the following: page opening, element positioning, click operations, input operations, scrolling operations, waiting for page to load, data extraction, result submission, and page redirection.

[0043] The automated execution module operates as follows when performing automated webpage processing: First, it obtains the target Uniform Resource Locator (URL), action type, timeout parameters, and result extraction rules from the webpage processing task. Second, it invokes the browser automation engine to control the browser within the target container to access the target URL. Third, the target session data, already written to the browser context before accessing the target URL, automatically takes effect with the page request, ensuring the browser maintains its existing login or authorization state during page loading and interaction. Fourth, it executes preset page interaction actions according to the task requirements of the webpage processing task. Fifth, it reads the page content, API return results, or target element information to form the task processing result. The task processing result includes at least one of the following: page source code, structured field extraction results, page screenshots, execution logs, task execution status, and page return data.

[0044] Understandably, the automated execution module does not directly manipulate external resources of the container. Instead, it completes browser control, page rendering, session usage, and result extraction within the target container, thus ensuring that the task execution process is always confined to the same container, the same browser instance, and the same session context. This structure facilitates direct state switching, manual takeover, and recovery of the current target container in the event of subsequent page errors, access verification, execution errors, timeouts, or browser malfunctions, without requiring the creation of a new browser environment or the re-import of the session.

[0045] In one specific implementation, the automated web page processing also includes execution status monitoring. Execution status monitoring is used to detect whether abnormal pages, execution errors, network errors, or browser errors occur during the current automated web page processing, so as to promptly trigger subsequent container removal and manual takeover processes when an error is detected.

[0046] S105: When a processing exception is detected, mark the target container as an exception and remove the target container from the schedulable set.

[0047] In the automated web page processing method provided in this application embodiment, when a processing abnormality is detected in the target container, the target container is not directly destroyed, nor is a new browser instance immediately created to replace the current target container. Instead, the target container is first marked as abnormal and removed from the schedulable set to prevent it from receiving new web page processing tasks. Thus, the impact of the abnormality can be limited to a single target container without affecting other healthy browser containers in the container pool from continuing to execute tasks.

[0048] In one specific implementation, exception handling includes at least one of the following: First, the page content returned by the automated webpage processing includes a preset exception title string. Second, the target container's task application programming interface (API) call exceeds a preset timeout period. Third, the target container's network request returns a connection timeout or connection failure error. Fourth, the target container's browser process exits abnormally. Any of the above situations can serve as triggering conditions for removing the target container from the schedulable set and entering the manual takeover process. It should be noted that this application does not limit the specific method for handling exceptions.

[0049] In one specific implementation, when a processing anomaly is detected, runtime information related to the anomaly can also be recorded, including but not limited to at least one of the following: anomaly type, anomaly occurrence time, task identifier, target webpage address, and container identifier information, for subsequent anomaly analysis, problem localization, manual intervention, or container recovery processing.

[0050] In one specific implementation, the target container can be marked as abnormal by modifying at least one of the health status field, schedulable status field, abnormal status flag field, and container name identifier corresponding to the target container, so that the target container will no longer participate in the scheduling of subsequent web page processing tasks.

[0051] S106: Enable the remote graphical access interface of the target container so that manual takeover processing can be performed in the browser context and session context of the target container, and the target container can be re-inspected after the takeover processing is completed.

[0052] In the webpage automation processing method provided in this application embodiment, step S106 is used to open the remote graphical access portal corresponding to the target container after the target container is marked as abnormal and removed from the schedulable set, without destroying the target container, rebuilding the browser instance, or losing the current session state, so that the human operator can directly enter the graphical browser runtime environment of the target container to take over the processing.

[0053] The remote graphical access interface can be a Virtual Network Computing (VNC) interface, a remote desktop interface, a Web graphical access interface, or other interface forms that can enter the target container's graphical browser runtime environment. Manual intervention can include at least one of the following: completing verification page interactions, confirming page prompts, continuing page redirection, correcting input status, refreshing the page, or re-triggering a specific operation.

[0054] It is understandable that manual takeover is performed within the original target container, the original browser instance, and the original session context. Since the page state, loaded resources, login state, cookie state, form input state, page navigation history, and browsing context before the exception occurred are all preserved in the current target container, manual operators do not need to recreate the browser environment, re-import the session, or re-execute previous page operations; they can continue processing based on the current exception situation. Therefore, compared to directly discarding the instance and creating a new one after an exception, this application can more effectively preserve the browsing state and session continuity at the time of the exception, thereby reducing recovery time and computational resource consumption.

[0055] In the webpage automation processing method provided in this application embodiment, after the takeover processing is completed, the target container is re-inspected to determine whether the target container has been restored to a state where it can continue to execute automation tasks.

[0056] In one specific implementation, the target container is re-checked, including at least one of the following: First, the target container is re-checked by accessing a preset health check page through the browser that drives the target container. The health check page can be a pre-set test page, connectivity verification page, login status verification page, or other page that reflects whether the browser, network, and session status are normal. The target container is judged to have recovered by detecting whether the health check page is successfully accessed, whether it completes loading within a preset time, and whether it returns preset normal content. Second, the target container is re-checked by re-executing automated web page processing within the browser context of the target container through the browser that drives the target container. This can involve re-executing the original web page processing task that caused the exception, or executing a preset lightweight test task to detect whether the browser instance, automated execution module, network link, and current session context of the target container have recovered to a usable state. It should be noted that the above re-checking methods can be used individually or in combination. This application does not limit the specific re-checking method.

[0057] In one specific implementation, passing the re-inspection may include at least one of the following: the browser process within the target container is running normally; the health check page can be successfully accessed and returns preset normal content within a preset time; the preset abnormal title string no longer exists in the page content of the health check page; the task interface can return normally within a preset time limit; network requests have not experienced connection timeouts or connection failures; and the current session state remains valid. Subsequent pooling operations are only performed when the re-inspection result indicates that the target container has recovered its normal processing capabilities. It should be noted that this application does not limit the form in which the re-inspection passes.

[0058] S107: When the re-inspection passes, mark the target container as healthy and add it to the schedulable set.

[0059] In the webpage automated processing method provided in this application embodiment, when the re-inspection result in step S106 shows that the target container has recovered its normal execution capability, the target container is remarked as healthy and added back to the schedulable set so that it can re-participate in the scheduling and execution of subsequent webpage processing tasks.

[0060] In one specific implementation, the target container can be switched from an abnormal state to a healthy state by updating at least one of the following methods: the health status field, the schedulable status field, the abnormal status flag field, or the instance information record. Therefore, after manual takeover and re-inspection, the target container is not directly discarded but is reintegrated into the container pool for continuous reuse, thereby reducing the overhead of image startup, browser initialization, desktop environment readiness, and network link establishment required to recreate the browser container instance.

[0061] The webpage automation processing method provided in this application further includes: First, receiving a network switching instruction for a target container. The network switching instruction instructs the execution of at least one of the following operations: network interface switching, IP address change, network segment migration, proxy switching, or service routing switching. Subsequently, since the target container may experience network instability or temporary unavailability during the network switching operation, the target container is removed from the schedulable set (or marked as temporarily unschedulable to prevent new tasks from entering the target container during the network switching) during the switching operation. Finally, when the target container completes the switching and passes the re-inspection, the target container is added back to the schedulable set. Thus, this application incorporates network exit switching into the state management process of browser container instances, creating a linkage between network environment changes, container availability judgment, and task scheduling control, which is beneficial for improving the stability and manageability of the container pool in long-term operating scenarios.

[0062] It should be noted that each browser container includes at least the following components: a graphical browser process, a browser automation execution module, a task interface service, a remote graphical access service, and a network egress control module. The task interface service receives web page processing tasks from an external scheduling module and hands them over to the browser automation execution module. The browser automation execution module drives the graphical browser to perform page access, session writing, page interaction, and result extraction. The remote graphical access service provides a graphical access channel for manual intervention when the target container encounters an anomaly. The network egress control module adjusts and switches the network access links of the browser container at the container level. In this structure, the browser container is both the execution unit for automated web page processing and the management unit for anomaly isolation, manual intervention, status review, and recovery back to the pool. By integrating browser control capabilities, manual graphical access capabilities, network link control capabilities, and task interface capabilities within the same container, the target container can achieve better independence, maintainability, and reusability in long-term operating scenarios.

[0063] In summary, this application discloses an automated webpage processing method. This application maintains a container pool consisting of multiple continuously running browser containers, and when an anomaly is detected in a target container, the target container is removed from the schedulable set of the container pool and manually taken over the processing in situ within the target container. After the processing is completed and passed the re-inspection, the target container is added back to the schedulable set, thereby reducing the time cost and computational resource consumption of anomaly handling.

[0064] See Figure 2 The figure is a schematic diagram of a webpage automated processing device provided in an embodiment of this application. The webpage automated processing device 200 includes: a first determining module 201, a second determining module 202, a data acquisition module 203, a processing driving module 204, a container removal module 205, a container re-inspection module 206, and a container addition module 207.

[0065] The first determining module 201 is used to determine the schedulable set based on the health status of each browser container in the container pool; the container pool consists of multiple continuously running browser containers. The second determining module 202 is used to determine the target container from the schedulable set in response to the web page processing task; The data acquisition module 203 is used to acquire target session data and input the target session data into the browser context of the target container; the target session data is pre-formed by the local real browser; Processing driver module 204 is used to drive the browser within the target container to perform automated web page processing in the browser context of the target container; The container removal module 205 is used to mark the target container as an abnormal state and remove the target container from the schedulable set when a processing abnormality is detected. The container re-inspection module 206 is used to open the remote graphical access interface of the target container so that manual takeover processing can be performed in the browser context and session context of the target container, and the target container can be re-inspected after the takeover processing is completed. The container addition module 207 is used to mark the target container as healthy when the re-inspection passes and add the target container to the schedulable set.

[0066] In one specific implementation, the first determining module 201 is specifically used to: obtain the health status and task interface access status of each browser container in the container pool; and determine the browser containers whose health status is normal and whose task interface access status is connectable as a schedulable set.

[0067] In one specific implementation, exception handling includes at least one of the following: the page content returned by the automated web page processing includes a preset exception title string; the target container's task application programming interface (API) call exceeds a preset timeout period; the target container's network request returns a connection timeout or connection failure error; the target container's browser process exits abnormally.

[0068] In one specific implementation, the container re-inspection module 206 is specifically used to: access a preset health check page through the browser that drives the target container, or to re-execute automated web page processing in the browser context of the target container through the browser that drives the target container, and re-inspect the target container.

[0069] In one specific implementation, the web page automation processing device 200 further includes: a network switching module; The network switching module is used to receive network switching instructions for the target container; remove the target container from the schedulable set during the switching action of the target container; and add the target container to the schedulable set when the switching is completed and the re-inspection is passed.

[0070] In summary, this application discloses an automated webpage processing device. This application maintains a container pool consisting of multiple continuously running browser containers, and when an anomaly is detected in a target container, removes the target container from the schedulable set of the container pool, allows manual intervention in the target container to handle it in situ, and adds it back to the schedulable set after processing and re-inspection. This reduces the time cost and computational resource consumption of anomaly handling.

[0071] This application also provides corresponding electronic devices and computer-readable media for implementing the web page automation processing method provided in this application.

[0072] The electronic device includes a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code to cause the device to perform a web page automation processing method according to any embodiment of this application.

[0073] It should be noted that, in the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0074] It should be noted that the machine-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0075] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0076] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0077] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0078] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for automated webpage processing, characterized in that, The method includes: The schedulable set is determined based on the health status of each browser container in the container pool; the container pool consists of multiple continuously running browser containers. In response to a web page processing task, a target container is determined from the schedulable set; Acquire target session data and input the target session data into the browser context of the target container; the target session data is pre-generated by the local real browser; Drive the browser within the target container to perform automated webpage processing within the browser context of the target container; When a processing anomaly is detected, the target container is marked as an anomaly and removed from the schedulable set; Enable the remote graphical access interface of the target container so that manual takeover processing can be performed in the browser context and session context of the target container, and the target container can be re-inspected after the takeover processing is completed. When the re-inspection passes, the target container is marked as healthy and added to the schedulable set.

2. The method according to claim 1, characterized in that, The step of determining the schedulable set based on the health status of each browser container in the container pool includes: Get the health status and task interface access status of each browser container in the container pool; Browser containers whose health status indicates normal and whose task interface access status indicates connectable are identified as the schedulable set.

3. The method according to claim 1, characterized in that, The exception handling includes at least one of the following: the page content returned by the automated web page processing includes a preset exception title string; the target container's task application programming interface (API) call exceeds a preset timeout period; the target container's network request returns a connection timeout or connection failure error; the target container's browser process exits abnormally.

4. The method according to claim 1, characterized in that, The re-inspection of the target container includes: The target container is re-inspected by either driving the browser of the target container to access a preset health check page or by driving the browser of the target container to re-execute automated web page processing in the browser context of the target container.

5. The method according to claim 1, characterized in that, The method further includes: Receive a network switching instruction for the target container; During the switching action of the target container, the target container is removed from the schedulable set; When the target container has been switched over and passed the re-inspection, the target container is added to the schedulable set.

6. A webpage automated processing device, characterized in that, The device includes: a first determining module, a second determining module, a data acquisition module, a processing driving module, a container removal module, a container re-inspection module, and a container addition module; The first determining module is used to determine a schedulable set based on the health status of each browser container in the container pool; the container pool consists of multiple continuously running browser containers. The second determining module is configured to determine a target container from the schedulable set in response to a web page processing task; The data acquisition module is used to acquire target session data and input the target session data into the browser context of the target container; the target session data is pre-generated by the local real browser; The processing driver module is used to drive the browser within the target container to perform automated webpage processing in the browser context of the target container. The container removal module is used to mark the target container as an abnormal state and remove the target container from the schedulable set when a processing abnormality is detected. The container re-inspection module is used to open the remote graphical access interface of the target container so that a human can take over the target container in the browser context and session context, and re-inspect the target container after the takeover is completed. The container re-inspection module is used to mark the target container as healthy when the re-inspection passes, and to add the target container to the schedulable set.

7. The apparatus according to claim 6, characterized in that, The first determining module is specifically used to: obtain the health status and task interface access status of each browser container in the container pool; and determine the browser containers whose health status indicates normal and whose task interface access status indicates connectable as a schedulable set.

8. The apparatus according to claim 6, characterized in that, The exception handling includes at least one of the following: the page content returned by the automated web page processing includes a preset exception title string; the target container's task application programming interface (API) call exceeds a preset timeout period; the target container's network request returns a connection timeout or connection failure error; the target container's browser process exits abnormally.

9. The apparatus according to claim 6, characterized in that, The container re-inspection module is specifically used to: re-inspect the target container by driving the browser of the target container to access a preset health check page, or by driving the browser of the target container to re-execute webpage automation processing in the browser context of the target container.

10. The apparatus according to claim 6, characterized in that, The device further includes: a network switching module; The network switching module is configured to receive a network switching instruction for the target container; remove the target container from the schedulable set during the switching operation of the target container; and add the target container to the schedulable set when the switching of the target container is completed and the re-inspection is passed.

Images