Distributed browser collection method based on docker and related equipment

Abstract

The invention discloses a docker-based distributed browser acquisition method and related equipment, and relates to the field of data acquisition frameworks. A total demand is divided into a plurality of sub-tasks which are processed by using a docker container, the use of the docker container ensures that each task has an independent running environment and network configuration, and the system can simultaneously access a target website through different IP addresses of different nodes in cooperation with the deployment of distributed nodes. According to the distributed access mode, the access frequency of a single IP is remarkably reduced, and the access limitation of the target website is effectively avoided. Meanwhile, the independent browser environment can simulate the user access behavior more truly, and the stability and reliability of the collection task are improved.

Description

Technical Field

[0001] The present application relates to the field of data collection framework, and in particular to a Docker-based distributed browser collection method and related equipment. Background Art

[0002] In the wave of informatization in modern society, artificial intelligence generated content (AIGC) is playing an increasingly critical role in the production activities of many industries. As the cornerstone of the AIGC field, data collection plays an important role. It is responsible for efficiently collecting massive data from the Internet and providing the necessary training materials and information resources for the AIGC system. After careful processing and analysis, these data can be used to cultivate more advanced and accurate artificial intelligence models. These models have achieved automated operations and innovative applications in many fields such as content creation, language translation, and data analysis.

[0003] With the continuous advancement of data collection technology, AIGC's capabilities have been significantly enhanced, enabling it to demonstrate even more outstanding performance in personalized and high-quality content production. This not only improves production efficiency, but also brings users a richer and more accurate personalized experience. Therefore, the innovation and development of data collection technology is of vital importance to promoting the growth and progress of the AIGC field.

[0004] In traditional data collection systems, the collection program interacts directly with the target website server through a browser and uses a single request-response model to obtain data. This method configures access parameters in the collection program, directly sends data requests to the target website, obtains the required web page content, and stores the collected data in a local database.

[0005] However, this direct access method faces many limitations in practical applications. Since the target website usually sets access frequency limits and IP blocking mechanisms, when the collection requests are too frequent, the IP address of the collection program is easily identified and blocked by the target website, resulting in the interruption of the collection task. At the same time, a single access mode is also difficult to simulate real user browsing behavior and is easily identified by the website's anti-crawler system. Therefore, the data collection capabilities of related technologies are relatively poor. Summary of the invention

[0006] The present application provides a Docker-based distributed browser collection method and related equipment to improve data collection capabilities.

[0007] On the one hand, the present application provides a distributed browser collection method based on Docker, including: splitting the total demand data into multiple sub-demand data, creating sub-tasks according to the sub-demand data, the sub-tasks including the task link and collection parameters of the target website; building corresponding Docker containers based on the sub-tasks, integrating the browser operating environment in the Docker container; distributing the sub-tasks and the corresponding Docker containers to distributed nodes, so that the distributed nodes run the Docker containers to execute the sub-tasks and collect website data; storing the collected website data in corresponding storage areas according to different types.

[0008] By adopting the above technical solution, the total demand is divided into multiple subtasks and processed using docker containers. The use of docker containers ensures that each task has an independent operating environment and network configuration. Combined with the deployment of distributed nodes, the system can access the target website simultaneously through different IP addresses of different nodes. This decentralized access method significantly reduces the access frequency of a single IP and effectively circumvents the access restrictions of the target website. At the same time, the independent browser environment can more realistically simulate user access behavior and improve the stability and reliability of the collection task.

[0009] In combination with some embodiments of the first aspect, in some embodiments, the step of splitting the total demand data into multiple sub-demand data specifically includes: splitting the total demand data into multiple partial demand data according to the web page address characteristics; splitting the partial demand data into multiple sub-demand data according to preset rules; after the step of creating subtasks according to the sub-demand data, the method also includes: storing the subtasks in different levels of the multi-level task queue according to the partial demand data to which they belong, and sorting the subtasks in the same level according to the order of storage time; sorting the levels of the multi-level task queue according to the priority of the partial demand data; the step of distributing the subtasks and the corresponding docker containers to distributed nodes specifically includes: distributing tasks in sequence according to the level order of the multi-level task queue, giving priority to distributing subtasks in the high-level queue, and when the subtasks in the high-level queue are distributed, transferring them to the subtasks in the low-level queue in sequence.

[0010] By adopting the above technical solution, the demand data is preliminarily classified by the characteristics of the web page address, and further subdivided according to the preset rules, so that the system can handle the collection tasks more orderly. The design of the multi-level task queue allows high-priority tasks to be executed first, and the processing of tasks of the same level in chronological order ensures that some subtasks will not wait too long. This allows the system to flexibly dispatch resources according to the importance of the business, ensuring that important tasks are completed in a timely manner while not causing low-priority tasks to be shelved for a long time, thereby improving the overall task processing efficiency.

[0011] In combination with some embodiments of the first aspect, in some embodiments, the step of distributing tasks in sequence according to the hierarchical order of the multi-level task queue specifically includes: selecting multiple subtasks from the same-level queue of the multi-level task queue, and assigning the multiple subtasks to the same distributed node; after the distributed node receives the multiple subtasks, starting multiple execution threads to perform concurrent processing to monitor the task execution status of the distributed node, and when the distributed node completes the subtask processing of the current-level queue, continuing to assign other subtasks in the same-level queue to it, until the subtasks of the current-level queue are assigned, and then assigning the subtasks of the next-level queue.

[0012] By adopting the above technical solution, a multi-thread concurrent processing mechanism is adopted in the task distribution process. Multiple subtasks are assigned to the same node and multiple execution threads are started, making full use of the computing resources of the node.

[0013] In combination with some embodiments of the first aspect, in some embodiments, after the step of storing the collected website data in corresponding storage areas according to different types, the method also includes: performing integrity verification on the page content of the website data, the integrity verification including detection of the web page title and page structure; when the integrity verification fails, rewriting the corresponding subtask into the multi-level task queue; before the corresponding subtask, controlling the corresponding distributed node to switch the network proxy resources.

[0014] By adopting the above technical solution, the system verifies the integrity of the collected data by detecting the web page title and page structure. When the data is found to be abnormal, the task retry mechanism will be triggered. During the retry process, the system will actively switch network proxy resources. This dynamic adjustment strategy not only improves the success rate of the retry task, but also effectively circumvents the access restrictions of the target website.

[0015] In combination with some embodiments of the first aspect, in some embodiments, after the step of distributing subtasks and corresponding docker containers to distributed nodes, the method also includes: creating a network environment, setting the network driver type to bridge mode, and obtaining a custom bridge network; allocating an IP address range to the custom bridge network, and allocating addresses within the IP address range to each docker container, so that the docker containers communicate through internal IP addresses; and configuring the network isolation policy of the docker container.

[0016] By adopting the above technical solutions, the network environment of Docker containers can be managed in a refined manner. The network configuration of bridge mode enables efficient internal communication between containers, while independent IP address allocation ensures network isolation between containers.

[0017] In combination with some embodiments of the first aspect, in some embodiments, after the step of distributing subtasks and corresponding docker containers to distributed nodes, the method also includes: configuring shared memory resources for the docker container; configuring the browser kernel in the docker container to run in a non-graphical interface environment.

[0018] By adopting the above technical solution, the resource utilization of the Docker container is optimized by configuring shared memory resources and non-graphical interface operation mode. The use of shared memory reduces the overhead of data exchange between containers, while the non-graphical interface mode greatly reduces the consumption of system resources. While ensuring the integrity of the collection function, the concurrent processing capability of the system is significantly improved, allowing a single node to run more collection tasks at the same time, thereby improving the overall collection efficiency.

[0019] In combination with some embodiments of the first aspect, in some embodiments, after the step of distributing subtasks and corresponding docker containers to distributed nodes, the method also includes: setting an automatic cleanup mechanism for the browser kernel instance in the docker container, and performing resource recovery after the browser kernel instance ends.

[0020] By adopting the above technical solution, automatic resource management of browser kernel instances is realized. The system will recycle resources immediately after the browser instance completes the task, avoiding problems such as memory leaks and resource occupation. This timely resource release mechanism ensures that the Docker container always maintains an efficient operating state and does not cause performance degradation due to long-term operation.

[0021] In a second aspect, the present application provides a Docker-based distributed browser collection system, the Docker-based distributed browser collection system comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the Docker-based distributed browser collection system to perform the method described in the first aspect and any possible implementation method of the first aspect.

[0022] In a third aspect, the present application provides a computer program product comprising instructions. When the computer program product is run on a Docker-based distributed browser collection system, the Docker-based distributed browser collection system executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a Docker-based distributed browser collection system, enables the Docker-based distributed browser collection system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Split the total demand into multiple subtasks and use docker containers for processing. The use of docker containers ensures that each task has an independent operating environment and network configuration. Combined with the deployment of distributed nodes, the system can access the target website simultaneously through different IP addresses of different nodes. This decentralized access method significantly reduces the access frequency of a single IP and effectively circumvents the access restrictions of the target website. At the same time, the independent browser environment can more realistically simulate user access behavior and improve the stability and reliability of the collection task.

[0025] 2. The demand data is preliminarily classified by web page address features, and further subdivided according to preset rules, so that the system can process collection tasks in a more orderly manner. The design of multi-level task queues allows high-priority tasks to be executed first, and the chronological processing of tasks at the same level ensures that some subtasks will not wait too long. The system can flexibly dispatch resources according to the importance of the business, ensuring that important tasks are completed in a timely manner while preventing low-priority tasks from being shelved for a long time, thereby improving the overall task processing efficiency.

[0026] 3. The system verifies the integrity of the collected data by detecting the web page title and page structure. When abnormal data is found, the task retry mechanism will be triggered. During the retry process, the system will actively switch network proxy resources. This dynamic adjustment strategy not only improves the success rate of the retry task, but also effectively circumvents the access restrictions of the target website. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a distributed browser collection method based on Docker in an embodiment of the present application; Figure 2 It is another flow chart of the distributed browser collection method based on Docker in the embodiment of the present application; Figure 3 It is another flow chart of the distributed browser collection method based on Docker in the embodiment of the present application; Figure 4 It is another flow chart of the distributed browser collection method based on Docker in the embodiment of the present application; Figure 5 It is an exemplary hardware structure diagram of a distributed browser collection system based on Docker in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0030] See also Figure 1 , Figure 1 It is a flow chart of a distributed browser collection method based on Docker in an embodiment of the present application; S101, splitting the total demand data into multiple sub-demand data, creating subtasks according to the sub-demand data, wherein the subtasks include task links and collection parameters of target websites; The total demand data represents the complete data demand set of the website to be collected, which is the task in the related technology and the ultimate goal of this embodiment; In some embodiments, the system first receives the collection requirements including multiple websites and multiple pages as the total demand data. Then, based on the website characteristics and page relevance, the total demand data is split into multiple independent sub-demand data. Then, the system creates a corresponding subtask for each sub-demand data, configures the necessary task links and collection parameters in the subtask, and prepares for the subsequent collection work.

[0031] In some specific embodiments, the splitting of total demand data and the creation of subtasks can be achieved in a variety of ways: optionally, preliminary classification is performed by website domain name, and then secondary classification is performed based on page URL features, and finally the subtask granularity is determined according to the amount of data, and corresponding collection parameters are configured for each subtask; Continuing with the above example, the total demand data is split into multiple sub-demand data according to the website domain name (such as Tencent, Sina, and NetEase).

[0032] Optionally, first analyze the logical relationship between the pages and group the interrelated pages into a group to generate subtasks.

[0033] Continuing with the above example, the logical relationship can be: Sites can turn off js loading to shorten the loading time of a single page.

[0034] The site requires a specific browser size for the page to load.

[0035] Site requires incognito mode for processing etc.

[0036] The site requires a proxy.

[0037] The site needs to extract keywords from the HTML tags of the website.

[0038] The site requires a post form.

[0039] Does the site need to continue turning pages? How to turn pages? Is the page turning action based on sliding the mouse down to turn pages, or clicking the next page to turn pages? Does the site require login? Whether the site can prohibit image loading.

[0040] What is the category of the site? News, social, etc.

[0041] It is understandable that other splitting methods based on business characteristics or technical requirements may also be adopted, which are not limited here.

[0042] S102, building a corresponding Docker container based on the subtask, and integrating a browser operating environment in the Docker container; Among them, the Docker container refers to a lightweight, independent operating environment.

[0043] After completing the creation of the subtasks, an independent execution environment needs to be prepared for each subtask. In some embodiments, the system will create a dedicated Docker container based on the specific needs of each subtask. Inside the container, the system will install and configure a complete browser operating environment, including necessary components such as the browser kernel, JavaScript engine, and network components to ensure that the container can perform web page access and data extraction operations normally.

[0044] It should be noted that after the task is generated, the system distributes it to multiple nodes for parallel downloading. The downloader is developed based on Chromium, and improves the collection efficiency through deep customization and performance optimization. Specific optimizations include: configuring the interface-free operation mode, implementing the intelligent cache cleaning mechanism, establishing precise resource management, optimizing the process management strategy, etc. These optimizations ensure that the resources of the Chromium instance can be released in time and the residual cache can be effectively cleaned. Through these optimization measures, each server can run multiple Chromium instances stably and efficiently, improving the concurrent collection capability.

[0045] S103, distributing the subtasks and the corresponding docker containers to the distributed nodes, so that the distributed nodes run the docker containers to execute the subtasks and collect website data; In some embodiments, the system will select appropriate nodes for task distribution based on the load and performance characteristics of the distributed nodes. The distribution process includes the steps of transmitting the Docker container image, configuring the operating environment, and starting the container. After the node receives the task, it will run the container according to the preset parameters to perform web page access and data collection operations.

[0046] S104: storing the collected website data in corresponding storage areas according to different types.

[0047] Among them, different types refer to the format, structure or purpose classification of data; storage area represents dedicated storage space for different types of data; data classification refers to the process of classifying data according to its characteristics.

[0048] After completing data collection, the data needs to be properly stored and managed. In some embodiments, the system will analyze the collected website data, identify its type and characteristics, and then allocate the data to the corresponding storage area. This process includes steps such as data format conversion, type identification, and storage path selection to ensure that the data can be efficiently stored and subsequently used.

[0049] In an exemplary embodiment, the system distributes tasks to multiple servers for parallel processing, and each server implements local concurrent downloading through a multi-threading mechanism. After the download is completed, the system stores the data in categories according to the data type, for example: file data is stored in the file storage service, and text data is stored in the text storage service, thus implementing classified management of data.

[0050] It can be seen that the total demand is divided into multiple subtasks and processed using docker containers. The use of docker containers ensures that each task has an independent operating environment and network configuration. Combined with the deployment of distributed nodes, the system can access the target website simultaneously through different IP addresses of different nodes. This decentralized access method significantly reduces the access frequency of a single IP and effectively circumvents the access restrictions of the target website. At the same time, the independent browser environment can more realistically simulate user access behavior and improve the stability and reliability of the collection task.

[0051] The above embodiments have improved the stability and reliability of the acquisition task. The subtasks generated by S101 have the characteristics of diverse data types, uneven data quality, and non-uniform formats, which may lead to problems such as low data processing efficiency and unbalanced system load.

[0052] See also Figure 2 , Figure 2 It is another flow chart of the distributed browser collection method based on Docker in the embodiment of the present application; In some embodiments, step S101 specifically includes: S1011, splitting the total demand data into multiple partial demand data according to the web page address characteristics; Among them, the web page address feature represents the information contained in the URL, such as the domain name, path structure, and parameter pattern; Before starting the data collection task, the overall demand needs to be preliminarily classified. In some embodiments, the system first analyzes all URL addresses contained in the total demand data and extracts feature information therein, such as domain name, path level, parameter structure, etc. Then, based on these feature information, URL addresses with similar features are classified together to form multiple partial demand data, preparing for subsequent detailed splitting.

[0053] S1012, splitting part of the demand data into multiple sub-demand data according to preset rules; The preset rules represent the standard conditions for splitting data, including data volume restrictions, correlation requirements, etc. After completing the preliminary classification, the task granularity needs to be further refined. In some embodiments, the system will further split each part of the demand data into multiple independent sub-demand data based on pre-set rules, such as the maximum data volume of a single task. This detailed splitting ensures the execution efficiency and resource utilization of tasks.

[0054] After step S101, the following steps are also included: S201, storing the subtasks in different levels of the multi-level task queue according to the demand data of the subtasks to which they belong, and sorting the subtasks in the same level according to the order of storage time; Among them, the multi-level task queue refers to a task storage system with a hierarchical structure; After the task splitting is completed, the tasks need to be stored in order. In some embodiments, the system will create a multi-level task queue structure and store subtasks from the same part of the demand data in the same queue level. Within each level, the system will sort the tasks according to the creation time sequence to ensure that the tasks can be executed in chronological order and ensure the continuity and integrity of data collection.

[0055] S202, sorting the levels of the multi-level task queue according to the priority of the partial demand data; The sorting rules represent the standards and methods for determining the hierarchical order.

[0056] In some specific embodiments, the priority of some demand data is pre-set and related to the importance of the business.

[0057] In some specific embodiments, the priority of some demand data is related to the timeliness of the data.

[0058] Step S103 specifically includes: S1031. Distribute tasks in order according to the hierarchical order of the multi-level task queues, give priority to distributing subtasks in the high-level queues, and when the subtasks in the high-level queues are distributed, transfer to the subtasks in the low-level queues in order.

[0059] Before starting task execution, tasks need to be distributed reasonably in order of priority. In some embodiments, the system will first distribute tasks from the highest-level task queue to ensure that the highest-priority tasks are processed first. When all tasks at a certain level have been distributed, the system will automatically switch to the next level of task queues and continue to distribute tasks until all tasks at all levels have been processed.

[0060] It can be seen that the demand data is initially classified through the characteristics of the web page address, and further subdivided in combination with the preset rules, so that the system can process the collection tasks more orderly. The design of the multi-level task queue allows high-priority tasks to be executed first, and the processing of tasks of the same level in chronological order ensures that some subtasks will not wait too long. This allows the system to flexibly dispatch resources according to the importance of the business, ensuring that important tasks are completed in a timely manner while preventing low-priority tasks from being shelved for a long time, thereby improving the overall task processing efficiency.

[0061] In some embodiments, step S1031 specifically includes: S10311, selecting multiple subtasks from the same level queue of the multi-level task queue, and assigning the multiple subtasks to the same distributed node; after the distributed node receives the multiple subtasks, it starts multiple execution threads for concurrent processing; When starting task execution, tasks need to be reasonably allocated to improve execution efficiency. In some embodiments, the system will batch select multiple subtasks from the currently processed hierarchical queue and assign these tasks to the same distributed node at one time. When the node receives these tasks, it will create multiple execution threads, each thread is responsible for processing a subtask, and improve task processing efficiency and resource utilization through concurrent execution.

[0062] In some specific embodiments, by real-time monitoring of indicators such as CPU usage, memory usage, and network bandwidth of computing nodes, the actual processing capacity of each node is evaluated, the optimal number of batch tasks is calculated based on historical performance data, a priority-based dynamic task queue is established, and thread pool parameters are intelligently configured according to system load, ultimately achieving concurrent processing of tasks; this is not limited here.

[0063] S10312. Monitor the task execution status of the distributed nodes. When the distributed nodes complete the subtask processing of the current level queue, continue to assign other subtasks in the same level queue to them until the subtasks of the current level queue are assigned, and then assign subtasks of the next level queue.

[0064] In some embodiments, the system monitors the task execution of each distributed node in real time, including indicators such as task progress and resource utilization. When a node completes the current batch of tasks, the system will continue to select new tasks from the same level queue for assignment until all tasks at that level are assigned. After that, the system will automatically switch to the next level queue to continue task assignment and execution.

[0065] It can be seen that a multi-threaded concurrent processing mechanism is used in the task distribution process. Multiple subtasks are assigned to the same node and multiple execution threads are started, making full use of the node's computing resources.

[0066] See also Figure 3 , Figure 3 It is another flow chart of the distributed browser collection method based on Docker in the embodiment of the present application; In some embodiments, after step S104, the method further includes: S301, verifying the integrity of the page content of the website data, including the detection of the web page title and page structure; Among them, the web page title represents the content of the title tag of the HTML document; the page structure represents the hierarchical relationship of the DOM tree and the integrity of key elements;.

[0067] After data collection is completed, quality control of the collection results is required. In some embodiments, the system will perform integrity verification on the collected website data, including checking whether the web page title exists and conforms to the expected format, verifying whether the page DOM structure is complete, and verifying whether key data nodes are fully acquired. This verification mechanism can promptly detect data missing or structural damage problems that occur during the collection process and ensure data quality.

[0068] In some specific embodiments, the integrity of the web page is verified by parsing the returned content. The verification process mainly includes the detection of key elements such as the page title, DOM structure, and necessary content nodes, combined with preset verification rules and page features, to comprehensively determine whether the page is completely and successfully downloaded to ensure the quality of the collected data, which is not limited here.

[0069] S302, when the integrity verification fails, re-write the corresponding subtask into the multi-level task queue; When integrity verification fails, the final integrity of the data needs to be ensured. In some embodiments, the system will re-add the subtasks that failed verification to the multi-level task queue, assign different priorities according to the failure reasons and the number of retries, and ensure that these tasks can be re-executed at the appropriate time to maximize the integrity of data collection.

[0070] S303: Before the corresponding subtask, control the corresponding distributed node to switch the network proxy resource.

[0071] In some embodiments, the system switches the network agents of distributed nodes before executing the task, and selects the most suitable agent resources by evaluating the agent quality, load status and target site characteristics to ensure that the collection task can be executed stably.

[0072] In some specific embodiments, the proxy server status and response time are monitored in real time, the quality score and stability of the proxy IP are evaluated, the available proxy resources that meet the requirements of the target site are screened, and the proxy switching strategy is executed; this is not limited here.

[0073] It can be seen that the system verifies the integrity of the collected data by detecting the web page title and page structure. When the data is found to be abnormal, the task retry mechanism will be triggered. During the retry process, the system will actively switch network proxy resources. This dynamic adjustment strategy not only improves the success rate of the retry task, but also effectively circumvents the access restrictions of the target website.

[0074] See also Figure 4 , Figure 4 It is another flow chart of the distributed browser collection method based on Docker in the embodiment of the present application; In some embodiments, after step S103, the method further includes: S401, creating a network environment, setting the network driver type to bridge mode, and obtaining a custom bridge network; Among them, bridge mode represents a network connection method that allows communication between containers.

[0075] In some embodiments, the system creates an independent network environment and sets its network driver type to bridge mode. In this mode, the system generates a custom bridge network that provides infrastructure for communication between containers, supports network isolation and connection management between containers, and ensures that containers can communicate securely and efficiently.

[0076] In some embodiments, a custom bridge network allows the creation of an independent network environment designed specifically for a distributed browser collection system. The system first creates a default bridge network through the Docker command: dockernetwork create --driver bridge my_custom_bridge. In order to have more precise control over the network configuration, a custom bridge network can be created by specifying a subnet and a gateway: docker network create --driver bridge --subnet 192.168.10.0 / 24 --gateway 192.168.10.1 example_bridge. This configuration specifies a subnet range of 192.168.10.0 / 24 and a gateway address of 192.168.10.1. This means that containers in the network will obtain IP addresses in the range of 192.168.10.0 to 192.168.10.255, with 192.168.10.1 as the gateway address. This network configuration ensures smooth and reliable communication between containers. For example, the system can specify a sufficiently large IP address range for the network to accommodate large-scale container deployment requirements, while ensuring efficient routing of network traffic by properly setting gateway parameters.

[0077] S402, allocating an IP address range to the custom bridge network, and allocating addresses within the IP address range to each Docker container, so that the Docker containers communicate through internal IP addresses; After creating a custom bridge network, address allocation is required. In some embodiments, the system specifies a specific IP address range for the created custom bridge network, and then allocates a unique IP address from this address range to each Docker container. Through these internal IP addresses, containers can directly communicate with each other over the network to achieve data exchange and service calls.

[0078] S403: Configure the network isolation policy of the Docker container.

[0079] The network isolation policy represents a set of rules for controlling network access between containers.

[0080] After completing the network configuration, you need to establish a security mechanism. In some embodiments, the system will configure network isolation policies for Docker containers to define access rights and communication rules between containers. These policies can control network traffic between containers, limit unauthorized access, ensure the security of the container network, and keep necessary communication channels open.

[0081] It can be seen that the refined management of the network environment of the Docker container is achieved. The network configuration of the bridge mode enables efficient internal communication between containers, and the independent IP address allocation ensures the network isolation between containers.

[0082] In some embodiments, after step S103, the method further includes: S404, configuring shared memory resources for the Docker container; The shared memory resource refers to the memory space that can be accessed by containers.

[0083] When setting up the container runtime environment, you need to configure a data exchange mechanism. In some embodiments, the system will configure shared memory resources for the Docker container, and by setting the size, permissions, and access mode of the shared memory segment, different containers can quickly exchange data through the shared memory mechanism. This method avoids the overhead of network transmission and improves the efficiency of communication between containers.

[0084] In some specific embodiments, the configuration of shared memory can be implemented in multiple ways: Optionally, create a system-level memory segment and set read and write permissions, allocate memory space reasonably according to the actual needs of the container, configure sharing parameters and access policies between multiple containers, and verify access functions and performance through memory read and write tests to ensure the reliability of the memory sharing mechanism. Optionally, initialize the shared memory area based on the container operation requirements, set the memory usage limit and dynamic expansion strategy, establish the mutual exclusive access mechanism and priority control between containers, configure data synchronization and consistency maintenance strategy, and perform multi-container concurrent access test to verify the sharing effect and stability.

[0085] It is understandable that other memory sharing solutions based on performance optimization may also be adopted, which are not limited here.

[0086] S405. Configure the browser kernel in the docker container to run in a non-graphical interface environment.

[0087] The non-graphical interface environment refers to an operating mode that does not require a display device.

[0088] In some embodiments, the system configures the browser kernel in the Docker container so that it can run normally in an environment without a graphical interface. This configuration includes setting the browser's headless mode parameters, configuring necessary system dependencies, adjusting memory usage policies, etc., to ensure that the browser can perform tasks efficiently and stably in the container environment.

[0089] It can be seen that by configuring shared memory resources and non-graphical interface operation mode, the resource utilization of Docker containers is optimized. The use of shared memory reduces the overhead of data exchange between containers, while the non-graphical interface mode greatly reduces the consumption of system resources. While ensuring the integrity of the collection function, the concurrent processing capability of the system is significantly improved, allowing a single node to run more collection tasks at the same time, thereby improving the overall collection efficiency.

[0090] In some embodiments, after step S103, the method further includes: S406: Set an automatic cleanup mechanism for the browser kernel instance in the docker container to recycle resources after the browser kernel instance ends.

[0091] Among them, the automatic cleanup mechanism refers to the program that the system automatically recycles resources; resource recycling means releasing used system resources, including memory, CPU, etc.

[0092] During the execution of tasks by the browser kernel, system resources need to be managed in a timely manner. In some embodiments, the system configures a resource cleanup mechanism for each Docker container and monitors the running status of the browser kernel instance. When it is detected that the instance has ended, the system automatically executes a cleanup program to release the memory, file handles and other resources occupied by the instance, prevent resource leakage, and ensure that the system can continue to run stably.

[0093] It can be seen that the automatic resource management of the browser kernel instance is realized. The system will recycle resources immediately after the browser instance completes the task, avoiding problems such as memory leaks and resource occupation. This timely resource release mechanism ensures that the Docker container always maintains an efficient running state and does not cause performance degradation due to long-term operation.

[0094] The following introduces a Docker-based distributed browser collection system 500 provided in an embodiment of the present application. Figure 5 It is a schematic diagram of an exemplary hardware structure of a Docker-based distributed browser collection system 500 provided in an embodiment of the present application.

[0095] In some embodiments, the docker-based distributed browser collection system 500 is a computer device or the docker-based distributed browser collection system 500 includes a computer device. The computer device includes a processor, a memory and a network interface connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, the method in the embodiment of the present application is implemented.

[0096] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0097] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0098] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0099] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0100] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A distributed browser collection method based on docker, characterized in that: include: Splitting the total demand data into multiple sub-demand data, and creating sub-tasks according to the sub-demand data, wherein the sub-tasks include task links and collection parameters of target websites; Building a corresponding Docker container based on the subtask, wherein the browser operating environment is integrated in the Docker container; Distribute the subtask and the corresponding docker container to distributed nodes, so that the distributed nodes run the docker container to execute the subtask and collect website data; The collected website data is stored in corresponding storage areas according to different types.

2. The method according to claim 1, characterized in that The step of splitting the total demand data into a plurality of sub-demand data specifically includes: Splitting the total demand data into multiple partial demand data according to the web page address characteristics; Splitting the part of demand data into a plurality of sub-demand data according to a preset rule; After the step of creating a subtask according to the sub-demand data, the method further includes: The subtasks are stored in different levels of the multi-level task queue according to the part of the demand data to which they belong, and the subtasks in the same level are sorted according to the order of storage time; Sorting the levels of the multi-level task queue according to the priority of the partial demand data; The step of distributing the subtask and the corresponding Docker container to the distributed nodes specifically includes: The tasks are distributed in sequence according to the hierarchical order of the multi-level task queues, with the subtasks in the high-level queues being distributed first. When the subtasks in the high-level queues are distributed, they are transferred to the subtasks in the low-level queues in sequence.

3. The method according to claim 2, characterized in that The step of distributing tasks in order of the levels of the multi-level task queues specifically includes: Select multiple subtasks from the same level queue of the multi-level task queue, and assign the multiple subtasks to the same distributed node; after the distributed node receives the multiple subtasks, start multiple execution threads for concurrent processing Monitor the task execution status of the distributed node. When the distributed node completes the subtask processing of the current level queue, continue to assign other subtasks in the same level queue to it until the subtasks of the current level queue are assigned, and then assign subtasks of the next level queue.

4. The method according to claim 3, characterized in that After the step of storing the collected website data in corresponding storage areas according to different types, the method further includes: Performing integrity verification on the page content of the website data, wherein the integrity verification includes detecting the page title and page structure; When the integrity verification fails, rewriting the corresponding subtask into the multi-level task queue; Before the corresponding subtask, the corresponding distributed node is controlled to switch the network proxy resource.

5. The method according to claim 1, characterized in that After the step of distributing the subtask and the corresponding Docker container to the distributed nodes, the method further includes: Create a network environment, set the network driver type to bridge mode, and get a custom bridge network; Allocate an IP address range for the custom bridge network, and allocate addresses within the IP address range to each of the Docker containers, so that the Docker containers communicate through internal IP addresses; Configure the network isolation policy of the Docker container.

6. The method according to claim 5, characterized in that After the step of distributing the subtask and the corresponding Docker container to the distributed nodes, the method further includes: Configuring shared memory resources for the Docker container; The browser kernel in the docker container is configured to run in a non-graphical interface environment.

7. The method according to claim 5, characterized in that After the step of distributing the subtask and the corresponding Docker container to the distributed nodes, the method further includes: An automatic cleanup mechanism for the browser kernel instance in the docker container is set to recycle resources after the browser kernel instance ends.

8. A distributed browser collection system based on Docker, characterized in that: The Docker-based distributed browser collection system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the Docker-based distributed browser collection system to execute the method described in any one of claims 1-7.

9. A computer program product comprising instructions, characterized in that When the computer program product runs on a docker-based distributed browser collection system, the docker-based distributed browser collection system executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on a distributed browser collection system based on Docker, the distributed browser collection system based on Docker executes the method as described in any one of claims 1 to 7.

Images