Docker-based distributed browser collection method and related device

Abstract

The application discloses a docker-based distributed browser collection method and related equipment, and relates to the field of data collection frameworks. The total demand is split into multiple subtasks and processed using docker containers. The use of docker containers ensures that each task has an independent running environment and network configuration. In combination with the deployment of distributed nodes, the system can simultaneously access target websites through different IP addresses of different nodes. This decentralized access method significantly reduces the access frequency of a single IP and effectively avoids access restrictions of the target websites. Meanwhile, the independent browser environment can more realistically simulate user access behaviors, improving the stability and reliability of the collection task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data collection framework, and particularly relates to a docker-based distributed browser collection method and related equipment. BACKGROUND

[0002] In the tide of modern social informatization, artificial intelligence generated content (AIGC) is playing an increasingly critical role in the production activities of many industries. Data collection, as a cornerstone in the field of AIGC, cannot be underestimated. It is responsible for efficiently collecting massive amounts of data from the Internet to provide necessary training materials and information resources for AIGC systems. After careful processing and analysis, these data can be used to cultivate more advanced and accurate artificial intelligence models. These models realize automated operation and innovative applications in content creation, language translation, data analysis and other fields.

[0003] With the continuous progress of data collection technology, the capabilities of AIGC have been significantly enhanced, enabling it to exhibit superior performance in personalized and high-quality content production. This not only improves production efficiency, but also provides users with a more rich and accurate personalized experience. Therefore, the innovation and development of data collection technology are of great significance to the growth and progress of the AIGC field.

[0004] In traditional data collection systems, the collection program directly interacts with the target website server through the browser, using a single request-response mode to obtain data. This method configures access parameters in the collection program, directly sends data requests to the target website, obtains the required web page content, and stores the collected data in the local database.

[0005] However, this direct access method faces many limitations in practical applications. Since the target website usually sets access frequency limits and IP blocking mechanisms, when the collection request is too frequent, the IP address of the collection program is easily identified and blocked by the target website, resulting in the interruption of the collection task. At the same time, the single access mode also makes it difficult to simulate real user browsing behavior, which is easily identified by the website's anti-crawler system, so the data collection capability in related technologies is poor. SUMMARY

[0006] The present application provides a docker-based distributed browser collection method and related equipment to improve data collection capabilities.

[0007] In one aspect, the application provides a docker-based distributed browser collection method, comprising: splitting total demand data into multiple sub-demand data, creating sub-tasks according to the sub-demand data, the sub-tasks including task links and collection parameters of target websites; constructing corresponding docker containers based on the sub-tasks, integrating a browser running environment in the docker containers; distributing the sub-tasks and corresponding docker containers to distributed nodes, enabling the distributed nodes to run the docker containers to execute the sub-tasks and collect website data; and storing the collected website data according to different types into corresponding storage areas.

[0008] By adopting the above technical solution, the total demand is split into multiple sub-tasks and processed using docker containers, the use of docker containers ensures that each task has an independent running environment and network configuration, and in combination with the deployment of distributed nodes, the system can simultaneously access target websites through different IP addresses of different nodes. This decentralized access method significantly reduces the access frequency of a single IP and effectively avoids access restrictions of target websites. At the same time, the independent browser environment can more realistically simulate user access behavior, improving the stability and reliability of collection tasks.

[0009] In combination with some embodiments of the first aspect, in some embodiments, the step of splitting the total demand data into multiple sub-demand data specifically comprises: splitting the total demand data into multiple partial demand data according to webpage address features; splitting the partial demand data into multiple sub-demand data according to a preset rule; after the step of creating sub-tasks according to the sub-demand data, the method further comprises: storing the sub-tasks in different levels of a multi-level task queue according to their belonging partial demand data, and sorting the sub-tasks in the same level according to the storage time sequence; sorting the levels of the multi-level task queue according to the priority of the partial demand data; and the step of distributing the sub-tasks and corresponding docker containers to distributed nodes specifically comprises: sequentially distributing tasks according to the level order of the multi-level task queue, preferentially distributing sub-tasks in high-level queues, and transferring sub-tasks in low-level queues when the sub-tasks in high-level queues are distributed.

[0010] By adopting the above technical solution, the demand data is preliminarily classified according to webpage address features, and further subdivided in combination with a preset rule, so that the system can more orderly process collection tasks. The design of the multi-level task queue enables high-priority tasks to be executed preferentially, and the processing of tasks in the same level according to time sequence ensures that some sub-tasks will not wait for too much time. This enables the system to flexibly schedule resources according to business importance, ensuring that important tasks are completed in time, while also preventing low-priority tasks from being long-term shelved, thereby improving the overall task processing efficiency.

[0011] In combination with some embodiments of the first aspect, in some embodiments, the steps of distributing tasks in sequence according to the hierarchical order of the multi-level task queue specifically include: selecting multiple subtasks from the same-level queue of the multi-level task queue, and assigning the multiple subtasks to the same distributed node; after the distributed node receives the multiple subtasks, starting multiple execution threads to perform concurrent processing to monitor the task execution status of the distributed node; when the distributed node completes the subtask processing of the current-level queue, continue to assign other subtasks in the same-level queue to it, until the subtasks of the current-level queue are assigned, and then assign the subtasks of the next-level queue.

[0012] By adopting the above technical solution, a multi-threaded concurrent processing mechanism is adopted in the task distribution process. Multiple subtasks are assigned to the same node and multiple execution threads are started, making full use of the node's computing resources.

[0013] In combination with some embodiments of the first aspect, in some embodiments, after the step of storing the collected website data in corresponding storage areas according to different types, the method also includes: performing integrity verification on the page content of the website data, the integrity verification including detection of the web page title and page structure; when the integrity verification fails, rewriting the corresponding subtask into the multi-level task queue; before the corresponding subtask, controlling the corresponding distributed node to switch the network proxy resources.

[0014] By employing this technical solution, the system verifies the integrity of collected data by checking web page titles and structure. If any data anomalies are detected, a task retry mechanism is triggered. During the retry process, the system proactively switches network proxy resources. This dynamic adjustment strategy not only improves the success rate of retry tasks but also effectively circumvents access restrictions on target websites.

[0015] In combination with some embodiments of the first aspect, in some embodiments, after the step of distributing subtasks and corresponding docker containers to distributed nodes, the method also includes: creating a network environment, setting the network driver type to bridge mode, and obtaining a custom bridge network; allocating an IP address range to the custom bridge network, and allocating addresses within the IP address range to each docker container, so that the docker containers communicate through internal IP addresses; and configuring the network isolation policy of the docker container.

[0016] By adopting the above technical solutions, we can achieve refined management of the network environment of Docker containers. The bridge mode network configuration enables efficient internal communication between containers, while independent IP address allocation ensures network isolation between containers.

[0017] In some embodiments of the first aspect, after the step of distributing the sub-tasks and corresponding docker containers to the distributed nodes, the method further comprises: configuring shared memory resources for the docker containers; and configuring the browser kernel within the docker containers to run in a non-graphical interface environment.

[0018] By adopting the above technical solution, the resource utilization of the docker containers is optimized by configuring shared memory resources and a non-graphical interface running mode. The use of shared memory reduces the overhead of data exchange between containers, and the non-graphical interface mode significantly reduces the consumption of system resources. While ensuring the integrity of the acquisition function, the concurrent processing capability of the system is significantly improved, enabling a single node to run more acquisition tasks simultaneously, thereby improving the overall acquisition efficiency.

[0019] In some embodiments of the first aspect, after the step of distributing the sub-tasks and corresponding docker containers to the distributed nodes, the method further comprises: setting an automatic cleaning mechanism for the browser kernel instances within the docker containers, and performing resource recycling after the browser kernel instances complete running.

[0020] By adopting the above technical solution, the automatic resource management of the browser kernel instances is realized. The system performs resource recycling immediately after the browser instances complete tasks, avoiding problems such as memory leakage and resource occupation. This timely resource release mechanism ensures that the docker containers always remain in an efficient running state and do not cause performance degradation due to long-term running.

[0021] In a second aspect, the present application provides a docker-based distributed browser acquisition system. The docker-based distributed browser acquisition system comprises one or more processors and a memory. The memory is coupled to the one or more processors and is configured to store computer program code comprising computer instructions. The one or more processors invoke the computer instructions to cause the docker-based distributed browser acquisition system to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a third aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on a docker-based distributed browser acquisition system, cause the docker-based distributed browser acquisition system to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions that, when executed on a docker-based distributed browser collection system, cause the docker-based distributed browser collection system to perform the method as described in the first aspect and any possible implementation of the first aspect.

[0024] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] 1. The total demand is split into multiple sub-tasks and processed using docker containers. The use of docker containers ensures that each task has an independent running environment and network configuration. In combination with the deployment of distributed nodes, the system can access the target website through different IP addresses of different nodes at the same time. This decentralized access method significantly reduces the access frequency of a single IP and effectively avoids access restrictions of the target website. At the same time, the independent browser environment can more realistically simulate user access behavior, improving the stability and reliability of the collection task.

[0026] 2. The demand data is preliminarily classified by web address features and further subdivided in combination with preset rules, so that the system can more orderly process collection tasks. The design of multi-level task queues allows high-priority tasks to be executed first, while same-level tasks are processed in time sequence to ensure that some sub-tasks do not wait for too much time. This allows the system to flexibly schedule resources according to business importance, ensuring that important tasks are completed in a timely manner while not causing low-priority tasks to be long-term shelved, thereby improving overall task processing efficiency.

[0027] 3. The system verifies the integrity of the collected data by detecting the web page title and page structure. When data anomalies are found, a task retry mechanism is triggered. During the retry process, the system actively switches network proxy resources. This dynamic adjustment strategy not only improves the success rate of retry tasks, but also effectively avoids access restrictions of the target website. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of the docker-based distributed browser collection method in the embodiments of the present application;

[0029] Figure 2 is another flowchart of the docker-based distributed browser collection method in the embodiments of the present application;

[0030] Figure 3 is another flowchart of the docker-based distributed browser collection method in the embodiments of the present application;

[0031] Figure 4is another flow diagram of the docker-based distributed browser collection method in the embodiments of the present application;

[0032] Figure 5 is an exemplary hardware structure diagram of the docker-based distributed browser collection system in the embodiments of the present application. DETAILED DESCRIPTION

[0033] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the present application, the singular forms "a," "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used in the present application, signify and include any and all possible combinations of one or more of the associated listed items.

[0034] Hereinafter, the terms "first" and "second" are only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0035] Please refer to Figure 1 , Figure 1 is a flow diagram of the docker-based distributed browser collection method in the embodiments of the present application;

[0036] S101, split the total demand data into a plurality of sub-demand data, create a sub-task according to the sub-demand data, and the sub-task includes the task link and the collection parameter of the target website;

[0037] Among them, the total demand data represents the complete data demand set of the website to be collected, that is, the task in the related art, which is also the final goal of the embodiment;

[0038] In some embodiments, the system first receives the collection demand containing a plurality of websites and a plurality of pages as the total demand data. Then, based on the website features and the page relevance, the total demand data is split into a plurality of independent sub-demand data. Then, the system will create a corresponding sub-task for each sub-demand data, and configure the necessary task link and collection parameter in the sub-task, to prepare for the subsequent collection work.

[0039] In some specific embodiments, the splitting of total demand data and the creation of sub-tasks can be achieved in various ways: optionally, preliminary classification is performed by website domain name, then secondary classification is performed based on page URL features, and finally the granularity of sub-tasks is determined according to the size of data quantity, and the corresponding collection parameters are configured for each sub-task.

[0040] In the above example, the total demand data is split into multiple sub-demand data according to the website domain name (such as Tencent, Sina, and Netease).

[0041] Optionally, the logical relationship between pages is first analyzed, and pages that are associated with each other are grouped into a group to generate a sub-task.

[0042] In the above example, the logical relationship can be:

[0043] The site can close js loading to shorten the loading time of a single page.

[0044] The site needs a specific browser size to load the page.

[0045] The site needs to be processed in incognito mode.

[0046] The site needs a proxy.

[0047] The site needs keyword extraction within the website html tag.

[0048] The site needs post form.

[0049] Does the site need to continue to page, how to page? The action of paging is based on mouse sliding down to page, or clicking next page to page?

[0050] Does the site need to log in?

[0051] Can the site prohibit image loading?

[0052] What is the category of the site? It belongs to news, social media, etc.

[0053] It can be understood that other splitting methods based on business characteristics or technical requirements can also be used, which are not limited here.

[0054] S102, based on the sub-task, a corresponding docker container is constructed, and a browser running environment is integrated in the docker container;

[0055] Among them, the docker container refers to a lightweight, independent running environment.

[0056] After the subtask creation is completed, an independent execution environment needs to be prepared for each subtask. In some embodiments, the system will create a dedicated docker container based on the specific needs of each subtask. Inside the container, the system will install and configure a complete browser runtime environment, including browser kernel, JavaScript engine, network components, and other necessary components, to ensure that the container can normally perform web access and data extraction operations.

[0057] It should be noted that the system distributes the tasks to multiple nodes for parallel download after the tasks are generated. The downloader is developed based on Chromium and optimized for performance to improve collection efficiency. Specific optimizations include: configuring a no-interface running mode, implementing an intelligent cache cleaning mechanism, establishing accurate resource control, and optimizing process management strategies. These optimizations ensure that Chromium instances can release resources in a timely manner and residual caches are effectively cleaned. Through these optimization measures, each server can stably and efficiently run multiple Chromium instances, improving concurrent collection capabilities.

[0058] S103, distribute the subtasks and corresponding docker containers to the distributed nodes, so that the distributed nodes run the docker containers to execute the subtasks and collect website data;

[0059] In some embodiments, the system will select appropriate nodes for task distribution based on the load and performance characteristics of the distributed nodes. The distribution process includes steps such as transferring docker container images, configuring runtime environments, and starting containers. After receiving the task, the node will run the container according to the preset parameters to perform web access and data collection operations.

[0060] S104, store the collected website data according to different types in corresponding storage areas.

[0061] Different types refer to the format, structure, or purpose classification of data; storage areas represent dedicated storage spaces for different types of data; and data classification refers to the process of categorizing data based on characteristics.

[0062] After data collection is completed, the data needs to be reasonably stored and managed. In some embodiments, the system will analyze the collected website data, identify its type and characteristics, and then distribute the data to the corresponding storage area. This process includes data format conversion, type identification, storage path selection, and other steps to ensure that data can be efficiently stored and used later.

[0063] In an exemplary embodiment, the system distributes tasks to multiple servers for parallel processing, and each server implements local concurrent downloading through a multi-thread mechanism. After downloading is completed, the system stores the data according to data types, for example, file type data is stored in a file storage service, and text type data is stored in a text storage service, to achieve classified management of data.

[0064] It can be seen that the total demand is split into multiple sub-tasks and processed using docker containers. The use of docker containers ensures that each task has an independent running environment and network configuration. In combination with the deployment of distributed nodes, the system can access the target website through different IP addresses of different nodes at the same time. This decentralized access method significantly reduces the access frequency of a single IP and effectively avoids access restrictions of the target website. At the same time, the independent browser environment can more realistically simulate user access behavior, improving the stability and reliability of the collection task.

[0065] The above embodiments have improved the stability and reliability of the collection task. The sub-tasks generated in S101 have characteristics such as diverse data types, uneven data quality, and non-uniform formats, which can lead to low data processing efficiency and uneven system load.

[0066] Please refer to Figure 2 , Figure 2 is another flowchart of the docker-based distributed browser collection method in the embodiments of the present application;

[0067] In some embodiments, step S101 specifically includes:

[0068] S1011, splitting the total demand data into multiple partial demand data according to web address features;

[0069] The web address features represent information such as domain names, path structures, and parameter patterns contained in the URL.

[0070] Before starting the data collection task, the total demand needs to be preliminarily classified. In some embodiments, the system first analyzes all URL addresses contained in the total demand data and extracts feature information such as domain names, path levels, and parameter structures. Then, based on these feature information, URL addresses with similar features are classified together to form multiple partial demand data, preparing for subsequent refinement and splitting.

[0071] S1012, splitting the partial demand data into multiple sub-demand data according to a preset rule;

[0072] The preset rule represents a standard condition for splitting data, including data volume limit, correlation requirement, etc. After the initial classification, the task granularity needs to be further refined. In some embodiments, the system will further split each part of the demand data into multiple independent sub-demand data according to the pre-set rules, such as the maximum data volume of a single task, and other factors. This refined splitting ensures the execution efficiency of the task and the utilization rate of resources.

[0073] Step S101 further includes:

[0074] S201, respectively storing the sub-tasks in different levels of the multi-level task queue according to the part of the demand data to which the sub-tasks belong, and sorting the sub-tasks in the same level according to the storage time sequence;

[0075] The multi-level task queue refers to a task storage system with a hierarchical structure.

[0076] After the task splitting is completed, the tasks need to be stored in order. In some embodiments, the system will create a multi-level task queue structure, and store the sub-tasks from the same part of the demand data in the same queue level. Within each level, the system will sort the tasks according to the creation time sequence to ensure that the tasks are executed in time sequence, and to ensure the continuity and integrity of data collection.

[0077] S202, sorting the levels of the multi-level task queue according to the priority of the part of the demand data;

[0078] The sorting rule represents the standard and method for determining the level order.

[0079] In some specific embodiments, the priority of the part of the demand data is related to the pre-set business importance.

[0080] In some specific embodiments, the priority of the part of the demand data is related to the data timeliness.

[0081] Step S103 specifically includes:

[0082] S1031, distributing the tasks in sequence according to the level order of the multi-level task queue, preferentially distributing the sub-tasks in the high-level queue, and then transferring the sub-tasks in the low-level queue when the sub-tasks in the high-level queue have been distributed.

[0083] Before starting the task execution, the tasks need to be distributed reasonably according to the priority order. In some embodiments, the system will first distribute the tasks from the highest level of the task queue to ensure that the tasks with the highest priority are processed first. When all the tasks in a level have been distributed, the system will automatically switch to the task queue of the next level to continue distributing the tasks until all the tasks in all levels are processed.

[0084] It can be seen that the demand data is preliminarily classified by the webpage address features, and further subdivided in combination with preset rules, so that the system can more orderly process the collection tasks. The design of the multi-level task queue enables high-priority tasks to be executed preferentially, and the same-level tasks are processed in time sequence, which ensures that part of the sub-tasks will not wait for too much time. The system can flexibly schedule resources according to the importance of the business, ensuring that important tasks are completed in time, and also preventing low-priority tasks from being shelved for a long time, thereby improving the overall task processing efficiency.

[0085] In some embodiments, step S1031 specifically comprises:

[0086] S10311, selecting multiple sub-tasks from the same level queue of the multi-level task queue, and distributing the multiple sub-tasks to the same distributed node; after the distributed node receives the multiple sub-tasks, starting multiple execution threads for concurrent processing;

[0087] When starting task execution, it is necessary to reasonably allocate tasks to improve execution efficiency. In some embodiments, the system will select multiple sub-tasks in batches from the current processing level queue, and allocate these tasks to the same distributed node at one time. When the node receives these tasks, it will create multiple execution threads, each thread responsible for processing a sub-task, and improve task processing efficiency and resource utilization through concurrent execution.

[0088] In some specific embodiments, by monitoring the CPU usage, memory occupation, network bandwidth and other indicators of the computing node in real time, the actual processing capacity of each node is evaluated, the optimal batch task quantity is calculated in combination with historical performance data, a dynamic task queue based on priority is established, thread pool parameters are intelligently configured according to system load, and finally concurrent processing of tasks is realized; here is not limited.

[0089] S10312, monitoring the task execution of the distributed node, when the distributed node completes the processing of the sub-tasks of the current level queue, continuing to distribute other sub-tasks in the same level queue to it, until the sub-tasks of the current level queue are allocated, and then allocating the sub-tasks of the next level queue.

[0090] In some embodiments, the system will monitor the task execution of each distributed node in real time, including task progress, resource utilization and other indicators. When the node completes the current batch of tasks, the system will continue to select new tasks from the same level queue for distribution until all tasks in the level are allocated. Then, the system will automatically switch to the next level queue and continue to distribute and execute tasks.

[0091] It can be seen that a multi-thread concurrent processing mechanism is adopted in the task distribution process. Multiple sub-tasks are allocated to the same node and multiple execution threads are started, so that the computing resources of the node are fully utilized.

[0092] Referring to Figure 3 , Figure 3 is another flowchart of the docker-based distributed browser collection method in the embodiments of the present application.

[0093] In some embodiments after step S104, the method further comprises:

[0094] S301, integrity verification is performed on the page content of the website data, and the integrity verification includes detection of a web page title and a page structure.

[0095] The web page title represents the content of the title tag of the HTML document, and the page structure represents the hierarchical relationship of the DOM tree and the integrity of the key elements.

[0096] After data collection is completed, quality control needs to be performed on the collection results. In some embodiments, the system performs integrity verification on the collected website data, including checking whether the web page title exists and conforms to the expected format, verifying whether the page DOM structure is complete, and verifying whether the key data nodes are complete. This verification mechanism can timely discover data missing or structure damage problems occurring in the collection process, and ensure data quality.

[0097] In some specific embodiments, web page integrity verification is performed by parsing the returned content. The verification process mainly includes detection of the page title, the DOM structure, and necessary content nodes, and in combination with preset verification rules and page characteristics, it is comprehensively judged whether the page is complete and successfully downloaded, to ensure the quality of the collected data, which is not limited here.

[0098] S302, when the integrity verification fails, the corresponding sub-task is re-written into the multi-level task queue.

[0099] When the integrity verification fails, the final integrity of the data needs to be ensured. In some embodiments, the system re-adds the sub-task that fails in verification to the multi-level task queue, and allocates different priorities according to the failure reason and the number of retries, to ensure that these tasks can be re-executed at the appropriate time, and to maximize the integrity of data collection.

[0100] S303, before the corresponding sub-task, the corresponding distributed node is controlled to switch network proxy resources.

[0101] In some embodiments, the system will switch the network agent of the distributed node before task execution, select the most suitable agent resource by evaluating the agent quality, load condition and target site characteristics, and ensure stable execution of the collection task.

[0102] In some specific embodiments, by monitoring the agent server state and response time in real time, the quality score and stability of the agent IP are evaluated, the available agent resources that meet the requirements of the target site are screened, and the agent switching strategy is executed; here is not limited.

[0103] As can be seen, the system verifies the integrity of the collected data by detecting the page title and page structure, and when data anomalies are found, the task retry mechanism is triggered. In the retry process, the system will actively switch the network agent resource. This dynamic adjustment strategy not only improves the success rate of the retry task, but also effectively avoids access restrictions of the target website.

[0104] Please refer to Figure 4 , Figure 4 is another flowchart of the docker-based distributed browser collection method in the embodiments of the present application;

[0105] In some embodiments, after step S103, further comprising:

[0106] S401, create a network environment, set the network driver type to bridge mode, and get a custom bridge network;

[0107] Among them, the bridge mode represents a network connection method that allows communication between containers.

[0108] In some embodiments, the system will create a separate network environment and set its network driver type to bridge mode. In this mode, the system will generate a custom bridge network that provides the infrastructure for inter-container communication, supports network isolation and connection management between containers, and ensures that containers can communicate safely and efficiently.

[0109] In some embodiments, the custom bridge network allows the creation of a separate network environment specifically designed for the distributed browser scraping system. The system first creates a default bridge network through the Docker command: docker network create --driver bridge my_custom_bridge. To have more fine-grained control over the network configuration, a custom bridge network can be created by specifying the subnet and gateway: docker network create --driver bridge --subnet 192.168.10.0 / 24 --gateway 192.168.10.1 example_bridge. This configuration specifies a subnet range of 192.168.10.0 / 24 and a gateway address of 192.168.10.1. This means that containers in the network will obtain IP addresses in the range of 192.168.10.0 to 192.168.10.255, with 192.168.10.1 as the gateway address. This way of configuring the network ensures smooth and reliable communication between containers. For example, the system can specify a large enough IP address range for the network to accommodate large-scale container deployment needs, while setting reasonable gateway parameters to ensure efficient routing of network traffic.

[0110] S402, allocate an IP address range for the custom bridge network and assign addresses within the IP address range to each docker container, so that the docker containers communicate through internal IP addresses;

[0111] After creating the custom bridge network, address allocation is needed. In some embodiments, the system specifies a specific IP address range for the created custom bridge network, and then assigns a unique IP address to each Docker container from this address range. Through these internal IP addresses, containers can directly communicate with each other, enabling data exchange and service invocation.

[0112] S403, configure the network isolation policy of the docker container.

[0113] Among them, the network isolation policy represents a set of rules that control network access between containers.

[0114] After completing the network configuration, a security mechanism needs to be established. In some embodiments, the system configures a network isolation policy for Docker containers, defining access permissions and communication rules between containers. These policies can control network traffic between containers, limit unauthorized access, and ensure the security of the container network while keeping necessary communication channels open.

[0115] It can be seen that the network environment of the docker container is finely managed. The network configuration in bridge mode enables efficient internal communication between containers, and the independent IP address allocation ensures network isolation between containers.

[0116] In some embodiments, after step S103, the method further comprises:

[0117] S404, configuring a shared memory resource for the docker container;

[0118] The shared memory resource represents a memory space that can be accessed by containers.

[0119] When building a container running environment, a data exchange mechanism needs to be configured. In some embodiments, the system configures a shared memory resource for the Docker container, sets the size, permissions, and access mode of the shared memory segment, so that different containers can quickly exchange data through the shared memory mechanism. This approach avoids the overhead of network transmission and improves the efficiency of container communication.

[0120] In some specific embodiments, the configuration of shared memory can be achieved in various ways:

[0121] Optionally, a system-level memory segment is created and read / write permissions are set, the memory space size is reasonably allocated according to the actual needs of the container, the shared parameters and access strategies between multiple containers are configured, the access function and performance are verified through memory read / write testing, and the reliability of the memory sharing mechanism is ensured;

[0122] Optionally, the shared memory area is initialized based on the running requirements of the container, the upper limit of memory usage and dynamic expansion strategy are set, mutual access mechanism and priority control between containers are established, data synchronization and consistency maintenance strategy are configured, and multi-container concurrent access testing is performed to verify the sharing effect and stability.

[0123] It can be understood that other memory sharing schemes based on performance optimization can also be used, which are not limited here.

[0124] S405, configuring the browser kernel in the docker container to run in a non-graphical interface environment.

[0125] The non-graphical interface environment refers to a running mode that does not require a display device.

[0126] In some embodiments, the system configures the browser kernel in the Docker container to enable it to run normally in a non-graphical interface environment. This configuration includes setting the headless mode parameters of the browser, configuring necessary system dependencies, adjusting memory usage strategies, etc., to ensure that the browser can efficiently and stably perform tasks in the container environment.

[0127] It can be seen that by configuring the shared memory resource and the non-graphical interface running mode, the resource utilization of the docker container is optimized. The use of shared memory reduces the overhead of data exchange between containers, and the non-graphical interface mode greatly reduces the consumption of system resources. While ensuring the integrity of the acquisition function, the concurrent processing capability of the system is significantly improved, so that a single node can run more acquisition tasks at the same time, thereby improving the overall acquisition efficiency.

[0128] In some embodiments, after step S103, further comprising:

[0129] S406, setting an automatic cleaning mechanism of the browser kernel instance in the docker container, and recycling resources after the browser kernel instance is ended.

[0130] The automatic cleaning mechanism refers to a program for automatically recycling resources by the system; resource recycling refers to releasing used system resources, including memory, CPU, etc.

[0131] During the execution of the browser kernel, system resources need to be managed in a timely manner. In some embodiments, the system configures a resource cleaning mechanism for each Docker container to monitor the running state of the browser kernel instance. When it is detected that the instance is ended, the system automatically executes the cleaning program to release the memory, file handle and other resources occupied by the instance, preventing resource leakage and ensuring the continuous and stable operation of the system.

[0132] It can be seen that the automatic resource management of the browser kernel instance is realized. The system immediately recycles resources after the browser instance completes the task, avoiding memory leakage and resource occupation and other problems. This timely resource release mechanism ensures that the docker container always remains in an efficient running state and does not cause performance degradation due to long-time running.

[0133] The following describes a docker-based distributed browser acquisition system 500 provided by an embodiment of the present application. Figure 5 is an exemplary hardware structure schematic diagram of the docker-based distributed browser acquisition system 500 provided by an embodiment of the present application.

[0134] In some embodiments, the docker-based distributed browser collection system 500 is a computer device or includes a computer device therein. The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data. The network interface of the computer device is configured to communicate with other terminals or servers outside through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. The computer program is executed by the processor to implement the method in the embodiments of the present application.

[0135] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the foregoing embodiments of the present application have been described in detail, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0137] In the above embodiments, according to the context, the term "when" can be interpreted as meaning "if" or "after" or "in response to determining" or "in response to detecting". Similarly, according to the context, the phrase "upon determining" or "if detecting (the stated condition or event)" can be interpreted as meaning "if determining" or "in response to determining" or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)".

[0138] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like.

[0139] Those of ordinary skill in the art understand that all or part of the processes in the above embodiments can be implemented by a computer program to instruct the relevant hardware, which can be stored in a computer readable storage medium. The program can include the processes of the above method embodiments when executed. The aforementioned storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. A distributed browser collection method based on Docker, characterized in that: include: Split the total demand data into multiple parts of demand data according to the web page address characteristics; Splitting the partial demand data into multiple sub-demand data according to preset rules; Creating a subtask based on the sub-demand data, wherein the subtask includes a task link and collection parameters of a target website; Building a corresponding Docker container based on the subtask, wherein the browser operating environment is integrated into the Docker container; Select multiple subtasks from the same level queue of the multi-level task queue, and assign the multiple subtasks to the same distributed node; after the distributed node receives the multiple subtasks, start multiple execution threads to perform concurrent processing and monitor the task execution status of the distributed node; when the distributed node completes the processing of the subtasks of the current level queue, continue to assign other subtasks in the same level queue to it, until the subtasks of the current level queue are assigned, and then assign the subtasks of the next level queue, giving priority to the subtasks in the high-level queue, and when the subtasks in the high-level queue are assigned, transfer them to the subtasks in the low-level queue in turn, so that the distributed node runs the docker container to execute the subtasks and collect website data; The collected website data is stored in corresponding storage areas according to different types.

2. The method according to claim 1, characterized in that After the step of storing the collected website data in corresponding storage areas according to different types, the method further includes: Performing integrity verification on the page content of the website data, the integrity verification including detection of the web page title and page structure; When the integrity verification fails, rewriting the corresponding subtask into the multi-level task queue; Before the corresponding subtask, the corresponding distributed node is controlled to switch the network proxy resource.

3. The method according to claim 1, characterized in that After the step of distributing the subtask and the corresponding Docker container to the distributed nodes, the method further includes: Create a network environment, set the network driver type to bridge mode, and get a custom bridge network; Allocate an IP address range for the custom bridge network, and allocate addresses within the IP address range to each of the Docker containers, so that the Docker containers communicate through internal IP addresses; Configure the network isolation policy for the Docker container.

4. The method according to claim 3, characterized in that After the step of distributing the subtask and the corresponding Docker container to the distributed nodes, the method further includes: Configuring shared memory resources for the Docker container; The browser kernel in the Docker container is configured to run in a non-graphical interface environment.

5. The method according to claim 4, characterized in that After the step of distributing the subtask and the corresponding Docker container to the distributed nodes, the method further includes: An automatic cleanup mechanism for the browser kernel instance in the Docker container is set to recycle resources after the browser kernel instance ends.

6. A distributed browser collection system based on Docker, characterized in that: The Docker-based distributed browser collection system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the Docker-based distributed browser collection system to execute the method according to any one of claims 1 to 5.

7. A computer program product comprising instructions, characterized in that When the computer program product is run on a Docker-based distributed browser collection system, the Docker-based distributed browser collection system is enabled to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on a distributed browser collection system based on Docker, the distributed browser collection system based on Docker executes the method according to any one of claims 1 to 5.

Images