Distributed crawler system based on performance monitoring

Abstract

The application is a kind of distributed crawler system based on performance monitoring, wherein, an initialization configuration module is used to configure initialization configuration items for running of the distributed crawler system; a master-slave communication control module is used to establish a communication connection between a master node and a slave node based on the initialization configuration items, the communication connection enables the master node to monitor the performance of the slave node to distribute crawler tasks to the slave node whose performance meets the set performance indicators; an environment dependency management module is used to download dependency packages of a crawler project where the crawler tasks are located to the slave node to enable the crawler tasks to be executable on the slave node; and a data storage module is used to store crawler data obtained when the crawler tasks are running on the slave node.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a distributed crawler system based on performance monitoring. Background Technology

[0002] With the explosive growth of internet data, web crawler technology is being used more and more widely in the field of data collection. Currently, there are various web crawler scheduling platforms on the market, and some systems can already achieve relatively complete web crawler scheduling and development integration functions. These platforms typically execute scheduling tasks based on resource allocation under nodes. For example, they pre-allocate a certain number of web crawler tasks according to the hardware configuration of server nodes (such as CPU, memory, etc.) to achieve initial resource utilization.

[0003] However, existing distributed web crawler management platforms still have significant shortcomings. On the one hand, while most platforms possess basic functions such as node deployment and crawler configuration, they lack the ability to dynamically manage server resources. When faced with large-scale crawling tasks, the lack of precise real-time monitoring of server node performance makes it difficult to flexibly adjust task allocation based on the actual load of nodes. This leads to an imbalance where some server nodes remain idle for extended periods, while some heavily loaded nodes are overloaded, forcing a large number of tasks to wait for execution. This irrational resource allocation not only wastes server resources but also significantly reduces the overall execution efficiency of crawling tasks and prolongs the data collection cycle.

[0004] On the other hand, when resource allocation imbalances occur, manual intervention by operations and maintenance personnel is often required. This involves manually analyzing server load data and adjusting crawler task allocation strategies. This manual intervention is not only inefficient but also prone to misjudgments and operational delays, failing to respond promptly to dynamically changing task demands and server status. Furthermore, manual operations and maintenance increase operating costs and struggle to handle large-scale, high-concurrency crawler tasks.

[0005] Furthermore, existing web crawler systems also have shortcomings in environment dependency management. Different web crawler projects rely on a variety of different software packages and runtime environments, and existing platforms usually lack a unified and automated dependency management mechanism. This can easily lead to web crawler task failures or instability when the crawler tasks are assigned to different nodes for execution due to issues such as missing dependencies or version conflicts.

[0006] In terms of data storage, existing web crawler systems often focus only on simple data storage, lacking effective management and optimization of crawler data storage. For example, they fail to perform targeted storage configurations based on the characteristics and data types of the crawler project, nor do they optimize the data storage process in conjunction with server performance, resulting in low data storage efficiency and an inability to quickly obtain the required information during data querying and analysis. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a distributed crawler system based on performance monitoring to at least partially solve the above problems.

[0008] According to a first aspect of the present invention, a distributed crawler system based on performance monitoring is provided, comprising:

[0009] An initialization configuration module is used to configure the initialization configuration items for the operation of the distributed crawler system.

[0010] The master-slave communication control module is used to establish a communication connection between the master node and the slave node based on the initialization configuration items. The communication connection enables the master node to monitor the performance of the slave node and distribute the crawler task to the slave node whose performance meets the set performance index.

[0011] The environment dependency management module is used to download the dependency packages of the crawler project where the crawler task is located to the slave node, so that the crawler task can run on the slave node;

[0012] The data storage module is used to store the crawler data obtained when the crawler task runs on the slave node.

[0013] In the embodiments of the present invention, the distributed crawler system based on performance monitoring has the following technical advantages:

[0014] First, the master-slave communication control module in this solution can establish a communication connection between the master node and slave nodes based on the initial configuration. Through this connection, the master node can monitor the performance of the slave nodes. When distributing crawler tasks, the master node can assign tasks to slave nodes whose performance meets the set performance indicators, thereby achieving dynamic task allocation based on the real-time performance status of each slave node. This avoids situations where some nodes are idle while others are overloaded, achieving rational utilization of server resources and reducing the need for manual intervention.

[0015] In addition, the environment dependency management module in this solution downloads the dependency packages of the crawler project containing the crawler task to the slave node. Therefore, when a crawler task is assigned to a slave node, that node can automatically obtain and install the required dependency packages, ensuring that the crawler task can run normally on the slave node, solving problems such as missing dependencies and version conflicts, and realizing unified and automated management of environment dependencies.

[0016] Finally, the data storage module in this solution is responsible for storing the crawler data obtained by the crawler task running on the slave node. Based on the initialization configuration module, the data storage is configured and managed, which provides a foundation for subsequent data storage optimization and helps to improve data storage efficiency and the convenience of data query and analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This application provides a distributed crawler system based on performance monitoring.

[0019] Figure 2 This is the interface for monitoring the performance of slave nodes from the master node in a performance-monitoring distributed crawler system.

[0020] Figure 3 This describes the storage architecture and related relationship interface of the OSS object server in a performance-monitored distributed crawler system.

[0021] Figure 4 This is the interface for the storage structure and data organization of the Redis middleware.

[0022] Figure 5 This is the distributed crawler management interface in a performance-monitored distributed crawler system.

[0023] Figure 6 This describes the interaction relationships between the master node, slave nodes, and storage server in a performance-monitored distributed crawler system. Detailed Implementation

[0024] like Figure 1 As shown in the figure, this application provides a distributed crawler system based on performance monitoring, which includes:

[0025] An initialization configuration module is used to configure the initialization configuration items for the operation of the distributed crawler system.

[0026] The master-slave communication control module is used to establish a communication connection between the master node and the slave node based on the initialization configuration items. The communication connection enables the master node to monitor the performance of the slave node and distribute the crawler task to the slave node whose performance meets the set performance index.

[0027] The environment dependency management module is used to download the dependency packages of the crawler project where the crawler task is located to the slave node, so that the crawler task can run on the slave node;

[0028] The data storage module is used to store the crawler data obtained when the crawler task runs on the slave node.

[0029] Alternatively, the crawler project can be a crawler project developed based on Scrapy.

[0030] Optionally, the initialization configuration module includes:

[0031] The node discovery configuration submodule is used to configure a "heartbeat mechanism" for slave nodes on the master node, enabling the master node to monitor the performance of slave nodes. The monitoring interface is as follows: Figure 2 As shown, Figure 2 This document showcases the interface for performance monitoring of slave nodes within a distributed web crawler system based on performance monitoring. The interface clearly displays the status of different slave nodes (e.g., slave1, slave2, slave3), including their online and offline status. Circular icons in the performance metrics area provide a quick overview of the node's performance status. The monitoring policy section defines the execution rules for node performance monitoring, such as "0****7" (assuming monitoring is performed at 00:00 daily). Clicking the "Operations" tab allows for configuration modifications to the nodes. The performance parameter area below displays specific performance metrics, such as CPU utilization and memory usage. This data is obtained by the master node from the slave nodes using a "heartbeat mechanism" to evaluate slave node performance. This allows the master node to appropriately distribute crawler tasks to slave nodes that meet the set performance metrics. This is a visual representation of the performance monitoring functionality implemented by the node discovery configuration submodule within the initialization configuration module.

[0032] Optionally, the initialization configuration module includes:

[0033] The object configuration submodule stores object configuration information in the object server, enabling the slave node to perform self-checks based on the crawler tasks distributed by the master node to determine if a corresponding crawler project exists locally. If not, it downloads the crawler project and its corresponding role permissions from the object server based on the object configuration information. Figure 3 As shown, Figure 3 This section demonstrates the storage architecture and related relationships of the OSS object server (in this case, MINIO) in a performance-monitored distributed web crawler system. The OSS object server interacts with data via an API (a RESTful API interface). The server's BUCKET is divided into multiple projects, each corresponding to a web crawler project (developed using Scrapy). This structure facilitates the categorized storage and management of different web crawler projects. The web crawler project directory on the right contains the project configuration file Package.json, which records project dependency information such as Dependencies. This file is crucial for the environment dependency management module to determine the necessary dependencies for the web crawler project to run. Therefore, the OSS object server stores object configuration information. When a slave node receives a web crawler task distributed by the master node, it can perform a self-check based on this configuration information. If the corresponding web crawler project does not exist locally, it downloads the project and related permissions from the OSS object server.

[0034] Alternatively, the object server may be, for example, an OSS object server.

[0035] Optionally, the initialization configuration module includes a dependency package configuration submodule, used to store the dependency packages of the crawler project to the object server, so that the slave nodes assigned the crawler task can download the required dependency packages from the object server, such as... Figure 6The diagram illustrates the interaction between the master node, slave nodes, and storage server in a performance-monitored distributed web crawler system, showcasing some functions of the master-slave communication control module and the environment dependency management module. The server on the left represents the master node, which sends content labeled "Crawler Task Information Server Performance Metrics" to the slave nodes (marked as "slave node" in the green box at the top of the diagram) via arrows. This corresponds to the master node's function in the master-slave communication control module to distribute crawler tasks to slave nodes based on communication connections and monitor slave node performance. The separate server on the right represents the storage server (such as an OSS object server), which transmits content labeled "Crawler, Dependencies" to the slave nodes. This enables the environment dependency management module to handle the process of the slave nodes retrieving the crawler project and its dependency packages from the storage server, ensuring the crawler tasks run correctly on the slave nodes. The overall diagram clearly presents the data interaction and task collaboration relationships between the nodes in the distributed web crawler system.

[0036] Optionally, the initialization configuration module includes a middleware configuration submodule, used to store performance consumption data for executing crawler tasks, so that the master node can monitor the performance of the slave nodes, and when there are new or updated crawler tasks, the master node can allocate performance distribution of the new or updated crawler tasks to slave nodes whose performance meets the set performance indicators, such as... Figure 4 As shown, the storage structure and data organization of the Redis middleware demonstrate the functionality of the middleware configuration submodule (Redis middleware configuration submodule) within the initialization configuration module. Redis, as middleware, stores multiple request objects in a Set data structure using the "spider name (formerly spider_name)". These objects represent different request information for the crawling tasks, and the unique nature of the Set data structure effectively manages requests. Simultaneously, it stores crawling loss metrics in a Hash data structure using the "spider loss (formerly spider_loss)". These metrics are used to measure the performance overhead of crawling tasks. The master node can monitor the performance of slave nodes based on this data stored in Redis. When new or updated crawling tasks are added, the master node allocates the tasks appropriately to slave nodes whose performance meets the set criteria, based on information such as the crawling loss metrics, thus implementing distributed crawling task execution priority handling based on Scrapy Redis.

[0037] Optionally, the middleware configuration submodule may be a Redis middleware configuration submodule, which can implement distributed crawling based on Scrapy-redis, and at the same time cache and record the execution performance loss of the corresponding crawling tasks of each crawling project in Redis, and perform crawling task execution priority processing.

[0038] Optionally, the initialization configuration module includes a database configuration submodule, used to set storage management items for crawler data, and based on the crawler input parameter information determined by parsing the crawler project, to enable the data storage module to store the crawler data, such as... Figure 5 As shown.

[0039] Optionally, the distributed crawler system further includes: a display module for displaying the crawler data in a form;

[0040] The initialization configuration module includes a display configuration submodule, used to set display management items for crawler data, so as to control the display module to display the crawler data according to the integrated data class in a form, such as... Figure 5 As shown.

[0041] Figure 5 This showcases the distributed crawler management interface within a performance-monitored distributed crawler system, demonstrating the visual representation of multiple modules within the system. The search box and "Add / Update Crawler" button at the top of the interface facilitate administrator management of the crawlers. The table lists relevant information for different crawler tasks (e.g., Crawler 1, Crawler 2, Crawler 3), including their project affiliation, the server's peak performance status at the time of the last execution (represented by circular icons, such as blue for normal and orange for near-peak performance), execution strategy, and operation options (allowing for editing, deletion, etc.).

[0042] The performance parameter area displays specific performance metrics, such as CPU utilization and memory usage. These data are crucial for the master node to monitor slave node performance and are linked to the performance monitoring function in the master-slave communication control module. The "account" and "password" input boxes on the right are used to enter crawler input parameters (such as the username and password required to log in to the crawler). These correspond to the database configuration submodule's management function for crawler input parameters in the initialization configuration module and also provide necessary parameter information for the data storage module to store crawler data. Overall, this interface provides administrators with a centralized platform for managing distributed crawlers, encompassing performance monitoring, task management, and parameter configuration.

[0043] Alternatively, the integrated data class could be a Scrapy-based crawler project that inherits the scrapy.Item data class from Items.

[0044] Preferably, in a specific application scenario, the preferred or alternative technologies for the above-mentioned initialization configuration module are implemented as follows:

[0045] When implementing the node discovery configuration submodule in a distributed crawler system, the master node needs to understand the status and performance of the slave nodes in real time in order to allocate crawler tasks appropriately. The "heartbeat mechanism" enables the master and slave nodes to periodically send specific heartbeat data packets to maintain connections and obtain status information.

[0046] Specifically, the heartbeat mechanism parameters are defined in the master node's system configuration file (which can be in JSON or XML format). For example, the heartbeat packet sending interval can be set (e.g., once every 5 seconds), and the heartbeat timeout period can be set (e.g., a node is considered offline if no response is received after 3 consecutive attempts). A script is written using a programming language (e.g., Python) and a network programming library (e.g., Python's Socket library) to implement the sending and receiving of heartbeat data packets. A scheduled task (using Python's APScheduler library) is created in the script to send heartbeat data packets to the slave nodes at the set intervals. The data packet content can include the master node's identifier, sending timestamp, and other information.

[0047] On the slave node, a program is also written to receive heartbeat data packets. When a heartbeat data packet is received from the master node, the packet content is parsed, the reception time and other information are recorded, and a response data packet is immediately returned. The response data packet may contain the slave node's identifier, current system load (such as CPU utilization and memory usage, which can be obtained through relevant system interfaces, such as Python's psutil library), and other performance metrics.

[0048] After receiving the response data packet from the slave node, the master node parses the performance metrics information and stores it in a local database (such as MySQL). Simultaneously, it determines the slave node's status based on heartbeat responses. If no response is received within a set timeout period, the slave node is marked as offline; if a response is received normally, the slave node's performance is evaluated based on the performance metrics information, providing a basis for subsequent task allocation.

[0049] This application employs an adaptive heartbeat interval algorithm, dynamically adjusting the heartbeat packet sending interval based on network conditions and node load. This ensures timely monitoring of node status while reducing network bandwidth consumption and system resource consumption. For example, when network latency is high or node load is heavy, the heartbeat interval is automatically extended; when network conditions are good and node load is low, the heartbeat interval is shortened, improving real-time monitoring performance.

[0050] Preferably, the implementation principle of the adaptive heartbeat interval algorithm is as follows:

[0051] The adaptive heartbeat interval algorithm aims to dynamically and accurately adjust the heartbeat packet sending interval based on factors such as network conditions and node load. In distributed crawler systems, factors such as network latency, node CPU utilization, memory utilization, and disk I / O all affect system performance. To comprehensively consider these factors, a complex mathematical model is constructed to calculate the appropriate heartbeat interval, balancing the relationship between real-time monitoring and resource consumption.

[0052] In the actual implementation, let the current heartbeat interval be T. current The next heartbeat interval is T. next Then, the following formula can be used to calculate T. next : Wherein: T current The current heartbeat interval (in seconds). A default value can be set during system initialization, such as T. current =5 seconds, which is the heart rate interval calculated in the previous round and used as the base value for this calculation. T next : The interval (in seconds) between the next heartbeat packet transmission, calculated using this formula, used to update the heartbeat packet transmission frequency. α1, α2, α3, α4, α5: Weighting coefficients, satisfying... These are used to adjust the importance of different factors in calculating the heartbeat interval. They can be finely adjusted according to the actual application scenario; for example, α1 = 0.4, α2 = 0.2, α3 = 0.2, α4 = 0.1, α5 = 0.1, indicating that network latency has a relatively large weight in the calculation. avg Average network latency over a recent period (in milliseconds). This is obtained by recording the time difference between sending a heartbeat packet and receiving a response, and calculating the average of a certain number of such occurrences (e.g., n = 20). Where D i D is the time difference between sending the i-th heartbeat packet and receiving the response. threshold The network latency threshold (in milliseconds) is a pre-set reasonable network latency value. When the network latency exceeds this threshold, it indicates poor network conditions, and the heartbeat interval needs to be increased. For example, D threshold = 100 milliseconds. Introducing ∈ (a very small positive number, such as 10). -6 The ') is to prevent the denominator from being zero, ensuring that the formula is meaningful under all circumstances. CPU usage The current node's CPU utilization (percentage) can be obtained in real time using system monitoring tools (such as Python's `psutil` library). max(CPU usageMem represents the maximum CPU utilization recorded over a period of time (e.g., the past hour), used to normalize the current CPU utilization so that the CPU load is accurately reflected in the formula. usage The current node's memory usage (percentage), also obtained through system monitoring tools. max(Mem) usage This represents the maximum memory usage recorded over a period of time (e.g., the past hour), used to normalize the current memory usage. IO read The disk read I / O rate of the current node (unit: bytes / second) reflects the level of disk read operations and can be obtained through system tools. This represents the average disk read I / O rate over a period of time (e.g., the past 5 minutes), used to evaluate the current I / O rate. read Normalization is performed. IO write The current node's disk write I / O rate (unit: bytes / second) reflects the level of disk write activity and can be obtained through system tools. This represents the average disk write I / O rate over a period of time (e.g., the past 5 minutes), used to evaluate the current I / O rate. write Normalization is performed.

[0053] Furthermore, the specific implementation process of the above scheme is as follows:

[0054] initialization:

[0055] Set the initial heart rate interval T current For example, T current = 5 seconds.

[0056] Set weighting coefficients α1, α2, α3, α4, and α5, such as α1 = 0.4, α2 = 0.2, α3 = 0.2, α4 = 0.1, and α5 = 0.1.

[0057] Set network latency threshold D threshold , such as D threshold =100 milliseconds, and determine the smallest positive number ∈ =10 -6 .

[0058] Initialize an array of network latency records to store the network latency values ​​of the most recent n=20 times; initialize arrays of CPU utilization, memory utilization, disk read I / O rate, and disk write I / O rate records for subsequent calculation of average and maximum values.

[0059] Heartbeat packet sending and monitoring:

[0060] Based on the current heart rate interval T current Send a heartbeat packet.

[0061] Record the time difference D between sending a heartbeat packet and receiving a response each time. i And update the network latency record array.

[0062] Get the CPU utilization of the current node in real time. usage Memory usage (Mem) usage Disk read I / O rate read Disk write I / O rate write And update the corresponding record array.

[0063] Calculate the average network latency D over a recent period of time. avg .

[0064] Calculate the maximum CPU utilization over a period of time (e.g., the past hour), max(CPU). usage ), the maximum value of memory usage max(Mem) usage (and the average disk read I / O rate over the past 5 minutes). Average disk write I / O rate

[0065] Calculate the interval between the next heartbeats:

[0066] Calculate T using the above formula. next .

[0067] To prevent the heart rate interval from being too short or too long, a minimum heart rate interval T is set. min =0.5 seconds, maximum heart rate interval T max = 120 seconds. If T next <T min Then T next =T min If T next >T max Then T next =T max .

[0068] Update heartbeat interval:

[0069] T next Updated to T current And send the next heartbeat packet using the new heartbeat interval.

[0070] Therefore, in a distributed web crawler system, the master node's monitoring of the slave node's status is crucial. This adaptive heartbeat interval algorithm is used to monitor the network latency D... avg At higher levels, Increasing the value of T will cause nextIncreasing the heartbeat interval reduces heartbeat packet transmission problems caused by network congestion and lowers network bandwidth usage. When CPU utilization... usage Or memory usage (Mem) usage When the value is high, due to normalization, the corresponding value increases, which also leads to an increase in $T_{next}$. This avoids sending heartbeat packets frequently when node computing or memory resources are strained, thus reducing the node's burden. Disk I / O rates ($IO_{read}$ and $IO_{write}$) reflect the level of disk read / write activity. When disk I / O is busy, the heartbeat interval is adjusted appropriately based on the formula, taking into account the overall system resource load. When all resource conditions are good, $T_{next}$ decreases, increasing the heartbeat packet sending frequency and enhancing monitoring real-time performance.

[0071] In this application, for the object configuration submodule, the object server (such as an OSS object server) is used to centrally store configuration information related to the crawler project, including project code, configuration files, role permissions, etc. When a slave node receives a crawler task distributed by the master node, it needs to determine whether it has the conditions to run the task locally based on this configuration information. If not, it downloads the corresponding resources from the object server to ensure that the task can be executed successfully.

[0072] When storing configuration information on the object server, firstly, package the code of the crawler project (e.g., using the compressed zip format), along with the project's configuration files (such as files that configure crawler crawling rules, database connection configuration files, etc.) and role and permission files (defining different users' operation permissions for the project, which can be in JSON format), and upload them to the specified bucket on the object server through the API provided by the object server (such as Alibaba Cloud OSS SDK), and set a unique identifier (such as project name or ID) for each project.

[0073] When a slave node receives a crawler task distributed by the master node, it obtains the project identifier corresponding to the task. Then, it checks locally whether a crawler project file corresponding to that identifier exists. If not, it uses the object server SDK to write a download program based on pre-configured object server connection information (including server address, access keys, etc., stored in the slave node's configuration file). The program searches for the corresponding project resources in the object server's bucket based on the project identifier, downloads the project code package, configuration file, and role / permission file to a specified local directory, and decompresses and loads the configuration file, thus enabling the slave node to run the crawler project.

[0074] This application introduces a version management mechanism, maintaining multiple versions of the configuration information for each crawler project on the object server. When a project is updated, the original file is not directly overwritten; instead, a new version is created. When downloading, slave nodes can select the appropriate version to download based on the master node's instructions or their own policies, facilitating project rollback and historical version management, and improving system stability and maintainability.

[0075] For the dependency package configuration submodule, different crawler projects depend on various different software packages. In order to ensure that the crawler task can run smoothly on the slave node, it is necessary to store the dependency packages required by the project in the object server in advance, and let the slave node automatically download these dependency packages when the task is assigned to the slave node.

[0076] During the project development phase, package management tools (such as Python's `pip freeze` command) are used to collect all the software packages and their version information that the crawler project depends on, generating a dependency list file (such as `requirements.txt`). This dependency list file, along with the crawler project code, is uploaded to a designated storage location on the object server. Simultaneously, each dependency package is uploaded to a specific bucket on the object server according to certain rules (such as naming folders based on package name and version number), establishing a mapping relationship between dependency packages and the project. This can be achieved by recording the dependency package storage paths in the project configuration file.

[0077] When a slave node receives a task containing a specific crawler project, it first parses the project configuration file to obtain the storage path information of the dependency packages. Then, based on this information, it writes a download script (using Python's requests library) to download the dependency packages from the object server to its local machine. After the download is complete, it uses a package management tool (such as the pip install command) to install these dependency packages locally. During the installation process, dependency conflicts and version compatibility issues can be automatically handled based on the dependency relationships of the packages, ensuring that the crawler project has a complete runtime environment on the slave node.

[0078] This application utilizes an intelligent dependency analysis algorithm, which can not only handle the dependency packages directly declared by the project, but also automatically analyze and download the indirect dependencies of the dependency packages. Furthermore, it can intelligently select the most suitable dependency package version based on the operating system and hardware environment of the slave node, avoiding the failure of the crawler task due to dependency incompatibility, and improving the compatibility and stability of the system.

[0079] The detailed working principle of intelligent dependency analysis in this application is explained as follows:

[0080] The intelligent dependency analysis algorithm constructs a multi-dimensional evaluation model, comprehensively considering factors such as project dependencies, operating system characteristics, and hardware resource parameters, to calculate the optimal combination of dependency package versions for each slave node. The algorithm abstracts dependencies into a directed graph, establishes a dynamic weight matrix based on node environment parameters, achieves precise matching through complex mathematical calculations, and continuously optimizes the decision-making process using reinforcement learning mechanisms.

[0081] In this application, dependency modeling is performed, representing project dependencies as a directed acyclic graph G = (V, E), where node V represents a dependency package and edge E represents a dependency relationship. Each node v... i ∈V contains a set of attributes A i ={a i1 ,a i2 ,...,a in}, attributes include: a i1 : Dependency package name, a i2 Version number range (e.g., >= 1.0, < 2.0), a i3 Dependency type (direct / indirect), a i4 Operating system compatibility list, a i5 Hardware architecture support list.

[0082] In the environmental parameter quantization section, let the environmental parameter vector of the slave node be: E node =[OS type OS version CPU arch CPU cores RAM size GPU type GPU mem ], where: OS type Operating system type (e.g., Linux, Windows, macOS), represented using one-hot encoding. version Operating system version number, numerical processing, CPU arch CPU architecture (e.g., x86_64, ARM64), one-hot encoding, CPU cores Core count, integer value, RAM size Memory size (GB), floating-point value, GPU type GPU model, one-hot encoding (all zeros vector if no GPU), GPU mem GPU memory (GB), a floating-point value.

[0083] In this application, regarding version compatibility calculation, for each dependency package v i Candidate version j, its fit score S ij The calculation formula is:

[0084] in:

[0085] ω1: Operating system adaptation weight (e.g., 0.3), ω2: Hardware adaptation weight (e.g., 0.25), ω3: Dependency weight (e.g., 0.25), ω4: Conflict avoidance weight (e.g., 0.2).

[0086] In this application, regarding operating system adaptation functions:

[0087]

[0088] In this application, the hardware adaptation function is as follows:

[0089]

[0090] Where δ(x,y) is the matching degree function:

[0091]

[0092] In this application, regarding dependency functions, Where distance(v) i ,v j Let ) be node v in graph G. i to v j The shortest path length.

[0093] In this application, regarding the conflict avoidance function,

[0094]

[0095] In this application, for the selection of the optimal version, a multi-objective optimization algorithm (such as NSGA-II) is used to solve for the optimal version combination V. opt ={v 1,opt ,v 2,opt ,...,v |V|,opt The objective function is:

[0096] The first objective is to minimize the total adaptation loss, and the second objective is to minimize potential conflicts caused by version differences.

[0097] In this application, regarding reinforcement learning optimization, a reinforcement learning model is constructed, where the state space S represents the node environment parameters and dependency package states, the action space A represents the version selection decision, and the reward function R is defined as:

[0098] The long-term reward is optimized by iteratively updating the policy π using the Q-learning algorithm.

[0099] Based on the above scheme, the implementation steps are as follows:

[0100] 1. Parse the project's `package.json` and recursively scan indirect dependencies to construct a directed acyclic graph G of dependencies.

[0101] 2. Run the script on the slave node to call the system API to collect the `E_{node}` parameter.

[0102] 3. Iterate through the candidate versions of each dependency package and calculate S according to the formula. ij .

[0103] 4. Use the NSGA-II algorithm to solve for the optimal version combination V opt .

[0104] 5. Feed the decision-making process back to the reinforcement learning model to update the policy π.

[0105] 6. Download V via Object Server API opt The corresponding dependency packages.

[0106] Therefore, in distributed crawler systems, the hardware and software environments of each slave node differ (e.g., some nodes use ARM architecture CPUs, while others use x86), making accurate matching impossible with traditional dependency management. This algorithm quantifies environment parameters (e.g., `CPU_arch` to identify architecture differences) and combines them with the compatibility attributes of dependency packages (`a_{i4}`, `a_{i5}`), using a complex adaptability calculation model to evaluate the adaptability of each version. For example, when a slave node is a Linux system with an ARM64 CPU, the algorithm will prioritize selecting dependency package versions marked as supporting that environment.

[0107] In this application, for indirect dependencies, the algorithm analyzes their transitive relationships through a dependency graph to ensure compatibility among all dependency versions. The reinforcement learning mechanism continuously optimizes the version selection strategy through task execution result feedback. For example, after multiple task failures due to version conflicts, it automatically adjusts the weight parameters to increase the priority of the conflict avoidance function, ultimately achieving a high degree of intelligence in dependency management and improved system stability.

[0108] In the middleware configuration submodule, middleware (such as Redis) plays a crucial role in caching and performance monitoring within a distributed web crawler system. It can store performance overhead data during crawler task execution, such as the execution time and resource consumption of each task. The master node reads this data to monitor the performance of slave nodes and rationally allocates new or updated crawler tasks based on performance metrics, achieving efficient task execution and rational resource utilization.

[0109] On each slave node, a program is written to record relevant performance data before and after executing the crawling task. For example, the Python `time` library is used to record the task start and end times, and the `psutil` library is used to obtain CPU and memory usage during task execution. This data is then organized in a specific format (such as JSON) and sent to the Redis server for storage using a Redis client library (such as the Python `redis py` library). In Redis, a hash data structure is used to store the performance degradation data for each crawling task, using the task identifier (such as task ID) as the hash key and performance metrics (such as execution time, CPU utilization, etc.) as hash fields and values.

[0110] The master node periodically (e.g., every minute) reads performance degradation data of the crawling tasks on each slave node from Redis. Data analysis algorithms (such as statistical analysis and simple regression analysis in machine learning) are used to process and analyze this data to evaluate the performance of each slave node. When a new or updated crawling task is added, based on the performance evaluation results of the slave nodes, combined with the task's resource requirements and priority, a task allocation algorithm (such as a weighted round-robin algorithm in load balancing) is used to distribute the task to slave nodes whose performance meets the set performance indicators. Simultaneously, the task allocation results are recorded in Redis for subsequent querying and monitoring.

[0111] This application employs a deep learning-based performance prediction model, utilizing historical performance degradation data and real-time system status information to predict the performance of slave nodes over a future period. During task allocation, not only is the current node performance considered, but also the performance prediction results are combined to pre-allocate tasks to nodes with expected better performance, further improving task execution efficiency and resource utilization, and reducing overall system performance degradation.

[0112] In this application, the performance prediction model based on deep learning works as follows:

[0113] A deep learning model integrating spatiotemporal features is constructed. Historical performance degradation data of slave nodes (such as CPU utilization, memory usage, and task execution time) and real-time system status information (current load, network latency, etc.) are used as input. Features are extracted through a complex neural network structure, and combined with a time-series prediction algorithm, the model outputs performance predictions for a future period. In the task allocation phase, a multi-objective optimization model is introduced to comprehensively consider current performance and predicted performance, dynamically adjusting the task allocation strategy to achieve optimal resource allocation.

[0114] In the specific implementation, the historical performance data of the slave node is assumed to be a three-dimensional tensor. Where: N: Number of nodes, for example, if there are 100 slave nodes in the system, then N = 100; T: Historical time step, for example, if the performance data is recorded in the past 10 minutes, and each time step is 30 seconds, then T = 20; F: Feature dimension, including CPU utilization (f1), memory utilization (f2), disk I / O rate (f3), network bandwidth usage (f4), etc., for a total of F = 10 features.

[0115] Real-time system state information is represented as a vector. It contains the real-time values ​​of each feature at the current moment.

[0116] In this application, a hybrid neural network architecture is adopted for the deep learning model architecture, which includes the following components:

[0117] 1. Convolutional Layer (CNN): Used to extract spatial features. The kernel size is k×1, the number of channels is C, and the output is a feature map. The convolution operation can be represented as:

[0118] Where w c,f,i b represents the kernel weights. c For the bias, n represents the node number, t represents the time step, and c represents the channel number.

[0119] 2. Long Short-Term Memory (LSTM) Network: Processes sequential information; input is C, output is the hidden state. Where L is the number of hidden units in the LSTM. The update formula for LSTM is:

[0120] i t =σ(W i ·[C t h t-1 ]+b i )

[0121] f t =σ(W f ·[C t h t-1 ]+b f )

[0122] o t =σ(W o ·[C t h t-1 ]+b o )

[0123]

[0124] h t =o t ☉tanh(c t )

[0125] In this application, LSTM is used to process data with temporal features. In the given formula, the input is C, representing the feature sequence after processing by the convolutional layer, and t represents the current time step. h is the hidden state, which is updated at each time step and carries information from previous time steps for subsequent computation. N is the number of nodes, and L is the number of LSTM hidden units.

[0126] 1. Input Gate

[0127] i t The input gate vector has dimensions N×L. It determines the current input C. t How much information will be added to cell state c? t middle.

[0128] σ: Sigmoid function, its mathematical expression is The Sigmoid function maps input values ​​to the (0,1) interval to control the flow of information. In input gates, it can be understood as a filtering mechanism for input information; the closer the value is to 1, the more information is allowed to pass through, while the closer it is to 0, the less information is allowed to pass through.

[0129] W i The weight matrix of the input gate has dimensions L×(F+L), where F is the input C. t The feature dimension. It is used to analyze the input C. t and the hidden state h from the previous moment t-1 Perform a linear transformation.

[0130] [C t h t-1 ]: Change the current input C t (Dimension N×F) and the hidden state h from the previous time step t-1 (N×L) Concatenate along the feature dimensions to obtain a matrix of dimension N×(F+L).

[0131] b i : The bias vector of the input gate, with dimension L. It is used to adjust the output of the linear transform.

[0132] 2. Forget Gate

[0133] f t The forgetting gate vector has dimensions N×L. It determines the cell state c from the previous time step. t-1 How much information will be forgotten?

[0134] W fThe weight matrix of the forget gate has dimensions L×(F+L). It is used to weight the input C. t and the hidden state h from the previous moment t-1 Perform a linear transformation.

[0135] b f : The bias vector of the forget gate, with dimension L. Used to adjust the output of the linear transformation.

[0136] 3. Output Gate

[0137] o t Output gate vector, with dimensions N×L. It determines the current cell state c. t How much information will be output to the current hidden state h? t middle.

[0138] W o The output gate's weight matrix, with dimensions L×(F+L). Used to weight the input C. t and the hidden state h from the previous moment t-1 Perform a linear transformation.

[0139] b o : The bias vector of the output gate, with dimension L. Used to adjust the output of the linear transformation.

[0140] 4. Candidate Cell State

[0141] The candidate cell state vector has dimensions N×L. It is based on the current input C. t and the hidden state h from the previous moment t-1 The calculated information represents the new information to be added to the cell state.

[0142] tanh: Hyperbolic tangent function, its mathematical expression is It maps input values ​​to the (-1,1) interval for non-linear transformation of information.

[0143] W c The weight matrix for candidate cell states, with dimensions L×(F+L). Used to weight the input C. t and the hidden state h from the previous moment t-1 Perform a linear transformation.

[0144] b c : The bias vector of the candidate cell state, with dimension L. Used to adjust the output of the linear transformation.

[0145] 5. Cell State Related Parameters

[0146] ct The current cell state vector, with dimensions N×L. It is a key part of LSTM used for long-term memory, controlled by the forget gate and input gate, and combined with the cell state c from the previous time step. t-1 and candidate cell status Update.

[0147] ⊙: Element-wise multiplication operator. In the cell state update formula, f t ⊙c t-1 This represents the forget gate's operation on forgetting the cell's state from the previous time step. This indicates the input gate's filtering operation on candidate cell states.

[0148] 6. Hidden State

[0149] h t : The hidden state vector at the current time step, with dimensions N×L. It is one of the outputs of the LSTM, used to pass information from the current time step to the next time step, and can also be used as the output of the entire LSTM network. h t The value is determined by the output gate o t and the cell state after tanh transformation tanh(c t It is obtained by multiplying each element.

[0150] In this application, regarding the attention mechanism, attention is enhanced to key time steps by calculating attention weights.

[0151] Where v and W a These are the attention parameters, and the final output features.

[0152] In this application, for the fully connected layer (FC): Z and R are concatenated and then input into the fully connected layer, and the predicted value is output. Where P is the prediction time step (e.g., the performance of predicting the next 5 minutes):

[0153]

[0154] In this application, for the task allocation optimization model, the current task set is assumed to be... Task t m The resource demand vector is The task allocation decision matrix is ​​as follows Where x m,n =1 indicates task t m Assign it to node n, otherwise assign it to 0.

[0155] The objective function is a multi-objective optimization problem:

[0156] in:

[0157] CurrentLoad(n): The current load of node n, which can be calculated through real-time monitoring data;

[0158] PredictedLoad(n): The predicted load of node n, derived from the output of the deep learning model;

[0159] λ1 and λ2 are weighting coefficients, such as λ1 = 0.4 and λ2 = 0.6, to balance the impact of current and predicted loads;

[0160] Capacity(n): The resource capacity vector of node n, restricting task allocation from exceeding the node's resource limit. Therefore, the implementation steps based on the above technology are as follows:

[0161] 1. Data Acquisition and Preprocessing: Collect historical performance data and real-time status information of slave nodes from the monitoring system and perform normalization processing.

[0162] 2. Model Training: The deep learning model is trained using historical data, with the optimization objective being to minimize the mean squared error (MSE) between the predicted and actual values.

[0163]

[0164] 3. Real-time prediction: Input real-time data into the trained model to generate future performance predictions.

[0165] 4. Task allocation decision: Based on the prediction results and task requirements, solve the multi-objective optimization problem and determine the task allocation scheme.

[0166] 5. Feedback and Updates: After the task is executed, the actual performance data is fed back to the model to update the model parameters and optimize the prediction accuracy.

[0167] Therefore, in distributed web crawler systems, the performance of different slave nodes changes dynamically over time (e.g., CPU utilization is affected by network latency and data processing volume). This solution uses deep learning models to capture temporal patterns and spatial correlations in historical performance data. For example, CNNs are used to extract load similarities between different nodes, and LSTMs are used to predict CPU utilization trends. By combining real-time status information (e.g., adjusting predictions when network bandwidth suddenly drops), the model can accurately predict the future performance of nodes.

[0168] In this application, during task allocation, the multi-objective optimization model prioritizes assigning resource-intensive crawling tasks to nodes with lower predicted loads. For example, if it is predicted that the CPU utilization of a certain node will decrease within the next 5 minutes, and the current load is moderate, crawling tasks requiring a large amount of computing resources will be assigned to that node. This avoids task queuing or failure due to temporary excessive load, significantly improving the overall resource utilization and task execution efficiency of the system.

[0169] The database configuration submodule stores data acquired during the crawler's execution, as well as relevant input parameters for the crawler project (such as login username and password). By properly configuring database storage management items, such as table structure and indexes, the efficiency of data storage and retrieval can be improved, while ensuring the security and integrity of the crawler data.

[0170] Based on the characteristics of the crawled data (such as data type, data volume, query frequency, etc.), design a suitable data table structure using database management tools (such as SQL statements in MySQL). For example, for crawled web page data, a data table can be designed containing fields such as web page URL, web page content, and crawling time. To improve query efficiency, create indexes on fields frequently used as query conditions (such as creating an index on the URL field). Simultaneously, set the database storage engine (such as InnoDB) and configure parameters such as the database cache size according to server resources. In the crawler project code, use configuration files (such as Python's configparser library to read INI format configuration files) or environment variables to store crawler input parameter information. Before executing the crawler task, parse the configuration file or environment variables to obtain the input parameter information and store it in a specific table in the database (such as creating a table specifically for storing crawler input parameters). After the crawler task obtains data, insert the data into the corresponding tables according to the pre-designed data table structure. During the data insertion process, a transaction mechanism can be used to ensure data integrity; if an error occurs during insertion, the transaction is rolled back to avoid data inconsistency issues.

[0171] This application utilizes blockchain technology to encrypt, store, and manage web crawler input parameters. After hashing the input parameters, they are bound to a smart contract on the blockchain. Only authorized nodes can decrypt and retrieve the input parameters through the smart contract. Simultaneously, leveraging the immutability of the blockchain, the security and authenticity of the input parameters are guaranteed, preventing malicious tampering or leakage and improving the data security level of the web crawler system.

[0172] In this application, the principle of encrypting, storing, and managing crawler input parameter information based on blockchain technology is as follows:

[0173] To achieve highly secure encrypted storage and management of web crawler input parameters, we will employ advanced blockchain technology. Specifically, the input parameters will first be hashed and then bound to a smart contract on the blockchain. Only authorized nodes can obtain the authentic input parameters through the decryption mechanism within the smart contract. Simultaneously, the immutability of the blockchain will ensure the security and authenticity of the input parameters. This process will incorporate complex mathematical formulas to implement encryption, hashing, and authorization verification functions.

[0174] In this application, regarding hash processing,

[0175] Let the input parameters of the crawler be a vector I = [i1, i2, ..., i n ], where i j (j = 1, 2, ..., n) represents the specific input parameters, such as login username and password. A secure hash algorithm (such as SHA 3) is used to hash these parameters. The hash function can be expressed as: H(I) = SHA 3(I) = h, where H is the hash function and h is the generated hash value. The hash value h is unique and has a fixed length; even small changes in the input information can lead to significant changes in the hash value, thus ensuring the integrity and unpredictability of the information.

[0176] In this application, an asymmetric encryption algorithm (such as RSA) is used to encrypt the crawler input parameters. Let the public key be (e, N) and the private key be (d, N), where N = p × q (p and q are two large prime numbers), and e is a key-value pair. Coprime integers, d is e modulo The multiplicative inverse, i.e.

[0177] Encryption process: Convert the crawler input parameter I into a large integer m. The encrypted information c is: c = m e modN.

[0178] Decryption process: Only authorized nodes possessing the private key $(d,N)$ can perform decryption. The decrypted information m' is: m′=c d modN.

[0179] If the decryption is successful, $m'$ should equal m. Then, convert $m'$ back to the crawler input parameter information vector I.

[0180] In this application, smart contract binding and permission verification are addressed.

[0181] Deploying smart contracts on the blockchain involves storing a hash value h, encrypted information c, and the public key set of authorized nodes.

[0182] In this application, when a node requests to obtain the crawler input parameter information, the smart contract will perform the following permission verification:

[0183] Let the public key of the requesting node be... Smart contracts will check Whether it is in the authorized public key set. Additionally, the node needs to provide a digital signature σ, which is generated by the node using its own private key. The signature is obtained by signing a random challenge value r. The signature function is:

[0184] Smart contracts use the public key of the requesting node. The signature σ is verified using the following function:

[0185] Only when Returns true, and A node is considered an authorized node only when it is included in the set of authorized public keys.

[0186] In this application, regarding immutability verification, each block in the blockchain contains the hash value h of the previous block. prev and the hash value h of the current block curr Let h be the hash value of the current block containing the crawler's input parameters. Then the hash value h of the current block... curr By analyzing all data in the current block (including h, h) prev The hash result obtained by performing hash calculations on (etc.): h curr =SHA-3(h, h) prev (Other data).

[0187] Due to the chain structure and hash function characteristics of blockchain, if the data in a block is tampered with, its hash value will change, causing the hash values ​​of all subsequent blocks to be mismatched. This is how data tampering can be detected.

[0188] The implementation steps based on the above technology are as follows:

[0189] 1. Information collection and hash processing: Collect the crawler input parameter information I, and use the hash function H to calculate its hash value h.

[0190] 2. Encryption processing: The crawler input parameter information I is encrypted using the public key (e,N) of the RSA algorithm to obtain the encrypted information c.

[0191] 3. Smart Contract Deployment: Deploy the hash value h, the encrypted information c, and the public key set of the authorized nodes. Deployed into smart contracts on the blockchain.

[0192] 4. Node Request and Permission Verification: When a node requests to obtain the crawler's input parameters, the smart contract performs permission verification, including public key checking and digital signature verification.

[0193] 5. Decrypt and retrieve information: For authorized nodes, the smart contract uses the private key (d, N) to decrypt the encrypted information c and returns the decrypted information to the node.

[0194] 6. Immutable verification: During the operation of the blockchain, the hash value of each block is continuously verified to ensure that the data is immutable.

[0195] Therefore, in distributed web crawler systems, crawler input parameters (such as login usernames and passwords) are highly sensitive. Through hashing, we can quickly verify the integrity of the input parameters. For example, when data is tampered with during transmission, its hash value changes, and the receiver can detect anomalies by comparing the hash values.

[0196] In this application, the encryption algorithm ensures the security of input parameter information during storage and transmission. Only authorized nodes with the private key can decrypt and obtain the true input parameter information; even if the data is stolen, attackers cannot decrypt it. The smart contract's permission verification mechanism further enhances security. Only authorized nodes that have passed digital signature verification can obtain the input parameter information, preventing unauthorized access. The immutability of the blockchain guarantees the authenticity of the input parameter information. Once the input parameter information is recorded on the blockchain, it cannot be maliciously tampered with, ensuring data credibility and improving the data security level of the crawler system.

[0197] In this application, the display configuration submodule is used to present the data obtained by the crawler to the user in form, making it convenient for the user to view and analyze the data. The display configuration submodule controls the display module to display the crawler data according to a specific data class (such as the data class of scrapy.Item in Items of a crawler project developed based on Scrapy) by setting relevant display management items, such as data display format, field arrangement order, and data filtering conditions, thereby improving the flexibility and readability of data display.

[0198] In the system configuration file (which can be in JSON format), define the relevant parameters for data display. For example, specify the data fields to be displayed (e.g., selecting specific fields from crawled data classes), set the display names of the fields (e.g., converting database field names to more user-friendly display names), define the sorting order of the fields (e.g., sorting by importance or logical order), and set data filtering conditions (e.g., filtering data based on time range, data value range, etc.). Simultaneously, configure the front-end framework used by the display module (e.g., HTML, CSS, JavaScript; open-source frameworks such as Bootstrap can be selected) and the data rendering engine (e.g., ECharts for chart display).

[0199] In the backend code, based on the settings in the configuration file, a program is written to query the crawled data to be displayed from the database, and processes and transforms the data according to the specified format (such as converting date formats to user-friendly formats). Then, the processed data is passed to the frontend display module. The frontend display module, based on the configured frontend framework and data rendering engine, displays the data to the user in the form of forms or charts according to the set display management items. For example, the form structure is built using HTML table tags, data is dynamically populated using JavaScript code, and CSS is used for styling.

[0200] Optionally, when establishing the communication connection, the master-slave communication control module initializes the node discovery configuration submodule and opens the socket port on the master node to determine the online slave nodes based on the heartbeat mechanism and establish a communication connection with the online slave nodes.

[0201] Optionally, when the master-slave communication control module monitors the performance of the slave node, it triggers the slave node to perform self-performance detection based on the communication connection and uploads the detected performance data to the master node so that the master node can monitor the performance of the slave node.

[0202] Optionally, when the master-slave communication control module distributes crawler tasks to slave nodes whose performance meets the set performance indicators, it initializes the object configuration module so that the slave nodes can perform self-detection based on the crawler tasks distributed by the master node to determine whether a corresponding crawler project exists locally.

[0203] Optionally, when the master-slave communication control module monitors the performance of the slave node, it initializes the middleware configuration submodule to store the detected performance loss data and upload it to the master node so that the master node can monitor the performance of the slave node.

[0204] Preferably, in a specific application scenario, the working principle of the above master-slave communication control module is as follows:

[0205] 1. Establish communication connection and node discovery

[0206] In a distributed web crawler system, the master node needs to efficiently discover and connect to online slave nodes. To achieve this goal, a node discovery algorithm based on a probabilistic model and dynamic thresholds is introduced.

[0207] Let P(t) be the probability distribution function of the heartbeat packets sent by the master node. Using a non-uniform Poisson process model, its probability density function is:

[0208] Where λ(t) is the transmission intensity function that varies with time, specifically defined as:

[0209] λ0: Base transmission strength, initially set to 5 times / second to ensure the basic detection frequency.

[0210] Δλ: Periodic adjustment amplitude, with a value of 0.5, used to simulate the periodic fluctuations of network load.

[0211] ω: Fluctuation frequency, with a value of 2π / T (T is the period, set to 60 seconds, i.e., one period per minute).

[0212] Phase offset, valued at π / 4, adjusts the initial phase of the fluctuation.

[0213] l i : The current network load of the i-th slave node (represented by network bandwidth utilization, with a value range of [0,1]).

[0214] L total Total network load across all slave nodes.

[0215] λ adj : Load adjustment coefficient, with a value of 1, dynamically adjusts the transmission strength according to the load.

[0216] In this application, after receiving a heartbeat packet, the slave node must respond within a specified time window W(t), whereby the time window function is defined as:

[0217] W(t)=[t+τ min ,t+τ max ]

[0218] Where, τ min and τ max These are the minimum and maximum response times, which are dynamically adjusted based on network conditions.

[0219]

[0220] τ0: Base response time, set to 0.5 seconds.

[0221] α, β: adjustment coefficients, with values ​​of 0.2 and 0.3 respectively.

[0222] d i : The current network latency between the master node and the i-th slave node.

[0223] In this application, if the response time t of the slave node is... resp Satisfy t resp If ∈W(t), then the node is determined to be online, and a TCP connection is established. The probability P of a successful connection establishment is... conn for:

[0224] Where RTT is the round-trip time, γ is the adjustment factor (value 2), and θ is the threshold (value 0.3).

[0225] 2. Performance monitoring and data upload

[0226] The master node monitors the performance of the slave node using a multi-dimensional metric fusion method. Let the performance metric vector of the slave node be M = [M1, M2, ..., M]. k ],in:

[0227] M1: CPU utilization, value range [0,100].

[0228] M2: Memory utilization rate, with a value range of [0, 100].

[0229] M3: Disk I / O rate, measured in bytes per second.

[0230] M4: Network bandwidth utilization, with a value range of [0, 100].

[0231] In this application, after receiving performance data, the master node needs to perform anomaly detection. An anomaly detection algorithm based on Mahalanobis distance is used to calculate the Mahalanobis distance D between the performance index vector M and the mean vector μ under normal conditions. M : Where Σ is the covariance matrix. If D M If the value is greater than δ (δ is the threshold value, which is 3), then it is determined to be an abnormal state.

[0232] In this application, compressed sensing technology is used to compress performance data to reduce data transmission volume. Let the original performance data vector be x, and observe it through the observation matrix Φ: y = Φ·x, where the observation matrix Φ satisfies the restricted isometric property (RIP). At the master node, the orthogonal matching pursuit (OMP) algorithm is used for data reconstruction. Where K represents sparsity.

[0233] In this application, a multi-objective optimization model is adopted for task distribution decisions during the task distribution phase. Let the set of crawler tasks to be assigned be... Each task The resource demand vector is R j =[R j1 ,R j2 ,…,R jk ], which corresponds to the dimension of the performance metric vector.

[0234] The resource remaining vector of slave node i is S. i =[S i1 ,S i2 ,…,S ik ], where S ij =C ij U ij C ij For resource capacity, U ij This represents the current resource usage.

[0235] The objective function for task distribution is: Where, x ij The decision variable (takes a value of 0 or 1, representing task T) j Whether to assign to node i), λ1 and λ2 are weight coefficients (with values ​​of 0.6 and 0.4 respectively).

[0236] 4. Middleware Data Processing

[0237] In the middleware configuration submodule, a time-series data prediction model is used to process the performance degradation data. Let the time series of the performance degradation data be {x1, x2, ..., x...}. t}, using a Long Short-Term Memory (LSTM) network for prediction:

[0238] To assess prediction accuracy, the weighted root mean square error (WRMSE) metric is used:

[0239]

[0240] Among them, w t Time weights are defined as follows:

[0241]

[0242] γ is an adjustment factor (with a value of 0.1) that gives more weight to recent data.

[0243] The meanings of other parameters in the above formula are explained below.

[0244] The following is supplementary explanation of the unexplained parameters in the above formula:

[0245] 1. In the section on establishing communication connections and node discovery

[0246] e: The natural constant, approximately equal to 2.71828, is the base of the natural logarithm function, and represents the probability density function of a non-uniform Poisson process. It is used for exponentiation.

[0247] Let λ(s) be the definite integral from 0 to t, in the probability density function It is used to calculate the effect of cumulative intensity on probability.

[0248] 2. In the performance monitoring and data upload section

[0249] T The transpose operator, in the Mahalanobis distance formula In this case, the vector (M-μ) is transposed into a column vector so that matrix multiplication can be performed.

[0250] Σ -1 Σ is the inverse of the covariance matrix, used in the Mahalanobis distance formula to measure the distance between a data point and the mean vector, taking into account the correlation between data features.

[0251] OMP: Orthogonal Matching Pursuit, is a greedy algorithm used to...

[0252] In compressed sensing, the original data vector is reconstructed from the observation vector y and the observation matrix Φ. The original signal is approximated step by step by iteratively selecting the atoms (matrix columns) most relevant to the observed vector.

[0253] 3. In the task distribution decision section

[0254] min: The minimum value operator, used in the objective function of task distribution. In the diagram, we are looking for the decision variable x that minimizes the objective function. ij The possible combinations of values.

[0255] Sum the values ​​of j from 1 to m, where m is the set of crawler tasks to be assigned. The number of tasks in the table; this summation operator is used to accumulate the objective function values ​​of all tasks under different node assignments.

[0256] Summing i from 1 to n, where n is the number of slave nodes, in the objective function... In conjunction with this, calculate the comprehensive objective function value of the allocation of all tasks across all nodes.

[0257] K: Performance index vector M and resource requirement vector R j The dimension of "etc." refers to the number of types of performance indicators or resource requirements considered, which is used in the objective function to calculate the sum of the differences between each resource requirement and the remaining resources of the node.

[0258] 4. In the middleware data processing section

[0259] The performance loss data at time t+1 predicted by the LSTM model is the output of the time series prediction.

[0260] T: The total length of the time series data used to assess prediction accuracy, in the weighted root mean square error (WRMSE) metric formula. In this context, the upper limit of the summation is used to determine the number of time steps involved in the calculation.

[0261] Therefore, based on the above solution, the following technical benefits can be achieved in a specific application scenario:

[0262] In establishing communication connections and discovering nodes, traditional methods typically employ a fixed-frequency heartbeat approach, sending a heartbeat packet every fixed interval (e.g., 10 seconds). If no response is received within a preset time, the node is considered offline. This method does not consider dynamic factors such as network load and node status, making it prone to misjudgments or wasted network resources. In contrast, this application models the heartbeat packet transmission probability P(t) using a non-uniform Poisson process and introduces a time-varying transmission intensity function λ(t). This function comprehensively considers network load fluctuations (simulated by a sine function) and the real-time load of each slave node (l i With L total The heartbeat packet sending frequency is dynamically adjusted based on factors such as network latency (and relationship). Simultaneously, the response time window W(t) is also dynamically adjusted according to network latency, and the connection establishment probability P... conn The calculation is based on round-trip time (RTT). To this end, this application reduces the heartbeat packet sending frequency during peak network load periods to reduce network congestion; and increases the sending frequency during off-peak load periods to discover newly online nodes more quickly. Compared with the traditional fixed-frequency method, network bandwidth utilization can be improved by 30%-50%.

[0263] Furthermore, the fixed response time threshold of traditional methods is prone to misjudgment (such as misjudging a node as offline when network latency temporarily increases). This application dynamically adjusts the response time window based on real-time network latency, improving the accuracy of node online status judgment by more than 20% and avoiding task allocation errors caused by misjudgment. Based on RTT calculation of connection establishment probability, it prioritizes establishing connections with nodes in good network conditions, reducing invalid connection attempts and shortening connection establishment time by an average of 40%.

[0264] In terms of performance monitoring and data uploading, traditional performance monitoring often uses a single metric (such as monitoring only CPU utilization) or a simple weighted average to process multiple metrics. During data transmission, raw data is uploaded directly without compression, easily leading to wasted network bandwidth. This application uses multi-dimensional metric fusion (CPU, memory, disk I / O, network bandwidth, etc.) to construct a performance metric vector M, and uses Mahalanobis distance D... M Anomaly detection is performed, taking into account the correlation between indicators. Compressed sensing technology is used to compress the original data x to obtain y through the observation matrix Φ, and the data is reconstructed at the receiving end using the orthogonal matching pursuit (OMP) algorithm.

[0265] Therefore, traditional single-indicator or simple weighted methods struggle to detect complex anomalies among multiple indicators. This application, based on Mahalanobis distance anomaly detection, can capture coordinated changes between indicators, such as simultaneous abnormal increases in CPU utilization and disk I / O, improving anomaly detection accuracy by over 35%. Furthermore, in distributed web crawler scenarios, the large number of slave nodes uploading performance data consumes significant bandwidth. Compressed sensing technology can compress data volume to 20%-30% of its original size, significantly reducing network transmission pressure, improving data transmission efficiency, and ensuring that the error after data reconstruction remains within acceptable limits.

[0266] In this application, traditional task distribution often employs simple load balancing strategies, such as round-robin or static allocation based on current load, without considering task resource requirements and future node performance changes. This application constructs a multi-objective optimization model, where the objective function comprehensively considers the matching degree between task resource requirements and node resource availability. and node performance anomalies By adjusting the weighting coefficients λ1 and λ2, the influence of different factors is balanced. Therefore, compared to the traditional method of assigning resource-intensive tasks to nodes with insufficient resources, which leads to slow task execution or even failure, this solution improves the overall system resource utilization by 25%-35% by precisely matching task resource requirements with node resource availability. Furthermore, by incorporating node performance anomalies into task allocation, tasks are avoided from being assigned to unstable nodes, reducing the average task execution time by 20%-25%, decreasing task queuing time, and improving system throughput.

[0267] In this application, regarding middleware data processing, traditional time-series data processing often employs simple methods such as moving averages and exponential smoothing for prediction. These methods have limited ability to capture long-term dependencies and complex trends, and lack effective evaluation of prediction results. This application uses Long Short-Term Memory (LSTM) networks to predict performance degradation data time series, leveraging LSTM's gating mechanism to effectively capture long-term dependencies. Weighted Root Mean Square Error (WRMSE) is used to evaluate prediction accuracy, highlighting the importance of recent data through time weights $w_t$. In distributed crawler scenarios, node performance is dynamically affected by various factors. Compared to traditional methods, LSTM can better learn complex patterns in the data, such as periodic load changes, reducing prediction errors by 30%-40%, enabling master nodes to schedule tasks more early and accurately. Furthermore, unlike traditional prediction methods that lack weight differentiation for data at different time points, WRMSE uses dynamic time weights $w_t$ to address these issues. t This makes the evaluation results more in line with the actual situation, helps the system adjust the prediction model parameters in a timely manner, and further improves the accuracy and reliability of predictions.

[0268] Optionally, when the environment dependency management module downloads the dependency packages of the crawler project containing the crawler task to the slave node, it initializes the dependency package configuration submodule to parse the project configuration file stored on the object server to determine the dependency packages required for the crawler project containing the crawler task to run, and enables the slave node that has been assigned the crawler task to download the required dependency packages from the object server.

[0269] Optionally, the project configuration file may be, for example, package.json.

[0270] Preferably, the working principle of the above-mentioned environment dependency management module is as follows:

[0271] #1. Dependency Package Resolution and Requirements Determination

[0272] In a distributed web crawler system, the project configuration file (such as `package.json`) is crucial for determining the crawler project's dependencies. Let the project configuration file be a nested JSON object P, containing multiple dependencies. Each dependency d... i It can be represented as a tuple Where n i It is the name of the dependency package. This refers to the version range requirement of the dependency package.

[0273] Version range requirements It can be a complex expression, for example:

[0274]

[0275] in:

[0276] and These represent the lower and upper limits of the version range, respectively, and these values ​​can be obtained from the configuration file. For example, in `package.json`, it might be represented as `"version":">=1.0.0<2.0.0"`.

[0277] `compatible(v, OS, Arch)` is a compatibility function used to determine whether a given version `v` is compatible with the operating system (OS) and hardware architecture (Arch) of the current slave node. This function can be defined as:

[0278]

[0279] The specific judgment logic can be based on the metadata of the dependency package, such as the supported operating system and architecture, which will be clearly indicated in the description information of the dependency package.

[0280] Let D = {d1, d2, ..., d} k} is the collection of all dependencies parsed from the project configuration file P.

[0281] 2. Dependency package download decision

[0282] For each slave node assigned a crawler task j It is necessary to determine which dependency packages to download from the object server. To account for factors such as network conditions, storage capacity, and dependency package availability, the following mathematical model is introduced.

[0283] Let N j It is a slave node s j Network bandwidth (in Mbps) between the object server and the target server, S j It is a slave node s j Available storage capacity (in GB). For each dependency d i ,set up It is the collection of versions available on the object server, s i This is the size of the dependency packages (unit: GB).

[0284] Define a download priority function p ij Used to evaluate the performance of slave nodes. j Download dependencies d i Priority:

[0285] in:

[0286] α, β, and γ are weighting coefficients, and α + β + γ = 1. For example, α = 0.4, β = 0.3, and γ = 0.3 can be set to represent the importance of version matching, storage capacity, and network bandwidth in the download decision, respectively.

[0287] This is a version matching function used to calculate the version range. With available version set The degree of matching. This can be defined as:

[0288] Where |·| represents the cardinality of the set.

[0289] t i It is a dependency package d i The average download time (in seconds) can be obtained by statistical analysis of historical download data.

[0290] Based on download priority p ij All dependencies are sorted, and the dependencies with higher priority are downloaded first. At the same time, storage capacity limitations must be met, namely: Where download(i,j) is a boolean function indicating whether it is in slave node s j Download dependencies d i .

[0291] 3. Optimized dependency package download process

[0292] To improve download efficiency during the dependency package download process, a multi-threaded concurrent download approach is adopted. Let T... ij It is in slave node s j Download dependencies d i The actual download time, n ij This is the number of threads allocated to this download task.

[0293] Download time T ij It can be calculated using the following formula:

[0294] Where η(n) ij ) is the number of threads n ij The corresponding download efficiency function takes into account factors such as thread scheduling overhead and network congestion during multi-threaded downloads. This function can be obtained through experimental fitting, for example:

[0295] μ and v are fitting parameters, which are obtained by fitting download experimental data under different thread numbers.

[0296] In this application, in order to minimize the total download time T of all dependent packages total The following optimization problem can be established:

[0297] minT total =max i,j T ij st n ij ≤n max Where n max This is the maximum number of threads limit.

[0298] Based on the above technology, the implementation steps are as follows:

[0299] 1. Dependency Package Resolution: The dependency package configuration submodule parses the project configuration file P stored on the object server, extracts all dependencies D, and determines the version range requirements for each dependency.

[0300] 2. Node Information Collection: Collect information for each slave node. j Network bandwidth N j Available storage capacity S j Information such as...

[0301] 3. Download priority calculation: For each slave node s j Calculate d for each dependency i Download priority p ij And sort the dependencies.

[0302] 4. Download Task Allocation: Based on download priority and storage capacity limitations, determine the download task allocation for each slave node. j Download which dependencies and allocate n threads to each download task. ij .

[0303] 5. Concurrent Download: Start multi-threaded concurrent download tasks on each slave node, based on the calculated download time T. ij Monitor and optimize.

[0304] Preferably, in a specific application scenario, the above technical solution has the following technical advantages:

[0305] Traditional methods for determining dependencies by parsing project configuration files typically rely solely on simple version number ranges, with little consideration for operating system and hardware architecture compatibility. For example, they simply search for available versions within the version number range specified in the configuration file (e.g., `1.0.0 2.0.0`), without performing fine-grained compatibility checks. In contrast, this application defines version range requirements... In addition to considering the upper and lower limits of the version number, a compatibility function compatible(v,OS,Arch) is introduced to determine whether the version is compatible with the operating system and hardware architecture of the slave node.

[0306] Therefore, unlike traditional methods that often download incompatible dependency packages, leading to crawler task failures, this application uses compatibility checks to ensure that downloaded dependency packages are fully compatible with the node environment, thus improving the success rate of crawler tasks. For example, in a hybrid distributed system where different nodes use different operating systems and hardware architectures, traditional methods may result in dependency package incompatibility, while this solution avoids such problems, increasing the task success rate by 20%-30%. Furthermore, since the downloaded dependency packages are compatible, errors caused by dependency incompatibility are reduced, lowering the time and effort costs for developers and operations personnel to troubleshoot issues.

[0307] In this application, regarding dependency package download decisions,

[0308] Traditional dependency download decisions often only consider version matching, neglecting factors such as network conditions and storage capacity. They typically download dependencies sequentially according to the order in the configuration file, without prioritization. In contrast, this application introduces a download priority function p. ij The algorithm comprehensively considers factors such as version compatibility, storage capacity, and network bandwidth. It uses weighting coefficients α, β, and γ to adjust the importance of each factor, ranking dependencies and prioritizing the download of higher-priority dependency packages.

[0309] Therefore, under conditions of limited network bandwidth or insufficient storage capacity, traditional methods can lead to slow download processes or failures due to insufficient storage. This application dynamically adjusts download priorities based on network bandwidth and storage capacity, prioritizing the download of dependency packages with minimal impact on system resources and high matching accuracy, thus improving resource utilization. For example, on nodes with low network bandwidth, smaller dependency packages with high version matching accuracy are downloaded first, improving download efficiency by 30%-40%. Furthermore, through reasonable priority sorting, unnecessary download waiting and resource waste are avoided, shortening the overall download time of dependency packages. Especially in large-scale distributed crawler systems, this significantly improves system deployment and startup speed.

[0310] Regarding the optimization of dependency package downloading, traditional dependency package downloading typically employs a single-threaded approach, neglecting the impact of thread count on download efficiency and lacking optimization of the download process. This application adopts a multi-threaded concurrent downloading method and introduces a download efficiency function η(n ij We will consider the impact of the number of threads on download efficiency. We will formulate an optimization problem to minimize the total download time T of all dependent packages. totalTherefore, multi-threaded concurrent downloading can fully utilize network bandwidth and improve download speed. Simultaneously, by optimizing the allocation of the number of threads, the thread scheduling overhead and network congestion problems caused by too many threads are avoided. For example, in a high-bandwidth network environment, a reasonable allocation of the number of threads can improve download speed by 50%-80%. Furthermore, the download efficiency function η(n) ij The system can adaptively adjust the number of threads based on different network environments and the size of dependency packages, making the download process more stable and efficient. When network conditions change, the system can automatically adjust the number of threads to ensure that download efficiency is not significantly affected.

[0311] Optionally, when storing crawler data, the data storage module initializes the database configuration submodule to store the crawler data according to the storage management items for the crawler data and based on the crawler input parameter information.

[0312] Optionally, the crawler input parameters may include, for example, the account and password required to log in to the crawler.

[0313] Preferably, in a specific application scenario, the preferred or alternative implementation of the data storage module is as follows:

[0314] The data storage module plays a crucial role in the entire web crawler system, its main task being to effectively store and manage the data crawled by the crawler. To achieve this goal, we will use the database configuration submodule to store the data reasonably based on the storage management items and input parameters of the crawler data.

[0315] 2. Database configuration submodule initialization

[0316] Before storing crawler data, the database configuration submodule needs to be initialized. This process involves multiple steps to ensure that the database can correctly receive and manage the data.

[0317] 2.1 Database connection configuration

[0318] First, you need to determine the type of database you are using. Common types include relational databases (such as MySQL and PostgreSQL) and non-relational databases (such as MongoDB and Redis). Different database types have different connection methods and configuration parameters.

[0319] Here is a sample code example of using Python to connect to a MySQL database:

[0320] import mysql.connector

[0321] # Obtain database connection information from configuration files or environment variables

[0322] config={

[0323] 'user':'your_username',

[0324] 'password':'your_password',

[0325] 'host':'your_host',

[0326] 'database':'your_database',

[0327] 'raise_on_warnings':True

[0328] }

[0329] try:

[0330] # Establish database connection

[0331] cnx=mysql.connector.connect(config)

[0332] cursor = cnx.cursor()

[0333] print("Database connection successful")

[0334] except mysql.connector.Error as err:

[0335] print(f"Database connection failed: {err}")

[0336] ```

[0337] In the code above, the `config` dictionary contains the information required for database connection, such as username, password, host address, and database name. The database connection is established using the `mysql.connector.connect()` method, and a cursor object is created to execute SQL statements.

[0338] 2.2 Table Structure Creation

[0339] Based on the storage management items for the crawler data, a corresponding table structure needs to be created in the database. Storage management items can include information such as data field names, data types, and indexes.

[0340] Here is a sample code for creating a MySQL table:

[0341] Python

[0342] # Define table structure

[0343] create_table_query="""

[0344] CREATE TABLE IF NOT EXISTS crawler_data(

[0345] id INT AUTO_INCREMENT PRIMARY KEY,

[0346] url VARCHAR(255)NOT NULL,

[0347] title VARCHAR(255),

[0348] content TEXT,

[0349] timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP )

[0351] try:

[0352] #Execute the SQL statement to create the table

[0353] cursor.execute(create_table_query)

[0354] cnx.commit()

[0355] print("Table created successfully")

[0356] except mysql.connector.Error as err:

[0357] print(f"Table creation failed: {err}")

[0358] ```

[0359] In the code above, `create_table_query` defines a table named `crawler_data` containing fields such as `id`, `url`, `title`, `content`, and `timestamp`. The SQL statement is executed using the `cursor.execute()` method, and the transaction is committed using `cnx.commit()`.

[0360] 3. Storage strategy based on crawler input parameters

[0361] The crawler input parameters (such as login account and password) can be used to classify and store crawler data or add to it.

[0362] 3.1 Store by account category

[0363] The crawler data can be stored in different tables or collections based on the login account to facilitate data management and retrieval.

[0364] #Create table based on account

[0365] account = "your_account"

[0366] table_name=f"crawler_data_{account}"

[0367] create_table_query = f"""

[0368] CREATE TABLE IF NOT EXISTS{table_name}(

[0369] id INT AUTO_INCREMENT PRIMARY KEY,

[0370] url VARCHAR(255)NOT NULL,

[0371] title VARCHAR(255),

[0372] content TEXT,

[0373] timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP )

[0375] try:

[0376] cursor.execute(create_table_query)

[0377] cnx.commit()

[0378] print(f"Table {table_name} was created successfully")

[0379] except mysql.connector.Error as err:

[0380] print(f"Table {table_name} creation failed: {err}")

[0381] ```

[0382] In the code above, table names are dynamically generated based on the login account, and the corresponding tables are created.

[0383] 3.2 Adding Metadata

[0384] When storing crawler data, crawler input parameters can be added as metadata to facilitate subsequent data analysis and auditing.

[0385] #Simulated web crawler data

[0386] data = {

[0387] 'url':'https: / / example.com',

[0388] 'title':'Example Page',

[0389] 'content':'This is an example page.',

[0390] 'account':'your_account',

[0391] 'password':'your_password'

[0392] }

[0393] #SQL statement for inserting data

[0394] insert_query="""

[0395] INSERT INTO crawler_data(url,title,content,account,password)

[0396] VALUES(%s,%s,%s,%s,%s)

[0397] """

[0398] try:

[0399] #Execute the SQL statement to insert data

[0400] cursor.execute(insert_query,(data['url'],data['title'],data['content'],data['account'],data['password']))

[0401] cnx.commit()

[0402] print("Data insertion successful")

[0403] except mysql.connector.Error as err:

[0404] print(f"Data insertion failed: {err}")

[0405] ```

[0406] In the code above, the crawler input parameters (username and password) are inserted into the database as metadata.

[0407] 4. Data storage process

[0408] After determining the database configuration and storage strategy, you can begin storing the crawler data. Below is an example of a complete data storage process:

[0409] import mysql.connector

[0410] #Database connection configuration

[0411] config={

[0412] 'user':'your_username',

[0413] 'password':'your_password',

[0414] 'host':'your_host',

[0415] 'database':'your_database',

[0416] 'raise_on_warnings':True

[0417] }

[0418] try:

[0419] # Establish database connection

[0420] cnx=mysql.connector.connect(config)

[0421] cursor = cnx.cursor()

[0422] print("Database connection successful")

[0423] #Creating a table

[0424] create_table_query = """

[0425] CREATE TABLE IF NOT EXISTS crawler_data(

[0426] id INT AUTO_INCREMENT PRIMARY KEY,

[0427] url VARCHAR(255) NOT NULL,

[0428] title VARCHAR(255),

[0429] content TEXT,

[0430] account VARCHAR(255),

[0431] password VARCHAR(255),

[0432] timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP )

[0434] cursor.execute(create_table_query)

[0435] cnx.commit()

[0436] print("Table created successfully")

[0437] # Simulate crawler data

[0438] data = {

[0439] 'url': 'https: / / example.com',

[0440] 'title': 'Example Page',

[0441] 'content': 'This is an example page.',

[0442] 'account': 'your_account',

[0443] 'password': 'your_password'

[0444] }

[0445] #Insert data

[0446] insert_query="""

[0447] INSERT INTO crawler_data(url,title,content,account,password)

[0448] VALUES(%s,%s,%s,%s,%s)

[0449] """

[0450] cursor.execute(insert_query,(data['url'],data['title'],data['content'],data['account'],data['password']))

[0451] cnx.commit()

[0452] print("Data insertion successful")

[0453] except mysql.connector.Error as err:

[0454] print(f"An error occurred: {err}")

[0455] finally:

[0456] # Close database connection

[0457] ifcnx.is_connected():

[0458] cursor.close()

[0459] cnx.close()

[0460] print("Database connection closed")

[0461] ```

[0462] In the code above, a database connection is first established, then a table structure is created, then crawler data is simulated and inserted into the database, and finally the database connection is closed.

[0463] In the above solution, regarding the storage strategy based on input parameter information, by incorporating crawler input parameters (such as login account and password) into the storage strategy, data classification and metadata addition are achieved, improving data management and analysis efficiency. Regarding dynamic table structure creation, table structures are dynamically created based on different login accounts, making data storage more flexible and scalable. Regarding exception handling and resource management, exception handling and resource management mechanisms are added to the code to ensure the stability of database connections and data security.

[0464] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims. The systems, devices, modules, or units described in the above embodiments are specifically implemented by computer chips or entities, or by products with certain functions.

Claims

1. A distributed crawler system based on performance monitoring, characterized in that, include: An initialization configuration module is used to configure the initialization configuration items for the operation of the distributed crawler system. The master-slave communication control module is used to establish a communication connection between the master node and the slave node based on the initialization configuration items. The communication connection enables the master node to monitor the performance of the slave node and distribute the crawler task to the slave node whose performance meets the set performance index. The environment dependency management module is used to download the dependency packages of the crawler project where the crawler task is located to the slave node, so that the crawler task can run on the slave node; The data storage module is used to store the crawler data obtained when the crawler task runs on the slave node. The operation of the aforementioned environment dependency management module includes the following steps: Dependency package resolution: The dependency package configuration submodule stores the project configuration files on the object server. Perform parsing and extract all dependencies. And determine the version range requirements for each dependency. ; Node information collection: Collect information for each slave node. network bandwidth Available storage capacity information; Download priority calculation: for each slave node Calculate each dependency Download priority And sort the dependencies; Download task allocation: Based on download priority and storage capacity limits, the allocation is determined for each slave node. Which dependencies to download, and how many threads to allocate to each download task. ; Concurrent download: Start multi-threaded concurrent download tasks on each slave node, based on the calculated download time. Monitor and optimize; The dependency package download process incorporates the following mathematical model: For each slave node assigned a crawler task... Consider factors such as network conditions, storage capacity, and the availability of dependent packages: set up It is a slave node Network bandwidth between the object server and the target server It is a slave node Available storage capacity for each dependency ,set up It is a collection of versions available on the object server. It refers to the size of the dependency packages; Define a download priority function Used to evaluate on slave nodes Download dependencies Priority: ,in: , , It is a weighting coefficient, and , respectively, represent the importance of version matching degree, storage capacity and network bandwidth in download decisions; This is a version matching function used to calculate the version range. With available version set The degree of matching is defined as: ,in The cardinality of a set. It is a dependency package The average download time was obtained through historical download data statistics; Based on download priority Sort all dependencies and download the highest priority dependencies first, while also meeting storage capacity limits. ,in It is a boolean function that indicates whether it is on the slave node. Download dependencies .

2. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, The initialization configuration module includes: The node discovery configuration submodule is used to configure a "heartbeat mechanism" for slave nodes on the master node so that the master node can monitor the performance of the slave nodes.

3. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, The initialization configuration module includes: The object configuration submodule is used to store object configuration information in the object server, so that the slave node can perform self-detection based on the crawler task distributed by the master node to determine whether the corresponding crawler project exists locally. If it does not exist, the slave node downloads the crawler project and the corresponding project role permissions from the object server based on the object configuration information.

4. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, The initialization configuration module includes a dependency package configuration submodule, which stores the dependency packages of the crawler project to the object server, so that the slave nodes assigned the crawler task can download the required dependency packages from the object server.

5. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, The initialization configuration module includes a middleware configuration submodule, which stores performance loss data for executing crawler tasks, so that the master node can monitor the performance of the slave nodes, and when there are new or updated crawler tasks, the master node can allocate performance distribution of the new or updated crawler tasks to slave nodes whose performance meets the set performance indicators.

6. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, The initialization configuration module includes a database configuration submodule, which is used to set storage management items for crawler data and, based on the crawler input parameter information determined by parsing the crawler project, enables the data storage module to store the crawler data.

7. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, The distributed crawler system also includes: a display module, used to display the crawler data in a form; The initialization configuration module includes a display configuration submodule, which is used to set display management items for crawler data, so as to control the display module to display the crawler data according to the integrated data class in a form.

8. The distributed crawler system based on performance monitoring according to claim 2, characterized in that, When establishing the communication connection, the master-slave communication control module initializes the node discovery configuration submodule and opens the socket port on the master node to determine the online slave nodes based on the heartbeat mechanism and establish a communication connection with the online slave nodes.

9. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, When monitoring the performance of the slave node, the master-slave communication control module triggers the slave node to perform self-performance detection based on the communication connection, and uploads the detected performance data to the master node so that the master node can monitor the performance of the slave node.

10. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, When the master-slave communication control module distributes crawler tasks to slave nodes whose performance meets the set performance indicators, it initializes the object configuration module so that the slave nodes can perform self-detection based on the crawler tasks distributed by the master node to determine whether the corresponding crawler project exists locally.

11. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, When monitoring the performance of the slave node, the master-slave communication control module initializes the middleware configuration submodule to store the detected performance loss data and upload it to the master node so that the master node can monitor the performance of the slave node.

12. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, When the environment dependency management module downloads the dependency packages of the crawler project containing the crawler task to the slave node, it initializes the dependency package configuration submodule to parse the project configuration file stored on the object server to determine the dependency packages required for the crawler project containing the crawler task to run, and enables the slave node that has been assigned the crawler task to download the required dependency packages from the object server.

13. The distributed crawler system based on performance monitoring according to claim 1, characterized in that, When storing crawler data, the data storage module initializes the database configuration submodule to store the crawler data according to the storage management items and the crawler input parameters.

Images