Monitoring task self-adaptive arrangement issuing method and system

Abstract

This disclosure provides a method and system for adaptive orchestration and distribution of monitoring tasks. Within the update window, the central node's update module polls the domain name validity records uploaded by the monitoring nodes, parses the detection timestamp and status, deletes expired records, and filters out valid target domain names to store in the task master table. Subsequently, based on the last execution timestamp information in the task master table, the domain name to be detected is extracted, and a task snapshot is generated in conjunction with the current routing rules. After all snapshots are generated, they are batch-updated into the database, and the task digital fingerprint is recalculated. If the task digital fingerprint changes, all snapshot change channels are immediately frozen to prevent data inconsistency during the update period. Within the push window, the push module determines the page size according to the task type and pulls snapshot tasks page by page. Based on preset priorities, tasks are pushed to the message middleware at a dynamic rate for monitoring nodes to receive and execute.

Description

Technical Field

[0001] This disclosure relates to the field of monitoring, and in particular to a method and system for adaptive orchestration and distribution of monitoring tasks. Background Technology

[0002] With the increasing prevalence of cloud security and CDN services, monitoring systems need to simultaneously perform various types of probe tasks (HTTP / HTTPS availability, DNS resolution, IPv6 connectivity, WAF protection effectiveness, etc.) on tens of thousands of domains. These tasks need to be executed periodically, and the task parameters (target IP, probe node, port, path) change dynamically with business configuration.

[0003] Existing monitoring task generation solutions typically adopt a "full pull-direct distribution" model, which can only solve the problem of who will monitor, but does not achieve adaptive adjustment of monitoring tasks, nor does it balance monitoring efficiency and monitoring cost. Summary of the Invention

[0004] This disclosure provides at least one method and system for adaptive orchestration and distribution of monitoring tasks to solve at least one of the above-mentioned technical problems.

[0005] According to one aspect of this disclosure, an adaptive orchestration and distribution method for monitoring tasks is provided, applied to a central node, the central node including an update module and a push module, comprising: Within the update time window of the task cycle, the update module performs the following operations: Poll the domain validity record information uploaded to the storage middleware by each monitoring node, and delete expired records in the domain validity record information and filter out the target valid domains based on the detection timestamp and validity status obtained by parsing the domain validity record information, and store the target valid domains in the task master table; When extracting the target valid domain name to be detected for the first time, extraction starts from the first target valid domain name in the task master table; when extracting the target valid domain name to be detected for subsequent timestamps, the target valid domain name to be detected is extracted from the task master table based on the timestamp information of the last task execution; and a task snapshot is generated for the target valid domain name to be detected based on the current valid routing rules; wherein, the timestamp information of the last task execution is used to record the target valid domain name processed in the previous task; the task snapshot includes deletion transactions and insertion transactions for the changed task, and when generating the task snapshot of the changed task, the deletion transaction is executed first, and then the insertion transaction is executed; After all task snapshots are generated, all task snapshots are updated into the database, and the task digital fingerprints are updated. If the task digital fingerprints change, the timestamp information of the last task execution is updated, and all change channels for the task snapshots are frozen until the push time window of the current task cycle ends. If the task digital fingerprints do not change, the old version of the task data is visible to the push module. The task data includes task snapshots. The task digital fingerprint is a hash value generated based on the task configuration information in the task snapshot, used to identify the data version of the task snapshot. Within the push time window of the task cycle, the push module performs the following operations: The page size is determined based on the task type, and tasks in the task snapshot are retrieved page by page according to the page size; Based on the pre-set task priority, tasks are pushed to the message middleware according to the dynamic task push rate, so that the monitoring nodes can retrieve tasks from the message middleware.

[0006] In one possible implementation, the domain name validity record information is stored in the storage middleware in the form of a hash table, and different monitoring nodes in the storage middleware correspond to different hash tables; the hash table records the validity information and probe timestamp of each target domain name; the validity information and probe timestamp are uploaded by the corresponding monitoring node; the key of the hash table is a preset fixed prefix of the monitoring node dimension + the monitoring node name.

[0007] In one possible implementation, the target valid domain name is a valid domain name that has not expired and is stored in the storage middleware by at least one monitoring node.

[0008] In one possible implementation, the routing rules include host-level specified routing rules, node-level mandatory routing rules, and node-level ignored routing rules; The task snapshot also includes deleting transactions and inserting transactions.

[0009] In one possible implementation, freezing all change channels of the task snapshot in the event of a change in the task digital fingerprint includes: If the task's digital fingerprint changes, and before the push module completes the current push task, all change channels for the task snapshot are frozen.

[0010] In one possible implementation, the task priority is randomly generated during the generation of the task snapshot and stored in the task snapshot; or... The task priority is a random number generated by the application layer.

[0011] In one possible implementation, the dynamic task push rate is determined according to the following steps: The push interval is determined based on the length of the push time window and the number of tasks that need to be pushed in the task snapshot.

[0012] In one possible implementation, the storage middleware includes a shared cache or database table; the message middleware includes a message queue; the push time window and the update time window do not overlap, and the push time window is later than the update time window.

[0013] According to another aspect of this disclosure, a monitoring task adaptive orchestration and distribution system is provided, comprising: The monitoring node in the adaptive orchestration and distribution method for monitoring tasks described above; the monitoring node periodically queries the validity of the corresponding target domain name, generates and uploads the validity information and detection timestamp of the target domain name; The central node in any of the above-mentioned adaptive orchestration and distribution methods for monitoring tasks.

[0014] The adaptive orchestration and delivery method and system for monitoring tasks disclosed herein focuses on dividing each task execution cycle into mutually exclusive "update windows" and "push windows" according to a time ratio, thereby decoupling task generation and delivery. Specifically, within the update time window of the task cycle, the update module performs the following operations: polling the domain name validity record information uploaded by each monitoring node to the storage middleware, and based on the probe timestamp and validity status obtained by parsing the domain name validity record information, deleting expired records in the domain name validity record information and filtering out target valid domain names, storing the target valid domain names in the task master table; and based on the last task execution timestamp information in the task master table, retrieving the target valid domain names from the updated time window. The system extracts the target valid domain name to be detected from the task master table and generates a task snapshot for the target valid domain name to be detected based on the current valid routing rules. After all task snapshots are generated, all task snapshots are updated into the database, and the task digital fingerprint is updated. If the task digital fingerprint changes, all change channels of the task snapshot are frozen. Within the push time window of the task cycle, the push module performs the following operations: determines the page size according to the task type, and pulls the tasks in the task snapshot page by page according to the page size; according to the pre-set task priority, pushes the tasks to the message middleware according to the dynamic task push rate, so that the monitoring node can retrieve the tasks from the message middleware.

[0015] The above-mentioned technical solution disclosed herein automatically filters invalid tasks and dynamically determines whether to generate probe tasks based on the access status independently reported by each monitoring node, thereby reducing invalid probes, lowering costs, and improving efficiency. This disclosure can respond to topology changes in real time and can block faulty nodes and switch traffic paths through dynamic routing rule hot loading without restarting or modifying the task table. This disclosure adopts a snapshot isolation update mechanism and ensures that data is not modified during the push period through time window mutual exclusion. At the same time, this disclosure can achieve smooth and secure delivery. This disclosure combines pre-calculated random sorting and uniform rate push to avoid impacting the monitored sites. In addition, this disclosure achieves high-performance big data processing. This disclosure moves the random sorting calculation pressure forward to the write period, supporting efficient paging reads of tens of millions of tasks.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of a monitoring task adaptive orchestration and distribution method in one embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a monitoring task adaptive orchestration and distribution system in one embodiment of this disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] Existing monitoring task generation solutions typically adopt a "full fetch - direct distribution" model, which has the following prominent problems: Invalid probes are rampant, failing to determine whether a domain name is still connected to the system, and continuously sending requests to expired domain names, wasting bandwidth and computing resources. Node failure response is slow; when a monitoring or target node fails / is cut over, manual modification of the task table or service restart is required, making it impossible to avoid failed nodes in real time. Push traffic surges: a large number of tasks are suddenly released at the same time, easily causing the monitored site to experience instantaneous overload, or even being misjudged as a DDoS attack. Data inconsistency: if the database changes during task generation, the pushed tasks may contain a mix of old and new data, resulting in incorrect monitoring results. Large-scale data sorting bottlenecks: when dynamically and randomly sorting tens of millions of tasks, the database temporary table becomes too large, leading to query timeouts or even OutOfMemory errors. Some solutions attempt to address single points of failure, but fail to form a collaborative closed loop. Related patents focus on online device detection, and no solutions are found that use access probe results independently reported by each monitoring node as a pre-filter condition for task generation. There are many patents on network layer routing strategies, but they are not applied to monitoring task orchestration. Patents exist for randomizing and uniformly pushing scheduled tasks, but they are not organically combined with snapshot isolation and data consistency guarantees. The data is randomly sorted, and a dynamic ORDER BY RAND() scheme exists, but it has serious performance and paging consistency issues when dealing with tens of millions of data points. No scheme combining random number pre-computation with task snapshot freezing has been found.

[0020] To address the aforementioned issues, this disclosure proposes an adaptive orchestration and distribution method and system for monitoring tasks, which can be applied to scenarios such as CDN service quality monitoring, website availability detection, security protection effectiveness verification, and cloud WAF health checks, and can solve at least one of the above technical problems.

[0021] Before explaining the solution, the following technical terms will be explained.

[0022] Access Status: This refers to whether the domain name is still pointing to this monitoring system via NS (Nameserver) or CNAME. Each monitoring node independently detects and reports this information to Redis to determine whether monitoring and detection of the domain name still needs to be performed.

[0023] Routing rules include host-level specified routes, node-level mandatory routes, and node-level ignored routes, which are used to dynamically adjust the source probe nodes and target nodes during the task generation phase.

[0024] Snapshot Isolation: refers to the coexistence of old and new versions of data during task updates, with external queries always returning the old version until the new version is fully built before atomically switching.

[0025] Time Window Mutex: This refers to dividing the time into mutually exclusive "update window" and "push window" according to a preset ratio (15% / 85%) within each task execution cycle. Only task generation is performed in the update window, and only task distribution is performed in the push window while data changes are frozen.

[0026] Pre-computed Random Sorting: This refers to generating random weights for each record using the SQL rand() function and storing them persistently when the task is generated and written to the database; during reading, the records are sorted according to these weight fields, thus shifting the sorting computation burden from the reading period to the writing period.

[0027] Lazy deletion refers to not actively maintaining independent expiration cleanup tasks, but instead having the requester immediately perform the deletion operation when expired data is accessed, thus distributing the cleanup overhead to normal business processes.

[0028] The technical solution of this disclosure will be described below through specific embodiments.

[0029] like Figure 1 The diagram shows a flowchart of the adaptive orchestration and distribution method for monitoring tasks in this embodiment. The execution entity in this embodiment is the central node, which includes an update module and a push module. Specifically, the method in this embodiment may include the following steps: S110. Within the update time window of the task cycle, the update module performs the task update operation: The update time window mentioned above can be the first 15% of the task cycle.

[0030] (1) Poll the domain name validity record information uploaded to the storage middleware by each monitoring node, and delete the expired records in the domain name validity record information and filter out the target valid domain name based on the detection timestamp and validity status obtained by parsing the domain name validity record information, and store the target valid domain name in the task master table.

[0031] The domain name validity record information is stored in the storage middleware in the form of a hash table, and different monitoring nodes in the storage middleware correspond to different hash tables; the hash table records the validity information and probe timestamp of each target domain name; the validity information and probe timestamp are uploaded by the corresponding monitoring nodes; the key of the hash table is a preset fixed prefix of the monitoring node dimension + the monitoring node name.

[0032] The monitoring nodes periodically poll the status of the target domain name and upload the data to the monitoring nodes.

[0033] Each distributed monitoring node (Checker Node, such as shjt2, zzuc, yzm, etc.) in the system continuously probes the access validity of the target domains under its jurisdiction (i.e., whether the domain still points to the system's protected IP via NS or CNAME) through its own independent scheduled tasks. Each node stores its probe results independently in the form of a hash table in a storage middleware, such as Redis. The key is DnsAccessKey (a preset fixed prefix for the monitoring node dimension) + the monitoring node name (e.g., hash:nws:analysis:dns_access:shjt2), the field in the hash table is the target domain identifier (zone for NS access type, domain for CNAME access type), and the value is "validity status | probe timestamp" (e.g., "1|2026-07-2015:04:05"). The status "1" indicates that the monitoring node believes the domain is still correctly connected to the system. Each monitoring node reports independently without interfering with each other, and the central node does not need to actively probe.

[0034] At the start of each task cycle, the system iterates through all surviving monitoring nodes (CheckerNode list) and performs the following operations on each node: reads the reporting record corresponding to the target domain name (dest) from the DnsAccessKey + node name hash table of the node; parses out the validity status (arr[0]) and probe timestamp (arr[1]); then performs lazy deletion of expired records. Specifically, it calculates the difference between the current time and the probe timestamp. If it exceeds a preset threshold (e.g., 2 days), it determines that the record reported by the node has expired, executes del to delete the record from the hash table of the node, and skips the node to continue checking the next one. This process distributes the expiration cleanup overhead to normal queries, eliminating the need for a separate cleanup task.

[0035] This step determines access based on a single valid node during the screening of target valid domain names. That is, if a monitoring node has a record that has not expired and its status is "1" (valid access), it immediately returns true and does not check the remaining nodes. Only when all monitoring nodes have no valid access records (i.e., the record has either expired and been deleted, or its status is "0") is the target domain name determined to be invalid, and the generation of all subsequent tasks for that domain name (including various monitoring such as HTTP, DNS, and IPv6) is skipped.

[0036] This step can produce the following beneficial effects: Lightweight decentralized judgment: the central node does not need to actively probe, but only consumes the "ready-made data" reported by each monitoring node, which greatly reduces the load on the central node; lazy deletion: expired records are naturally cleaned up during the query process, without the need to maintain a separate cleanup task, with zero additional overhead; High fault tolerance: It adopts "OR" logic, which means that as long as any monitoring node can still confirm that the domain name is valid, it is considered valid, thus avoiding misjudgment caused by network jitter of individual nodes; Real-time performance: After the domain name is withdrawn, all the reporting records of all nodes will expire and be deleted within the latest TTL (2 days), and the system will automatically stop monitoring the domain name without manual intervention.

[0037] (2) When extracting the target valid domain name to be detected for the first time, the extraction starts from the first target valid domain name in the task master table; when extracting the target valid domain name to be detected for the second time, the target valid domain name to be detected is extracted from the task master table based on the timestamp information of the last task execution; and a task snapshot is generated for the target valid domain name to be detected based on the current valid routing rules; wherein, the timestamp information of the last task execution is used to record the target valid domain name processed in the last task; the task snapshot includes deletion transactions and insertion transactions for the changed task. When generating the task snapshot of the changed task, the deletion transaction is executed first, and then the insertion transaction is executed.

[0038] Task snapshots also include separate delete and insert transactions.

[0039] Before task generation, the currently effective policies are loaded from the database routing rule table. Each routing rule can be configured with a validity period (close_time), which automatically expires without manual intervention. Specific rules can include the following: Host-level specified route (router_force_host): forces traffic for a specific domain name to migrate from a specified source node to a target node; Node-level mandatory route (router_force_node): automatically replaces all traffic passing through a certain node with another node; Node-level ignore route (router_ignore_node): directly blocks faulty nodes, preventing the generation of any tasks originating from or destined for that node. During task generation, the target and source nodes involved are dynamically replaced or filtered according to the above rules.

[0040] Automatic fault recovery with the same operator: When a specified node failure is ignored, the system automatically selects a replacement node randomly from the node pool of the same operator (China Telecom / China Unicom / China Mobile) to ensure high availability. This logic is implemented through the getChecker function: if the passed filterNode is not empty, the node is filtered out from the pool of the same ISP and a replacement is selected. When the pool has more than 3 nodes, 3 nodes are randomly selected, which ensures high availability and prevents single point overload.

[0041] When a node fails or is cutover, the above routing configuration technology allows operations and maintenance personnel to simply configure routing rules, which will take effect in the next task cycle (minutes) without requiring modification of a large number of task records.

[0042] After all task snapshots are generated, all task snapshots are updated into the database, and the task digital fingerprints are updated. If the task digital fingerprints change, the timestamp information of the last task execution is updated, and all change channels for the task snapshots are frozen until the push time window of the current task cycle ends. If the task digital fingerprints do not change, the old version of the task data is visible to the push module. The task data includes task snapshots. The task digital fingerprint is a hash value generated based on the task configuration information in the task snapshot, used to identify the data version of the task snapshot.

[0043] During step S110, the push module does not push anything to the outside world.

[0044] S120. Within the push time window of the task cycle, the push module performs a task push operation: The push time window and the update time window have no overlap, and the push time window is later than the update time window. Specifically, the push time window can be the last 85% of the task cycle.

[0045] (1) Determine the page size based on the task type, and retrieve the tasks in the task snapshot page by page according to the page size.

[0046] This is a pagination strategy based on data field characteristics, namely: dynamically selecting the database pagination size according to the task type (HTTP or DNS): HTTP tasks, which carry large fields such as collect_header, raw, and cookies, use small pagination (e.g., 1000 records / time); DNS tasks, which have lightweight fields, use large pagination (e.g., 10000 records / time).

[0047] Then, intelligent URI reconstruction and Host normalization are performed, automatically adapting to multiple protocol scenarios when generating probe URLs: IPv6 addresses are automatically appended with square brackets [] (e.g., http: / / [2001:db8::1] / path); standard ports (80 / 443) omit the port number and automatically switch the scheme (http / https); non-standard ports are explicitly appended to the URL and automatically injected into the Host request header (Host: domain:port) to ensure that backend services (such as Nginx virtual hosts) can be correctly resolved.

[0048] This step balances database I / O and memory GC pressure, improves the efficiency of large-scale task extraction, and ensures the correctness of multi-protocol tasks.

[0049] (2) Based on the pre-set task priority, push the task to the message middleware according to the dynamic task push rate so that the monitoring node can receive the task from the message middleware.

[0050] The message middleware includes a message queue. Task priorities are randomly generated during the creation of a task snapshot and stored in the task snapshot; alternatively, the task priority is a random number generated by the application layer.

[0051] Specifically, the task priority generation is as follows: By shifting the computational burden of "random sorting" entirely forward to the data writing phase, the performance bottleneck of dynamic random sorting of tens of millions of data points is completely resolved. Random weights are solidified (during the writing phase). In step S11, when batch inserting tasks into the database, the `rand` field of each record is synchronously written with a random floating-point number generated by the `rand()` function, making the random attribute an inherent component of the task snapshot. During high-performance paginated reading (during the reading phase), i.e., after the push window starts, the `publish` module executes `SELECT ... ORDER BY rand` (for the preset `rand` column) to sort the tasks while paginating them. This solution optimizes the sorting computation complexity from dynamic full-table sorting to index-based sorting using static fields.

[0052] Because the rand value is frozen along with the snapshot, the data boundaries of each pagination query (such as LIMIT 0,1000 and LIMIT 1000,2000) are clear and do not change with external conditions, completely avoiding the problem of repeated or missed data readings caused by unstable sorting results when using traditional dynamic ORDER BY RAND() with pagination.

[0053] This technology can improve performance, reducing query time by more than 90%; at the same time, it can provide absolute protection against attacks, that is, tasks with the same target domain are completely scattered due to different random values ​​and evenly distributed across different pages of the push queue, avoiding sudden attacks on the same origin site; in addition, it implements enhanced snapshot isolation, with the random value being part of the task snapshot, further consolidating the reliability of snapshot isolation.

[0054] The dynamic task push rate is determined according to the following steps: the push interval is determined based on the length of the push time window and the number of tasks that need to be pushed in the task snapshot.

[0055] The push process is strictly executed after the update is complete and is controlled by the time window in the publish module: Start point control (15% wait): At the beginning of the cycle, the publish module actively waits for 15% of the total cycle duration based on the Next time parsed by cron.

[0056] To prevent task delivery from blocking, dynamic rate calculation is performed: based on the remaining time of the current cycle (85%) and the total number of tasks to be pushed, the push interval is dynamically calculated to push tasks into the message middleware at a uniform rate (non-burst). The system monitors the push progress in real time. If the push is delayed due to network jitter, it will automatically compress the push interval of subsequent tasks to ensure that all tasks are completed before the end of the cycle.

[0057] By using hard isolation at 15% and 85% of the time, data integrity is ensured while converting dense, pulsed traffic into smooth white noise, reducing the impact on the monitored sites by more than 80%.

[0058] In some embodiments, steps (2) and (3) in S110 implement snapshot atomic updates under time window isolation. This step is executed strictly in chronological order. When the push module (publish module) starts each task cycle, it parses the Next execution time through cron and actively waits for 15% of the total cycle duration to ensure that the update module (update module) has sufficient time to complete snapshot construction, thereby forming a hard isolation between the generation window and the push window. Specifically, it may include the following steps: First, incremental boundary cutting based on lastmodify. Each task class (Custom / Website / IP, etc.) independently maintains a sync.Map to store the timestamp of the last execution (last, the timestamp of the last task execution). Initialization is completed at program startup by querying the current maximum lastmodify value, implementing a fetch strategy of full fetch initially and incremental fetch thereafter. Initialization: Query the current maximum lastmodify value. Incremental queries: strictly limited to lastmodify > last `lastmodify` is a database field in the task master table that records the timestamp (accurate to the second) of the last time the task configuration was modified. When the monitoring parameters of a task change (such as a change in monitoring nodes, port adjustments, or routing rule updates), the system calculates the MD5 fingerprint of the task's associated data using the `UpdateHash` method. If the fingerprint changes, `lastmodify` is automatically updated to the current time. This serves as a boundary marker for incremental fetching, avoiding a full scan of the tens of millions of task tables each time.

[0059] The initial full and subsequent incremental strategies in this disclosure are detailed below: Table 1

[0060] Second, atomic replacement with deletion followed by insertion. In a database transaction, for each change task, the old record is deleted first, and then the new record is inserted.

[0061] Execute various task generation logics and construct INSERT statements, i.e., insert transactions. Third, delayed marker activation Only after all DELETE and INSERT operations are completed will an UPDATE statement be executed to update the data path and update the MD5 fingerprint and lastmodify status of the main task. Before this, the old version task data is always visible to the outside (publish module).

[0062] The UpdateHash method aggregates all data currently associated with the task (node ​​VIP, access status, routing rules, monitoring port string), performs MD5 weighted calculation, and only updates lastmodify when the hash value changes.

[0063] Fourth, activation is equivalent to freezing. Once the status flag is updated (new snapshot activated), the system immediately freezes all change channels for that snapshot data. No update program will modify this data until the current push task is complete. In the code implementation, the `goHttpTask` and `goDnsTask` functions of the `publish` module use `status=1` and `rate=hz` as filtering conditions when reading data, ensuring that this data is not concurrently modified during the push process.

[0064] This solution completely eliminates the risk of "task data being tampered with during the push process" and achieves lock-free, rollback-enabled distributed task snapshot updates.

[0065] The technical solution disclosed herein implements a service access validity filtering method based on multi-node independent reporting and polling lazy deletion. Each monitoring node independently writes the access status and probe timestamp of the target domain name to Redis through its own scheduled task. Before task generation, the central node polls the reporting records of all nodes, performs immediate lazy deletion on expired records, and uses OR aggregation logic of "any valid node is considered access" to dynamically filter expired domain names, realizing automatic start and stop of monitoring tasks. The technical solution disclosed herein is a real-time task orchestration method driven by routing rules, supporting three-layer routing strategies: host-level specification, mandatory node routing, and node ignore routing. Each rule can be configured to expire automatically. Combined with automatic backup disaster recovery within the same operator, it realizes minute-level adjustment of task topology. The technical solution disclosed herein is a generation / push mutual exclusion scheduling method based on time ratio slicing. Within each task cycle, "update window" and "push window" are divided according to preset ratios (15% and 85%). Pushing is strictly prohibited within the update window, and data changes are frozen within the push window. Absolute isolation in the time dimension ensures the consistency of large-scale task distribution. The technical solutions disclosed herein are based on a snapshot-isolated atomic update mechanism for tasks. This disclosure includes an initial full update strategy followed by incremental updates, atomic replacement with deletion followed by insertion, delayed activation marking, activation-based freeze of changes, and MD5 fingerprint incremental judgment, achieving lock-free and rollback-capable distributed task snapshot updates. The technical solutions disclosed herein are based on a static random sorting method for tasks using "pre-computation during generation." During the task data writing phase, a random weight is assigned to each record using the rand() function and persistently stored. During the reading phase, the records are sorted according to this weight. This front-loads computational pressure, solving the performance bottleneck of dynamic random sorting of massive amounts of data, while ensuring data consistency during pagination and the secure randomness of task distribution. The technical solutions disclosed herein are a differentiated pagination and protocol standardization extraction method for heterogeneous monitoring tasks. This involves dynamically adjusting pagination based on task field sizes and automatically adapting to IPv6 / non-standard port URLs and Host header standardization. The technical solutions disclosed herein are a dynamic uniform-speed push method for task snapshots. This disclosure dynamically calculates the push rate based on the total number of snapshot tasks and the remaining push time window, evenly distributing sorted tasks to the message middleware.

[0066] In this disclosure, there is no direct communication between the central node and the monitoring node. Neither party is aware of the other's instance count or deployment details. Instead, they interact asynchronously with each other through message queues (Kafka) and shared caches (Redis).

[0067] The monitoring node (Checker Node) is a distributed probe execution end responsible for performing specific HTTP / DNS probe tasks and reporting the probe results. The monitoring node completes the reporting by writing data to Redis, without needing to know when or by whom the central node reads it. The central node pulls data from Redis as needed. The two are decoupled through Redis.

[0068] The central node is the logical control end, responsible for generating and distributing monitoring tasks. The central node itself consists of two sub-modules with different deployment modes: the Publish module simply submits the task to the small message queue Kafka, without needing to know which specific monitoring node will execute it. Each monitoring node competes to consume tasks from Kafka through a consumer group mechanism. The two sides are decoupled through Kafka. The functions of the two modules in the central node are as follows: Table 2

[0069] The data flow can be summarized as follows: Table 3

[0070] In this disclosure, the central node and the monitoring node never establish any direct connection; all interactions are completed through middleware: the monitoring node does not configure the central node's IP or service address; the central node does not maintain the monitoring node's connection pool or heartbeat; the monitoring node is unaware of how many instances the central node has, which is the leader, or whether a switch has occurred; the central node only needs to know the list of monitoring node names (used to construct Redis keys for round-robin reading); the monitoring node only interacts with Redis and Kafka and does not directly connect to any central node instance.

[0071] The monitoring node doesn't care who the message producer is. The state it writes to Redis may be read by any central instance, and the Kafka messages it consumes may be produced by any Publish instance, but the monitoring node is completely unaware of this.

[0072] Even if multiple Publish instances are deployed on the central node, the monitoring node only sees the messages in Kafka, not multiple central nodes. The central node includes a task generation module (Update) and a task distribution module (Publish); the task generation module runs in single-instance mode; the task distribution module runs in multi-instance parallel mode.

[0073] This invention decouples the traditional "tight coupling between the control end and the execution end" into "loose coupling" through middleware, which greatly improves the scalability and fault tolerance of the system.

[0074] The technical solutions disclosed herein also have the following alternatives: (1) Access status reporting can use database tables instead of Redis hash storage, but Redis's read / write performance and TTL mechanism are more suitable for high-frequency reporting scenarios. (2) Access status awareness can use a distributed configuration center (such as Etcd) instead of Redis cache, but this will increase deployment complexity. (3) Routing rules can be integrated into the service mesh (Istio), but an additional adaptation layer needs to be developed. (4) Snapshot isolation can be implemented using database transactions (such as repeatable read isolation level), but this may increase lock contention under high concurrency. (5) Random weight pre-computation can use application layer to generate random numbers instead of the database rand() function, but this will increase the computational burden on the application layer. (6) Uniform push can be replaced by the delayed delivery function of message queues, but it is necessary to ensure that the clocks of each node are synchronized.

[0075] Based on the same inventive concept, this disclosure provides a monitoring task adaptive orchestration and distribution system. The steps performed by the components of this system are the same as or similar to the methods described above, therefore, similar details will not be repeated. Figure 2 As shown, the adaptive orchestration and distribution system for monitoring tasks in this embodiment includes: In any of the above embodiments, monitoring node 210; the monitoring node 210 periodically queries the validity of the corresponding target domain name, generates and uploads the validity information of the target domain name and the detection timestamp; in any of the above embodiments, central node 220.

[0076] The various embodiments of the techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0080] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0081] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0082] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for adaptive orchestration and distribution of monitoring tasks, applied to a central node, wherein the central node includes an update module and a push module, characterized in that, include: Within the update time window of the task cycle, the update module performs the following operations: Poll the domain validity record information uploaded to the storage middleware by each monitoring node, and delete expired records in the domain validity record information and filter out the target valid domains based on the detection timestamp and validity status obtained by parsing the domain validity record information, and store the target valid domains in the task master table; When extracting the target valid domain name to be detected for the first time, extraction starts from the first target valid domain name in the task master table; when extracting the target valid domain name to be detected for subsequent timestamps, the target valid domain name to be detected is extracted from the task master table based on the timestamp information of the last task execution; and a task snapshot is generated for the target valid domain name to be detected based on the current valid routing rules; wherein, the timestamp information of the last task execution is used to record the target valid domain name processed in the previous task; the task snapshot includes deletion transactions and insertion transactions for the changed task, and when generating the task snapshot of the changed task, the deletion transaction is executed first, and then the insertion transaction is executed; After all task snapshots are generated, all task snapshots are updated into the database, and the task digital fingerprints are updated. If the task digital fingerprints change, the timestamp information of the last task execution is updated, and all change channels for the task snapshots are frozen until the push time window of the current task cycle ends. If the task digital fingerprints do not change, the old version of the task data is visible to the push module. The task data includes task snapshots. The task digital fingerprint is a hash value generated based on the task configuration information in the task snapshot, used to identify the data version of the task snapshot. Within the push time window of the task cycle, the push module performs the following operations: The page size is determined based on the task type, and tasks in the task snapshot are retrieved page by page according to the page size; Based on the pre-set task priority, tasks are pushed to the message middleware according to the dynamic task push rate, so that the monitoring nodes can retrieve tasks from the message middleware.

2. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The domain name validity record information is stored in the storage middleware in the form of a hash table, and different monitoring nodes in the storage middleware correspond to different hash tables; the hash table records the validity information and detection timestamp of each target domain name; The validity information and detection timestamp are uploaded by the corresponding monitoring nodes; the key of the hash table is a preset fixed prefix of the monitoring node dimension plus the monitoring node name.

3. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The target valid domain name is a valid domain name that has not expired and is stored in the storage middleware by at least one monitoring node.

4. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The routing rules include host-level specified routing rules, node-level mandatory routing rules, and node-level ignored routing rules. The task snapshot also includes deleting transactions and inserting transactions.

5. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The step of freezing all change channels of the task snapshot when the task's digital fingerprint changes includes: If the task's digital fingerprint changes, and before the push module completes the current push task, all change channels for the task snapshot are frozen.

6. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The task priority is randomly generated during the process of generating the task snapshot and is stored in the task snapshot; or, The task priority is a random number generated by the application layer.

7. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The dynamic task push rate is determined according to the following steps: The push interval is determined based on the length of the push time window and the number of tasks that need to be pushed in the task snapshot.

8. The adaptive orchestration and distribution method for monitoring tasks according to claim 1, characterized in that, The storage middleware includes a shared cache or a database table; the message middleware includes a message queue; the push time window and the update time window have no overlap, and the push time window is later than the update time window.

9. A monitoring task adaptive orchestration and distribution system, characterized in that, include: The monitoring node in the adaptive orchestration and distribution method for monitoring tasks according to any one of claims 1 to 8; The monitoring node periodically queries the validity of the corresponding target domain name, generates and uploads the validity information and detection timestamp of the target domain name; The central node in the adaptive orchestration and distribution method for monitoring tasks according to any one of claims 1 to 8.

Images