A distributed security dispatching and self-consistent recovery method and system

Abstract

The application discloses a distributed security scheduling and self-consistent recovery method and system, and belongs to the technical field of distributed task scheduling. The technical problem to be solved is that the scheduling, authentication and disaster recovery logic of the existing pull-type distributed scheduling system are fragmented, and problems such as the inability to adapt to heterogeneous computing power, the easy forgery and replay of data, the loss of network interruption results and the easy triggering of concurrent avalanches in large-scale reconnection exist. The technical solution points are that a closed-loop signaling state machine is formed through four gRPC interfaces of a Master node; a dynamic session seed and a random number updated round by round are issued in the authentication stage, multi-dimensional load is periodically reported and a dynamic drift signature is generated, the task granularity is adaptively divided based on the node health degree, result tamper-proofing verification is realized through task fingerprints and joint hashing, the results are persisted in the form of local pre-write logs when the network is interrupted, an exponential backoff plus random jitter algorithm is adopted when the network is reconnected, and the backlog data is incrementally returned based on the state cursor.

Description

Technical Field

[0001] This invention belongs to the field of distributed task scheduling technology, specifically relating to a distributed secure scheduling and self-consistent recovery method and system. Background Technology

[0002] For scenarios requiring large-scale node collaboration, such as massive asset exploration and edge computing, the Master-Agent architecture is widely used due to its clear structure and ease of expansion. This type of architecture generally adopts a pull-based communication model, where Agent nodes actively request tasks from the Master node. Upon receiving the request, the Master node distributes the pre-assigned tasks to the Agent nodes, and the Agent nodes actively send back the results after execution.

[0003] In existing pull-based distributed scheduling systems, the Master node typically splits tasks into tasks at a preset fixed granularity, placing them in a queue or directly binding them to corresponding Agent nodes. Agent nodes retrieve tasks through periodic pulling or long polling. The Master node is unaware of the real-time load status of Agent nodes, allocating tasks solely based on simple idle or busy flags. Regarding security authentication, Agent nodes register with the Master node upon initial startup, obtaining a static access token with a long validity period. All subsequent requests carry this static access token until it expires or is manually refreshed. There is no cryptographic link between task issuance commands and result feedback data; the Master node only verifies the validity of the static access token. In terms of disaster recovery, when network anomalies cause the connection between Agent nodes and the Master node to be interrupted, currently executing tasks are typically discarded. After network recovery, Agent nodes reconnect and request new tasks, but the execution results obtained after consuming computational resources are not retained. In scenarios where a large number of nodes recover simultaneously after a network outage, all Agent nodes initiate reconnection requests almost simultaneously, lacking an effective concurrency control mechanism.

[0004] The above implementation can basically meet business needs in scenarios with stable networks, balanced node computing power, and a reliable security environment. However, in real-world complex environments such as edge computing and hybrid cloud deployments, the design of scheduling, authentication, and disaster recovery logic being isolated has the following technical defects: (1) Mismatch of computing resources leads to low task execution efficiency. The Master node cannot perceive the dynamic status information such as CPU load, memory level and network latency of the Agent node in real time. The fixed-granularity task allocation strategy cannot be adapted to heterogeneous computing nodes. High-configuration nodes with sufficient computing power are in a semi-idle state for a long time. Low-configuration nodes with limited computing power are prone to memory overflow and crash due to continuously pulling multiple tasks, resulting in a long tail effect in the distributed system. The overall task completion time is dragged down by the slowest node, and the system throughput is difficult to expand linearly with the node scale. (2) Static authentication mechanism has the risk of data forgery and replay attack. The existing system relies on a long-term static token issued once for identity verification. There is no cryptographic binding between task issuance and result return. Once the communication link is intercepted or the edge node is compromised, the attacker can extract the static token and reuse it to forge false execution results and send them back to the Master node. The Master node cannot verify the correlation between the result and the issued task and directly stores it, causing irreversible data pollution. This risk is particularly prominent in deployment scenarios where the physical security of the edge node is difficult to guarantee. (3) In a weak network environment, the results of computation are easily lost and there is a lack of anti-avalanche mechanism. When the network is interrupted abnormally, the intermediate or final computation results completed by the Agent are directly lost because they are not persisted. After the network is restored, all computation tasks need to be re-executed, resulting in a large waste of computing power. When a large number of Agent nodes go offline at the same time due to network failure and then recover instantly, all nodes initiate authentication and task retrieval requests at almost the same time, forming an instantaneous concurrent flood, which can easily cause the Master node control plane and backend database to crash due to overload, i.e. the thundering herd effect, which makes the system unable to heal itself or even completely paralyzed during the fault recovery phase.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] To address the aforementioned technical problems in existing technologies, this invention provides a distributed secure scheduling and self-consistent recovery method and system. This solves the problems of existing pull-based distributed scheduling systems, which suffer from fragmented scheduling, authentication, and disaster recovery logic, inability to adapt to heterogeneous computing power, susceptibility to data forgery and replay, loss of network outage results, and the risk of concurrent avalanche caused by large-scale reconnection.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: Firstly, a distributed secure scheduling and self-consistent recovery method is applied to a Master-Agent pull-based distributed task scheduling architecture. A closed-loop signaling state machine is constructed using four gRPC interfaces provided by the Master node: Authenticate, KeepAlive, GetScanData, and UpdateScanResult. The method includes the following steps: S1. The Agent node calls the Authenticate interface to send an authentication request to the Master node. After the Master node verifies the identity, it sends the session dynamic seed Base_Seed and the initial dynamic random number Nonce to the Agent node, and initializes the global task status cursor Offset for the Agent node. S2. The Agent node periodically calls the KeepAlive interface to collect local multi-dimensional load data, read the backlog offset of the local Write-Ahead Log (WAL), and generate a dynamic drift signature (Drift_Sign) based on Base_Seed, the locally maintained monotonically increasing sequence number SequenceID, and the current Nonce. The multi-dimensional load data, backlog offset, and Drift_Sign are then encapsulated and sent to the Master node. After the Master node verifies the signature, it calculates the Agent node health score based on the multi-dimensional load data, updates the Agent node status record, and generates a new Nonce to return to the Agent node. S3. The Agent node calls the GetScanData interface, carrying the current Drift_Sign and Nonce to request a task from the Master node. After the Master node verifies the signature, it determines the task granularity based on the health score and divides the task into blocks, generates a task fingerprint (TaskFingerprint) and stores it in association with the current Drift_Sign, and then sends the task blocks and TaskFingerprint to the Agent node. S4. After the Agent node completes the task block execution, it calls the UpdateScanResult interface to concatenate the execution result data, TaskFingerprint, current Drift_Sign, and current Nonce in a preset order and calculate the anti-counterfeiting digest Verification_Hash using a one-way hash function. The execution result data and Verification_Hash are then sent to the Master node. After the Master node verifies the signature, it stores the execution result data. S5. When the network is interrupted, the Agent node writes the completed but not yet returned execution result data into the local WAL in sequence and records the corresponding Offset. After the network is restored, the Agent node calculates the waiting time according to the preset backoff algorithm and then calls the Authenticate interface again to initiate authentication. After successful authentication, the backlog of Offset in the WAL is reported in the KeepAlive request. The Master node responds through the UpdateScanResult interface according to the Offset and instructs the Agent node to return the backlog of execution result data in the WAL in batches until the Offset is cleared to zero.

[0008] Further, in step S2, the multidimensional load data includes the CPU utilization, memory utilization, and round-trip time (RTT) between the Agent node and the Master node; the Master node calculates the Agent node health score according to the following formula:

[0009] in, For CPU utilization, For memory usage, W1, W2, and W3 are preset weight coefficients representing the network round-trip delay mapped to the 0-1 interval after max-min normalization.

[0010] Furthermore, the health score also includes a non-linear coupling penalty term, calculated as follows:

[0011] Where W4 is the coupling penalty weight coefficient, The coupling penalty function is a combination of a product term and an exponential adjustment function. , It is an exponential adjustment factor.

[0012] Furthermore, during the Agent node S1 authentication phase, the Master node, based on the hardware metadata reported by the Agent node, differentially configures the coupling of the penalty weight coefficient W4 and the exponential adjustment factor α: When the number of CPU cores on the Agent node is ≤2 and the total memory capacity is ≤4GB, W4=0.5, α=2.0; When the number of CPU cores of the Agent node is greater than 2 and not greater than 8, or the total memory capacity is greater than 4GB and not greater than 16GB, W4=0.3 and α=1.5. When the number of CPU cores on the Agent node is greater than 8 and the total memory capacity is greater than 16GB, W4=0.2 and α=1.2.

[0013] Furthermore, in step S3, the Master node determines the task granularity based on the health score according to the following rules: When the score is greater than or equal to 0.5, the task is divided into large-granularity blocks and distributed. When -0.3 ≤ Score < 0.5, the task is divided into default granularity blocks and distributed. When the score is less than -0.3, the task is divided into small-granularity blocks and either issued or an empty task is returned.

[0014] Furthermore, when the score is less than -0.5, the Master node triggers a scheduling circuit breaker, suspends the delivery of tasks to the corresponding Agent node, and issues a hibernation command to the Agent node, instructing the Agent node to enter a cooldown observation period of a preset duration.

[0015] Furthermore, in step S4, the Agent node maintains a write pointer and an acknowledgment watermark in its local WAL. The write pointer points to the end of the latest appended record, and the acknowledgment watermark points to the position of the last record that has been acknowledged and received by the Master node. When the WAL file size reaches a preset upper limit, the Agent node only reclaims the space of the acknowledged data before the acknowledgment watermark.

[0016] Furthermore, when it is confirmed that there is insufficient reclaimable space before the water level, the Agent node suspends the execution of new task blocks, prioritizes calling the UpdateScanResult interface to return the execution result data backed up in WAL, and resumes the execution of new tasks only after the reclaimable space meets the writing requirements.

[0017] Furthermore, in step S5, after the Agent node network recovers, the reconnection waiting time is calculated according to the following formula. :

[0018] Where Base is the basic retry interval, N is the number of consecutive failed retries, Cap is the maximum waiting time limit, and Random(0,Jitter) is a random value in the range of 0 to Jitter.

[0019] Secondly, a distributed security scheduling and self-consistent recovery system includes a Master node and at least one Agent node. The Master node and the Agent node communicate through a gRPC interface to form a closed-loop signaling state machine that includes an Authenticate interface, a KeepAlive interface, a GetScanData interface, and an UpdateScanResult interface. The Agent node includes: The authentication initiation unit is used to call the Authenticate interface to initiate an authentication request to the Master node, and to receive and store the session dynamic seed Base_Seed and the initial dynamic random number Nonce issued by the Master node; The telemetry signature unit is used to periodically collect local multidimensional load data, read the backlog offset of the local write-ahead log (WAL), generate a dynamic drift signature (Drift_Sign) based on Base_Seed, the locally maintained monotonically increasing sequence number SequenceID, and the current Nonce, and send the multidimensional load data, backlog offset, and Drift_Sign to the Master node through the KeepAlive interface. It also receives the new Nonce returned by the Master node and updates the Nonce stored locally. The task request unit is used to call the GetScanData interface, carrying the current Drift_Sign and Nonce to request a task from the Master node, and to receive the task block and task fingerprint TaskFingerprint issued by the Master node. The result feedback unit is used to concatenate the execution result data, TaskFingerprint, current Drift_Sign and current Nonce in a preset order after the task block is executed, and calculate the anti-counterfeiting digest Verification_Hash through a one-way hash function. Then, it sends the execution result data and Verification_Hash to the Master node through the UpdateScanResult interface. The local disaster recovery unit is used to write completed but not yet returned execution result data into the local WAL in sequence and record the corresponding offset when the network is interrupted; when the network is restored, the waiting time is calculated according to the preset backoff algorithm and the authentication initiation unit is triggered to reconnect. After the reconnection authentication is successful, the backlog offset in the WAL is reported through the telemetry signature unit, and the backlog execution result data in the WAL is returned in batches through the result return unit according to the instructions of the Master node until the offset is cleared to zero. The Master node includes: The authentication processing unit is used to receive authentication requests from Agent nodes through the Authenticate interface, and after verifying the identity, sends Base_Seed and initial Nonce to Agent nodes to initialize the global task status cursor Offset for Agent nodes; The telemetry processing unit is used to receive multidimensional load data, backlog offset and Drift_Sign sent by the Agent node through the KeepAlive interface. After the signature is verified, it calculates the Agent node health score based on the multidimensional load data, updates the Agent node status record, and generates a new Nonce to return to the Agent node. The task scheduling unit is used to receive task requests from Agent nodes through the GetScanData interface. After the signature is verified, the task granularity is determined according to the health score and the task blocks are divided. The TaskFingerprint is generated and stored in association with the current Drift_Sign. The task blocks and TaskFingerprint are then sent to Agent nodes. The result verification unit is used to receive the execution result data and Verification_Hash sent by the Agent node through the UpdateScanResult interface, and store the execution result data after the signature is verified. The disaster recovery control unit is used to send a feedback instruction to the Agent node through the UpdateScanResult interface based on the backlog offset reported by the Agent node, and control the Agent node to send back the backlog execution result data in WAL in batches until the offset is cleared to zero.

[0020] The present invention has at least the following beneficial effects: (1) To achieve adaptive scheduling of tasks in a heterogeneous computing environment, the KeepAlive process integrates multi-dimensional real-time telemetry data of CPU, memory, and network round-trip delay. The Master dynamically adjusts the task slice granularity based on the node health score, so that the task distribution scale matches the actual processing capacity of the node in real time. This avoids the problem of idle resources of high-configuration nodes and overload and crash of low-configuration nodes caused by fixed granularity allocation, eliminates the long tail effect of heterogeneous clusters, and improves the overall throughput and computing power utilization of the system. (2) Construct a request-level non-replayable dynamic authentication system, abandon the traditional long-term static token mechanism, generate an independent drift signature for each request through session dynamic seed and round-by-round update of random numbers, and combine task fingerprint and result return multi-field joint signature verification to realize full-link cryptographic binding of task issuance, node execution and result return. Even if the communication link is intercepted or the edge node is invaded, it is impossible to forge a legitimate return result, prevent replay attacks and fake data injection from the source, and ensure data credibility. (3) To achieve persistent retention and accurate retransmission of computing results in weak network scenarios, the Agent end pre-writes logs to persist the results completed during the network outage, and combines the global status cursor to mark the synchronization progress. After the network is restored, only the unsynchronized data is incrementally transmitted back, without having to repeat the completed computing tasks, thus avoiding the waste of computing power and the amplification of task delays. (4) By using the exponential backoff combined with the random jitter reconnection mechanism, the authentication requests of a large number of nodes recovering simultaneously are scattered on the time axis, avoiding the instantaneous concurrent flood from breaking through the Master control plane and the database connection pool, ensuring that the system can recover smoothly after a large-scale network outage, and improving the system availability in a complex environment with weak network. Attached Figure Description

[0021] Figure 1 A schematic diagram of the deployment architecture for Master nodes and Agent nodes; Figure 2 This is a schematic diagram of the Agent node security authentication and session initialization process; Figure 3 A schematic diagram of the Agent node KeepAlive liveness detection and multi-dimensional state telemetry process; Figure 4 A schematic diagram of the Agent node task retrieval and adaptive slice distribution process; Figure 5 A schematic diagram illustrating the process of preventing tampering with the execution results of the Agent node. Detailed Implementation

[0022] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement and numerical expressions of the components and steps described in these embodiments should not be construed as limiting the scope of the invention.

[0024] The following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail herein, but where applicable, such techniques, methods, and apparatus should be considered part of this specification.

[0025] Terminology Explanation: gRPC (gRPC Remote Procedure Call): gRPC remote procedure call; WAL (Write-Ahead Logging): Write-ahead logging; CPU (Central Processing Unit): Central processing unit; RTT (Round-Trip Time): Round-trip delay; Token (Security Token): A secure access token; JSON (JavaScript Object Notation): A simplified notation of JavaScript objects; SHA-256 (Secure Hash Algorithm 256-bit): A 256-bit secure hash algorithm; SM3 (Commercial Cryptography Standard SM3): The Chinese national cryptographic standard SM3 hash algorithm; HMAC (Hash-based Message Authentication Code): A message authentication code based on hashing. EWMA (Exponential Weighted Moving Average): Exponentially weighted moving average.

[0026] Example 1 This invention proposes a distributed secure scheduling and self-consistent recovery method, applied to a Master-Agent pull-based distributed task scheduling architecture. Using gRPC as the underlying communication framework, a closed-loop signaling state machine is constructed through four gRPC interfaces provided by the Master node: Authenticate, KeepAlive, GetScanData, and UpdateScanResult. The Agent node, acting as a gRPC client, actively initiates calls to the Master node. The overall deployment architecture and interface call relationships are as follows: Figure 1 As shown; specific steps may include: S1. Secure Connection Establishment and Session Seed Initialization: When the Agent node starts up for the first time or reconnects after a network interruption, it first maintains a consecutive failure counter N with an initial value of 0, and calculates the waiting time using the following formula:

[0027] In this embodiment, the base retry interval (Base) is 1 second, the maximum waiting limit (Cap) is 15 seconds, and the jitter range (Jitter) is 2 seconds. After waiting for the corresponding duration, the Authenticate interface is called to initiate an authentication request to the Master node. The request carries the Agent node's unique identifier and the identity credential (Token). If authentication fails, N is incremented by 1 to recalculate the waiting time. If authentication succeeds, N is reset to 0.

[0028] After receiving the request, the Master node verifies the validity of the identity credentials. If the verification is successful, it generates a 128-bit random string valid for the current session period as the dynamic session seed Base_Seed, and at the same time generates a 64-bit random string as the initial dynamic random number Nonce. The Base_Seed and the initial Nonce are then sent to the Agent node together. The Master node synchronously creates a state machine record for the Agent node in memory. The record fields include node identifier, current session Base_Seed, current valid Nonce, global task status cursor Offset, health score Score, and last heartbeat timestamp. The initial value of Offset is set to NULL, which is used to mark the data synchronization progress position of the Agent node in the global task status tree maintained by the Master node. The initial value of Score is set to 0. The nonce is regenerated after each response from the Master node to a request from the Agent node, exhibiting unidirectional unpredictability; the current Base_Seed becomes invalid after the session is closed. The security authentication signaling interaction flow for this step is as follows: Figure 2 As shown.

[0029] S2. Multidimensional State Telemetry and Dynamic Signature Synchronization: After completing the S1 authentication process, the Agent node calls the KeepAlive interface at 10-second intervals to perform state reporting. Each call first collects local multidimensional load data, including calling the operating system interface to obtain current CPU and memory usage. Simultaneously, it records the time difference between the last request sent and the response received as the current round-trip time (RTT). An Exponentially Weighted Moving Average (EWMA) algorithm is used to smooth the RTT, with a smoothing factor α of 0.3. The smoothing formula is as follows:

[0030] Next, scan the local write-ahead log (WAL) storage directory. If there is backlog of result data that was not successfully transmitted due to historical network interruptions, read the global task status tree offset Offset corresponding to that batch of data. If there is no backlog of data, set the Offset field to empty. Subsequently, based on the locally stored Base_Seed, the locally maintained monotonically increasing sequence number SequenceID (initial value 0, incremented by 1 after each request), and the currently stored valid Nonce, the dynamic drift signature Drift_Sign for this request is calculated using the SHA-256 one-way hash function. The calculation formula is as follows:

[0031] in, This indicates that the strings are concatenated sequentially. The SequenceID increments by 1 after each request is initiated by the Agent node.

[0032] The Agent node encapsulates the smoothed RTT, CPU utilization, memory utilization, backlog offset, Drift_Sign, and current SequenceID in a KeepAlive request and sends it to the Master node.

[0033] After receiving a KeepAlive request, the Master node first calculates the expected signature value using the same algorithm based on the Base_Seed of the Agent node stored locally, the currently valid Nonce, and the SequenceID carried in the request. It then compares the result with the Drift_Sign in the request. If the signature verification is successful, it continues with subsequent processing. The smoothed RTT is then mapped to the 0~1 interval using max-min normalization to obtain RTT_norm. The health score of the Agent node is then calculated according to a preset weighted formula. The basic linear calculation formula is as follows:

[0034] in, The default weighting coefficient, Score, ranges from -1 to 1; As a preferred implementation, the scoring formula can incorporate a nonlinear coupling penalty term, as shown in the following formula:

[0035] in, The coupling penalty weight coefficient, As an exponential adjustment factor, the Master node differentiates configuration parameters based on the hardware metadata reported by the Agent during the S1 authentication phase: For small-scale nodes with ≤2 CPU cores and ≤4GB total memory, the configuration is W4=0.5 and α=2.0. For medium-scale nodes with more than 2 but not more than 8 CPU cores or more than 4GB but not more than 16GB total memory, the configuration is W4=0.3 and α=1.5. For large-scale nodes with more than 8 CPU cores and more than 16GB total memory, the configuration is W4=0.2 and α=1.2.

[0036] The Master node updates the calculated Score in the state machine record of the Agent node, and simultaneously updates the Offset cursor value in the record. It then generates a new 64-bit Nonce, encapsulates it in the KeepAlive response, and returns it to the Agent node. Upon receiving the Nonce, the Agent node replaces the old Nonce stored locally for signature calculation in the next request. The KeepAlive liveness detection and telemetry interaction process in this step is as follows: Figure 3 As shown.

[0037] S3, Computational Power Awareness and Adaptive Task Slicing Distribution: After the Agent node completes the KeepAlive interaction with S2, it calls the GetScanData interface with the current Drift_Sign and the latest obtained Nonce to request a new task from the Master node.

[0038] After receiving a request, the Master node first performs signature verification. If the signature verification is successful, it reads the current health score (Score) of the Agent node from memory and determines the task distribution granularity based on the Score value, categorizing it into three levels: When the score is greater than or equal to 0.5, the node is deemed to have sufficient computing power. Large-granularity task blocks are then divided from the global task pool. In this embodiment, the large-granularity is 200 scanning targets / batch. When -0.3 ≤ Score < 0.5, the node's computing power is considered normal, and the task blocks are divided into default granularity blocks. In this embodiment, the default granularity is 100 scan targets / batch. When the score is less than -0.3, the node is determined to be under high load. The task is then divided into smaller granularity blocks or an empty task is returned. In this embodiment, the smaller granularity is 20 scan targets / batch.

[0039] As a preferred implementation, when Score < -0.5, the Master node triggers the scheduling circuit breaker for the Agent node, suspends the issuance of new tasks to it, and carries a sleep instruction in the response, instructing the Agent to enter a 30-second cooldown observation period, and the circuit breaker is lifted after the Score recovers to the safe range of ≥ -0.2. When -0.5 ≤ Score < -0.2, maintain a minimum granularity of 20 scan targets / batch, and attach a low priority flag to the task packet header, allowing the Agent to decide to slow down the processing based on its own load during execution.

[0040] After the Master node determines the task block to be distributed, it serializes the task data and performs a SHA-256 one-way hash operation on the serialized string to generate a unique task fingerprint, TaskFingerprint, for that task block. The TaskFingerprint is then associated with the current Drift_Sign in the Master's local memory, forming a three-way mapping record of Agent identifier, Drift_Sign, and TaskFingerprint. Subsequently, the task block and TaskFingerprint are encapsulated in a GetScanData response and distributed to the Agent node. The task acquisition and adaptive distribution process in this step is as follows: Figure 4 As shown.

[0041] S4. Anti-tampering result return and local persistence: After the Agent node receives and executes the task block issued by S3, it first serializes the execution result into a JSON string Result_Data. Then, it concatenates Result_Data, the received TaskFingerprint, the current Drift_Sign, and the current valid Nonce in a preset order. Finally, it calculates the anti-counterfeiting digest Verification_Hash using the SHA-256 one-way hash function. The calculation formula is as follows:

[0042] The Agent node encapsulates Result_Data and Verification_Hash in an UpdateScanResult request and sends it to the Master node.

[0043] After receiving the request, the Master node uses the same algorithm and concatenation order to calculate the expected digest value using the TaskFingerprint, Drift_Sign, and currently valid Nonce stored locally for the Agent node. It then compares this value with the Verification_Hash in the request. If the comparison matches, the result is deemed valid and unaltered, and the Result_Data is parsed and stored in the database. If the comparison does not match, the result is rejected and the Agent node is marked as abnormal.

[0044] The Agent node maintains a Write-Ahead Log (WAL) locally. The maximum size of the WAL file is set to 500MB. When the connection with the Master node is interrupted, the completed but not yet returned execution results are not discarded. Instead, the result data is appended to the WAL file in order. Each WAL record contains four fields: result data body, corresponding Offset, write timestamp, and cycle sequence number. The Agent node maintains a write pointer and an acknowledgment watermark in memory. The write pointer points to the end of the latest appended record, and the acknowledgment watermark points to the position of the last record that has been acknowledged and received by the Master node.

[0045] When the WAL file size reaches the 500MB limit, the Agent node calculates the reclaimable space before the confirmation waterline. If the reclaimable space meets the data to be written, the confirmed data space before the confirmation waterline is reclaimed. If there is insufficient reclaimable space, the Agent node will suspend the execution of new task blocks and prioritize sending back the backlogged result data in the WAL. Once it is confirmed that the water level has advanced and the reclaimable space meets the requirements, the execution of new tasks and the writing of WAL will be resumed. In extreme scenarios where the Master node is unavailable for an extended period and WAL usage exceeds 95%, the Agent node compresses the earliest unconfirmed result data and temporarily stores it in a local backup storage partition. This data is marked as an emergency transfer in the WAL, and once the Master recovers, the corresponding data is preferentially transferred from the backup partition. The result return and local persistence process for this step is as follows: Figure 5 As shown.

[0046] S5. Anomaly Recovery and Incremental Self-Consistent Recovery: When the Agent node detects a connection interruption with the Master node, it continues to write the completed execution results to the local WAL and record the corresponding Offset as described in S4. After the network is restored, the Agent node does not immediately initiate an authentication request. It calculates the reconnection waiting time according to the formula and parameters described in S1, waits for the corresponding duration, and then returns to execute the authentication process of S1 to re-establish the session. As a preferred implementation, the Master node can dynamically adjust the backoff parameters of the Agent node based on the historical load data reported by the Agent node through the KeepAlive interface before the Agent node went offline, and send the adjusted parameters to the Agent node for execution through the Authenticate interface response or the KeepAlive interface response.

[0047] After the Agent node successfully reconnects and authenticates, it piggybacks the backlog offset of the local WAL record in the first KeepAlive request. Upon receiving this, the Master node updates the Offset cursor of the Agent node, sets the incremental alignment flag in the UpdateScanResult response, and returns the current backlog starting offset.

[0048] After receiving the flag, the Agent node pauses calling the GetScanData interface to request new tasks, reads the backlog result data from the local WAL starting from the corresponding Offset, and sends it back to the Master node in batches through multiple UpdateScanResult calls. After each batch is successfully sent back, the Master node updates the Offset and returns the latest progress until the Master node confirms that the Offset is cleared to zero, and the Agent node resumes the normal task retrieval process.

[0049] In this embodiment, the one-way hash function can be replaced with the SM3 national cryptographic algorithm, SHA-384, SHA-512, or HMAC structure. The backoff algorithm can be replaced with a linear backoff algorithm or the retry time can be specified by the Master through the Retry-After field. The task granularity decision can be replaced with a mapping function that changes continuously with the score. All of the above replacements do not exceed the technical scope of this solution.

[0050] Example 2 The present invention proposes a distributed security scheduling and self-consistent recovery system, which includes a Master node and at least one Agent node. The Master node and the Agent node communicate through a gRPC interface to form a closed-loop signaling state machine including an Authenticate interface, a KeepAlive interface, a GetScanData interface, and an UpdateScanResult interface. The Agent node includes: The authentication initiation unit is used to call the Authenticate interface to initiate an authentication request to the Master node, and to receive and store the session dynamic seed Base_Seed and the initial dynamic random number Nonce issued by the Master node; The telemetry signature unit is used to periodically collect local multidimensional load data, read the backlog offset of the local write-ahead log (WAL), generate a dynamic drift signature (Drift_Sign) based on Base_Seed, the locally maintained monotonically increasing sequence number SequenceID, and the current Nonce, and send the multidimensional load data, backlog offset, and Drift_Sign to the Master node through the KeepAlive interface. It also receives the new Nonce returned by the Master node and updates the Nonce stored locally. The task request unit is used to call the GetScanData interface, carrying the current Drift_Sign and Nonce to request a task from the Master node, and to receive the task block and task fingerprint TaskFingerprint issued by the Master node. The result feedback unit is used to concatenate the execution result data, TaskFingerprint, current Drift_Sign and current Nonce in a preset order after the task block is executed, and calculate the anti-counterfeiting digest Verification_Hash through a one-way hash function. Then, it sends the execution result data and Verification_Hash to the Master node through the UpdateScanResult interface. The local disaster recovery unit is used to write completed but not yet returned execution result data into the local WAL in sequence and record the corresponding offset when the network is interrupted; when the network is restored, the waiting time is calculated according to the preset backoff algorithm and the authentication initiation unit is triggered to reconnect. After the reconnection authentication is successful, the backlog offset in the WAL is reported through the telemetry signature unit, and the backlog execution result data in the WAL is returned in batches through the result return unit according to the instructions of the Master node until the offset is cleared to zero. The Master node includes: The authentication processing unit is used to receive authentication requests from Agent nodes through the Authenticate interface, and after verifying the identity, sends Base_Seed and initial Nonce to Agent nodes to initialize the global task status cursor Offset for Agent nodes; The telemetry processing unit is used to receive multidimensional load data, backlog offset and Drift_Sign sent by the Agent node through the KeepAlive interface. After the signature is verified, it calculates the Agent node health score based on the multidimensional load data, updates the Agent node status record, and generates a new Nonce to return to the Agent node. The task scheduling unit is used to receive task requests from Agent nodes through the GetScanData interface. After the signature is verified, the task granularity is determined according to the health score and the task blocks are divided. The TaskFingerprint is generated and stored in association with the current Drift_Sign. The task blocks and TaskFingerprint are then sent to Agent nodes. The result verification unit is used to receive the execution result data and Verification_Hash sent by the Agent node through the UpdateScanResult interface, and store the execution result data after the signature is verified. The disaster recovery control unit is used to send a feedback instruction to the Agent node through the UpdateScanResult interface based on the backlog offset reported by the Agent node, and control the Agent node to send back the backlog execution result data in WAL in batches until the offset is cleared to zero.

[0051] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A distributed secure scheduling and self-consistent recovery method, applied to a Master-Agent pull-based distributed task scheduling architecture, characterized in that, The method comprises the following steps: A closed-loop signaling state machine is constructed using four gRPC interfaces provided by the Master node: Authenticate, KeepAlive, GetScanData, and UpdateScanResult. S1. The Agent node calls the Authenticate interface to send an authentication request to the Master node. After the Master node verifies the identity, it sends the session dynamic seed Base_Seed and the initial dynamic random number Nonce to the Agent node, and initializes the global task status cursor Offset for the Agent node. S2. The Agent node periodically calls the KeepAlive interface to collect local multi-dimensional load data, read the backlog offset of the local Write-Ahead Log (WAL), and generate a dynamic drift signature (Drift_Sign) based on Base_Seed, the locally maintained monotonically increasing sequence number SequenceID, and the current Nonce. The multi-dimensional load data, backlog offset, and Drift_Sign are then encapsulated and sent to the Master node. After the Master node verifies the signature, it calculates the Agent node health score based on the multi-dimensional load data, updates the Agent node status record, and generates a new Nonce to return to the Agent node. S3. The Agent node calls the GetScanData interface, carrying the current Drift_Sign and Nonce to request a task from the Master node. After the Master node verifies the signature, it determines the task granularity based on the health score and divides the task into blocks, generates a task fingerprint (TaskFingerprint) and stores it in association with the current Drift_Sign, and then sends the task blocks and TaskFingerprint to the Agent node. S4. After the Agent node completes the task block execution, it calls the UpdateScanResult interface to concatenate the execution result data, TaskFingerprint, current Drift_Sign, and current Nonce in a preset order and calculate the anti-counterfeiting digest Verification_Hash using a one-way hash function. The execution result data and Verification_Hash are then sent to the Master node. After the Master node verifies the signature, it stores the execution result data. S5. When the network is interrupted, the Agent node writes the completed but not yet returned execution result data into the local WAL in sequence and records the corresponding Offset. After the network is restored, the Agent node calculates the waiting time according to the preset backoff algorithm and then calls the Authenticate interface again to initiate authentication. After successful authentication, the backlog of Offset in the WAL is reported in the KeepAlive request. The Master node responds through the UpdateScanResult interface according to the Offset and instructs the Agent node to return the backlog of execution result data in the WAL in batches until the Offset is cleared to zero.

2. The distributed secure scheduling and self-consistent recovery method according to claim 1, characterized in that, In step S2, the multidimensional load data includes the CPU utilization, memory utilization, and round-trip time (RTT) between the Agent node and the Master node; the Master node calculates the Agent node health score according to the following formula: in, For CPU utilization, For memory usage, W1, W2, and W3 are preset weight coefficients representing the network round-trip delay mapped to the 0-1 interval after max-min normalization.

3. The distributed secure scheduling and self-consistent recovery method according to claim 2, characterized in that, The health score also includes a non-linear coupling penalty term, calculated as follows: Where W4 is the coupling penalty weight coefficient, The coupling penalty function is a combination of a product term and an exponential adjustment function. , It is an exponential adjustment factor.

4. The distributed secure scheduling and self-consistent recovery method according to claim 3, characterized in that, During the Agent node S1 authentication phase, the Master node, based on the hardware metadata reported by the Agent node, differentially configures the coupling of the penalty weight coefficient W4 and the exponential adjustment factor α: When the number of CPU cores on the Agent node is ≤2 and the total memory capacity is ≤4GB, W4=0.5, α=2.0; When the number of CPU cores of the Agent node is greater than 2 and not greater than 8, or the total memory capacity is greater than 4GB and not greater than 16GB, W4=0.3 and α=1.

5. When the number of CPU cores on the Agent node is greater than 8 and the total memory capacity is greater than 16GB, W4=0.2 and α=1.

2.

5. The distributed secure scheduling and self-consistent recovery method according to claim 1, characterized in that, In step S3, the Master node determines the task granularity based on the health score according to the following rules: When the score is greater than or equal to 0.5, the task is divided into large-granularity blocks and distributed. When -0.3 ≤ Score < 0.5, the task is divided into default granularity blocks and distributed. When the score is less than -0.3, the task is divided into small-granularity blocks and either issued or an empty task is returned.

6. The distributed secure scheduling and self-consistent recovery method according to claim 5, characterized in that, When the score is less than -0.5, the Master node triggers a scheduling circuit breaker, suspends the delivery of tasks to the corresponding Agent node, and sends a hibernation command to the Agent node, instructing the Agent node to enter a cooldown observation period of a preset duration.

7. The distributed secure scheduling and self-consistent recovery method according to claim 1, characterized in that, In step S4, the Agent node maintains a write pointer and an acknowledgment watermark in its local WAL. The write pointer points to the end of the latest appended record, and the acknowledgment watermark points to the position of the last record that has been acknowledged and received by the Master node. When the WAL file size reaches a preset upper limit, the Agent node only reclaims the space of the acknowledged data before the acknowledgment watermark.

8. The distributed secure scheduling and self-consistent recovery method according to claim 7, characterized in that, When it is confirmed that there is insufficient reclaimable space before the water level, the Agent node suspends the execution of new task blocks, prioritizes calling the UpdateScanResult interface to return the execution result data backed up in WAL, and resumes the execution of new tasks once the reclaimable space meets the writing requirements.

9. The distributed secure scheduling and self-consistent recovery method according to claim 1, characterized in that, In step S5, after the Agent node network recovers, the reconnection waiting time is calculated according to the following formula. : Where Base is the basic retry interval, N is the number of consecutive failed retries, Cap is the maximum waiting time limit, and Random(0,Jitter) is a random value in the range of 0 to Jitter.

10. A distributed secure scheduling and self-consistent recovery system, characterized in that, It includes a Master node and at least one Agent node. The Master node and Agent node communicate through a gRPC interface to form a closed-loop signaling state machine that includes the Authenticate interface, KeepAlive interface, GetScanData interface, and UpdateScanResult interface. The Agent node includes: The authentication initiation unit is used to call the Authenticate interface to initiate an authentication request to the Master node, and to receive and store the session dynamic seed Base_Seed and the initial dynamic random number Nonce issued by the Master node; The telemetry signature unit is used to periodically collect local multidimensional load data, read the backlog offset of the local write-ahead log (WAL), generate a dynamic drift signature (Drift_Sign) based on Base_Seed, the locally maintained monotonically increasing sequence number SequenceID, and the current Nonce, and send the multidimensional load data, backlog offset, and Drift_Sign to the Master node through the KeepAlive interface. It also receives the new Nonce returned by the Master node and updates the Nonce stored locally. The task request unit is used to call the GetScanData interface, carrying the current Drift_Sign and Nonce to request a task from the Master node, and to receive the task block and task fingerprint TaskFingerprint issued by the Master node. The result feedback unit is used to concatenate the execution result data, TaskFingerprint, current Drift_Sign and current Nonce in a preset order after the task block is executed, and calculate the anti-counterfeiting digest Verification_Hash through a one-way hash function. Then, it sends the execution result data and Verification_Hash to the Master node through the UpdateScanResult interface. The local disaster recovery unit is used to write completed but not yet returned execution result data into the local WAL in sequence and record the corresponding offset when the network is interrupted; when the network is restored, the waiting time is calculated according to the preset backoff algorithm and the authentication initiation unit is triggered to reconnect. After the reconnection authentication is successful, the backlog offset in the WAL is reported through the telemetry signature unit, and the backlog execution result data in the WAL is returned in batches through the result return unit according to the instructions of the Master node until the offset is cleared to zero. The Master node includes: The authentication processing unit is used to receive authentication requests from Agent nodes through the Authenticate interface, and after verifying the identity, sends Base_Seed and initial Nonce to Agent nodes to initialize the global task status cursor Offset for Agent nodes; The telemetry processing unit is used to receive multidimensional load data, backlog offset and Drift_Sign sent by the Agent node through the KeepAlive interface. After the signature is verified, it calculates the Agent node health score based on the multidimensional load data, updates the Agent node status record, and generates a new Nonce to return to the Agent node. The task scheduling unit is used to receive task requests from Agent nodes through the GetScanData interface. After the signature is verified, the task granularity is determined according to the health score and the task blocks are divided. The TaskFingerprint is generated and stored in association with the current Drift_Sign. The task blocks and TaskFingerprint are then sent to Agent nodes. The result verification unit is used to receive the execution result data and Verification_Hash sent by the Agent node through the UpdateScanResult interface, and store the execution result data after the signature is verified. The disaster recovery control unit is used to send a feedback instruction to the Agent node through the UpdateScanResult interface based on the backlog offset reported by the Agent node, and control the Agent node to send back the backlog execution result data in WAL in batches until the offset is cleared to zero.

Images