A network scanning task intelligent segmentation and load balancing method and system

Abstract

The application discloses a network scanning task intelligent segmentation and load balancing method and system, which identifies abnormal fragments by performing integrity check on scanning tasks, analyzes task dependency to determine a segmentable position, and generates a segmentation scheme in combination with granularity constraints; collects node resource states to calculate a load index, configures and schedules weights according to load levels to establish parallel scheduling channels; constructs node cooperation groups based on load differences and formulates inter-group flow transfer rules to form a cooperative scheduling space, fragments tasks according to node capacities and generates a parallel execution plan; monitors execution progress to identify backlog nodes, diagnoses bottleneck types and plans a dredging path to execute task migration; collects resource time sequence data to identify low-load time periods, and distributes tasks to optimal time slots for execution, which improves the rationality of task segmentation and the degree of node load balancing, and improves the distributed scanning execution efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a network scanning task intelligent segmentation and load balancing method and system. BACKGROUND

[0002] As an important part of the network security protection system, network scanning undertakes key tasks such as asset discovery, port detection, and vulnerability detection. With the continuous expansion of enterprise network scale, the single-node scanning mode has been difficult to meet the scanning timeliness requirements in large-scale network environment, and distributed multi-node collaborative scanning has become an inevitable choice.

[0003] However, the existing distributed scanning scheduling scheme has many deficiencies. The scanning task is usually submitted in the form of a whole, lacking analysis of the internal dependency of the task, making it difficult to identify reasonable segmentation boundaries, resulting in improper task splitting granularity. The evaluation mechanism of node load state is relatively extensive, and the task distribution adopts static strategy or simple polling, which cannot dynamically adapt according to the real-time carrying capacity of the node. In addition, there is a lack of effective bottleneck identification means in the execution process, and it is difficult to dredge in time when some nodes have task backlog, resulting in a decline in overall scanning efficiency. Therefore, an intelligent scanning task processing method is needed to solve at least one of the above problems. SUMMARY

[0004] The present application discloses a network scanning task intelligent segmentation and load balancing method and system, aiming to solve the problems of improper task segmentation granularity, node load imbalance, and execution bottleneck difficult to eliminate in existing schemes. The present application determines the segmentation boundary through task integrity verification and dependency analysis, and establishes parallel scheduling links based on node load grading; through collaborative relationship modeling, a scheduling space is constructed to realize balanced distribution of tasks; through progress tracking and bottleneck diagnosis, task dredging is executed, combined with resource timing analysis to optimize distribution timing, realizing efficient and balanced execution of scanning tasks.

[0005] The first aspect of the present application proposes a network scanning task intelligent segmentation and load balancing method, comprising the following steps:

[0006] Obtain the network scanning task request to be processed, perform task integrity verification on the network scanning task request to identify abnormal task fragments, and generate a segmentation reference sequence based on the task feature analysis of the abnormal task fragments;

[0007] According to the segmentation reference sequence, task unit reorganization processing is carried out to generate a standard task queue, node state synchronization analysis is performed on the standard task queue to extract node load identifiers, and a scheduling mode adaptation mechanism is triggered according to the node load identifiers to generate mode adaptation instructions, and a multi-node parallel scheduling link is established through the mode adaptation instructions;

[0008] A node balance index is generated by performing load coordination analysis on the multi-node parallel scheduling link, multi-node coordination balancing processing is performed according to the node balance index to form a coordination scheduling space, and a multi-node parallel execution sequence is generated by implementing task fragmentation distribution on the coordination scheduling space;

[0009] A task retention node is identified by performing execution progress tracking on the multi-node parallel execution sequence, a bottleneck resolution path graph is constructed according to the task retention node, and an equalized execution configuration is generated by performing task dynamic migration according to the bottleneck resolution path graph;

[0010] A node carrying topology matrix is generated by collecting cluster resource states according to the equalized execution configuration, an optimal distribution time slot is determined by performing low-load interval matching on the node carrying topology matrix, and a task equalization distribution instruction set is generated according to the optimal distribution time slot and the node carrying topology matrix.

[0011] The second aspect of the application provides a network scanning task intelligent segmentation and load balancing system, comprising:

[0012] A task acquisition module is configured to acquire a network scanning task request to be processed, perform task integrity verification on the network scanning task request to identify an abnormal task segment, and generate a segmentation reference sequence by performing task feature analysis on the abnormal task segment;

[0013] A scheduling establishment module is configured to generate a standard task queue by performing task unit reorganization processing according to the segmentation reference sequence, extract a node load identifier by performing node state synchronization analysis on the standard task queue, generate a mode adaptation instruction by triggering a scheduling mode adaptation mechanism according to the node load identifier, and establish a multi-node parallel scheduling link through the mode adaptation instruction.

[0014] A coordination balancing module is configured to generate a node balance index by performing load coordination analysis on the multi-node parallel scheduling link, perform multi-node coordination balancing processing according to the node balance index to form a coordination scheduling space, and generate a multi-node parallel execution sequence by implementing task fragmentation distribution on the coordination scheduling space.

[0015] A feedback correction module is configured to identify a task retention node by performing execution progress tracking on the multi-node parallel execution sequence, construct a bottleneck resolution path graph according to the task retention node, and generate an equalized execution configuration by performing task dynamic migration according to the bottleneck resolution path graph.

[0016] A distribution execution module is configured to generate a node carrying topology matrix by collecting cluster resource states according to the equalized execution configuration, determine an optimal distribution time slot by performing low-load interval matching on the node carrying topology matrix, and generate a task equalization distribution instruction set according to the optimal distribution time slot and the node carrying topology matrix.

[0017] The beneficial effects of the present application are embodied in the following points: first, by performing integrity check on the scan task request to identify abnormal fragments, constructing the dependency association structure within the task based on the abnormal fragments, using breakpoint detection to identify the dividable position and combining the granularity constraint to generate the division scheme, normalizing and packaging the divided task units to form a standard task queue, collecting the resource state of each node to calculate the comprehensive load index, classifying the nodes according to the load level and configuring the differentiated scheduling weight to establish the parallel scheduling channel, the task division granularity can adapt to nodes with different processing capabilities, avoiding the waste of parallel capability or the increase of scheduling overhead caused by improper granularity. Second, based on the load balancing score and the coordination coefficient of the nodes, the load difference between the nodes is calculated, the nodes with similar load and close coordination are divided into cooperative groups by clustering method, the connection nodes of the group boundary are identified and the cross-group communication cost is evaluated, the flow direction constraint rules are formulated according to the cost level, the cooperative scheduling view is formed by integrating each group and the flow rules, and the task is divided into standard fragments according to the node capacity to adapt and match, and the parallel execution plan is generated, realizing the cooperative and balanced distribution of task fragments among multiple nodes. Finally, by continuously comparing the execution progress to identify the stagnation nodes of task accumulation, collecting the state of the stagnation nodes to diagnose the bottleneck reason and labeling the type tag, analyzing the influence propagation range of the bottleneck along the dependency relationship and the cooperation relationship, planning the dredging path to the idle node according to the urgency and executing the migration, collecting the resource state time series data to identify the low-load period with stable idle rate as the preferred distribution opportunity, matching the task with the time slot and the node to generate the timing distribution instruction, realizing the dynamic elimination of the bottleneck and the optimization of the time and space distribution. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings herein show specific examples of the technical solutions described in the present application, and constitute part of the specification together with the specific embodiments, for explaining the technical solutions, principles and effects of the present application.

[0019] Figure 1 is a flow chart of a network scanning task intelligent segmentation and load balancing method of the present application.

[0020] Figure 2 is a structural block diagram of a network scanning task intelligent segmentation and load balancing system of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details are set forth such as particular system architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted in order to not obscure the description of the present application with unnecessary detail.

[0022] It should be understood that the word "comprising" when used in the specification and claims of this application indicates the existence of the stated features, integers, steps, operations, elements, or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] Reference throughout this specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specified. The terms "including," "containing," "having," and variations thereof are meant to encompass the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

[0024] The technical solutions of the embodiments of the application are described below.

[0025] As shown in Figure 1 The embodiment of the application provides a network scanning task intelligent segmentation and load balancing method, which comprises the following steps S110-S150:

[0026] S110, a network scanning task request to be processed is acquired, task integrity checking is performed on the network scanning task request to identify an abnormal task segment, and task feature analysis is performed according to the abnormal task segment to generate a segmentation reference sequence.

[0027] Specifically, a network scanning task request to be processed is acquired. The scanning task submitted by the upper scheduling system or the user terminal is listened to through a task receiving interface, the received task data is transmitted in a structured message form, and the message format adopts JSON or XML coding. The received task message is parsed in format, and field information such as a task identifier, a scanning target, a scanning type, scanning parameters and a priority in the message is extracted. The network scanning task request contains complete description information of the scanning task, the scanning target field specifies an IP address range or a domain name list to be scanned, the scanning type field specifies a specific scanning mode such as port scanning, vulnerability scanning or service identification, and the scanning parameter field specifies execution parameters such as a timeout time, a number of concurrent connections and a scanning depth. Syntax checking is performed on the network scanning task request, and whether the data type and the value range of each field conform to the specification definition is verified. The request that does not pass the syntax checking is marked as an invalid request and error information is returned. The network scanning task request that passes the syntax checking is assigned a unique task number, and the task number is generated by combining a time stamp and a random array to ensure global uniqueness. The network scanning task request and the task number are written into a task cache queue, and wait for subsequent integrity checking processing. A large-scale asset discovery task usually contains thousands of IP address segments, and the scanning target field of the network scanning task request specifies the address range in batches in the CIDR format. For a deep vulnerability scanning task for a specific host, the scanning target field of the network scanning task request contains only a small number of specific IP addresses, but the scanning parameter field contains detailed vulnerability detection rule configurations.

[0028] The task integrity check is performed on the network scanning task request to identify abnormal task fragments. The network scanning task request is read from the task cache queue, and the integrity of the request content is checked. The integrity check includes three levels of mandatory field check, field association check and task logic check. The mandatory field check verifies whether the core fields such as the scanning target and the scanning type of the network scanning task request exist and are not empty. The request with missing mandatory fields is determined to be incomplete. The field association check verifies whether the constraint relationship between the related fields in the network scanning task request is satisfied. When the scanning type is vulnerability scanning, the vulnerability database version field needs to be specified at the same time. The request with unsatisfied field association is determined to be abnormal in configuration. The task logic check verifies whether the execution logic of the request is reasonable. The ratio of the number of scanning targets and the concurrency parameter is within the system bearing range. The request with unreasonable logic is determined to be resource over limit. The various abnormalities found in the integrity check process are located and marked. The abnormal task fragments are extracted from the task content fragments corresponding to the abnormalities to form abnormal task fragments. The abnormal task fragments record the field position, abnormal type and abnormal description information of the abnormalities. A single network scanning task request can generate multiple abnormal task fragments. The abnormal task fragments are independent of each other but share the same task number. If the specified port range of the port scanning task exceeds the effective port number interval, the port range field is marked as an abnormal task fragment. If the access protocol of the target URL is not specified for the web application scanning task, the target address field is marked as an abnormal task fragment.

[0029] In some embodiments, the task feature analysis according to the abnormal task fragment generates a segmentation reference sequence, including: extracting a task dependency correlation graph according to the abnormal task fragment; performing breakpoint detection on the task dependency correlation graph to identify a set of divisible nodes; constructing a segmentation granularity constraint table based on the set of divisible nodes; and generating a segmentation reference sequence according to the segmentation granularity constraint table.

[0030] According to the abnormal task fragment, a task dependency association graph is extracted. The dependency relationship between the task field involved in the abnormal task fragment and other fields is analyzed, including data dependency, control dependency and resource dependency. Data dependency refers to that the value of a field depends on the output result of another field, control dependency refers to that the effective condition of a field is determined by the value of another field, and resource dependency refers to that multiple fields compete for the same system resource. According to the abnormal type of the abnormal task fragment, the range of dependency relationship to be analyzed is determined. The field missing type abnormality needs to analyze all downstream dependencies of the field, and the value out-of-range type abnormality needs to analyze the bidirectional dependencies of the field. All fields associated with the abnormal task fragment are traversed, and the dependency relationship between each field and other fields in the task is detected. The field pairs with the dependency relationship are recorded as dependency edges. A directed graph structure is constructed with the task field as the node and the dependency relationship as the edge, and a task dependency association graph is formed. The task dependency association graph is stored in the form of an adjacency list. The node attributes in the graph include the field name, the field type and the abnormal mark, and the edge attributes include the dependency type and the dependency strength. The dependency strength reflects the closeness of the dependency relationship between two fields. Strong dependency means that the two fields must be processed simultaneously, and weak dependency means that the two fields can be processed independently. In the vulnerability scanning task, the scanning target field and the vulnerability rule field have a data dependency relationship. When the abnormal task fragment points to the target field, the out-edge of the node in the task dependency association graph is connected to multiple downstream nodes such as rule matching and result storage. In the service identification task, the port list field and the protocol detection field have a control dependency relationship. The edge between the two nodes in the task dependency association graph is marked as a control dependency type.

[0031] The breakpoint detection is performed on the task dependency association graph to identify the set of splittable nodes. Each node in the task dependency association graph is traversed to analyze the in-degree, out-degree and dependency strength distribution of the node. The breakpoint is defined as an edge or node that, when cut, does not affect the integrity of other strongly connected components in the task dependency association graph. Splitting the task at the breakpoint position will not damage the independent execution capability of the task sub-unit. The strongly connected component decomposition is performed on the association graph to identify the node cluster that is strongly dependent on each other. The nodes within each strongly connected component are not splittable, and the connection edges between the strongly connected components are candidate breakpoints. The splittability of the candidate breakpoint is evaluated, and the evaluation indicators include the scale balance degree of the subgraph on both sides of the breakpoint, the dependency strength of the breakpoint edge and the abnormal association degree of the breakpoint position. The scale balance degree reflects whether the workload of each sub-task after splitting is balanced. The breakpoint with a dependency strength lower than the threshold has better splittability, and the breakpoint with a high abnormal association degree is preferentially used as a splitting position to isolate the abnormal influence range. The nodes corresponding to the breakpoints evaluated by the splittability are marked as splittable nodes, and all the splittable nodes are summarized to form the set of splittable nodes. The set of splittable nodes is stored in the form of a list, and the list elements include the node number, the node field name and the splittability score. In the batch IP scanning task, the task dependency association graph presents a multiple independent subgraph structure, and there is no strong dependency relationship between each IP address segment. The address segment boundary node enters the set of splittable nodes. In the composite task with scanning result association analysis, the association graph presents a chain structure, and there is a data dependency between each scanning stage. The splittability score of the stage boundary node is low, and only part of the nodes enter the set of splittable nodes.

[0032] A segmentation granularity constraint table is constructed based on the set of divisible nodes. For each divisible node in the set of divisible nodes, the granularity of the subtasks generated after segmentation at the node position is analyzed. The granularity of the subtasks is measured in three dimensions: estimated execution time, resource occupation, and data size. The estimated execution time is estimated according to the number of scan targets and the complexity of the scan type contained in the subtask, the resource occupation is estimated according to the concurrency parameters and memory requirements of the subtask, and the data size is estimated according to the input and output data volume of the subtask. The granularity indicators of the subtasks after segmentation of each node are calculated by traversing the set of divisible nodes, and the granularity indicators are compared with the preset granularity constraint range. The granularity constraint range specifies the minimum granularity and the maximum granularity of the subtasks, the minimum granularity ensures that the subtasks have the value of independent scheduling, and the maximum granularity ensures that the subtasks do not exceed the processing capacity of a single node. The divisible nodes, their granularity indicators, and constraint judgment results are organized in table form to form the segmentation granularity constraint table. The rows of the segmentation granularity constraint table correspond to each node in the set of divisible nodes, and the columns include node number, estimated execution time, resource occupation, data size, minimum granularity satisfaction flag, and maximum granularity satisfaction flag. The nodes that satisfy both the minimum granularity and the maximum granularity constraints are marked as valid segmentation points in the segmentation granularity constraint table, the nodes that satisfy only part of the constraints are marked as conditional segmentation points, and the nodes that do not satisfy any of the constraints are marked as invalid segmentation points. For an asset discovery task containing hundreds of C-class address segments, the segmentation granularity constraint table marks most of the address segment boundary nodes as valid segmentation points; while for a single target deep vulnerability scanning task, the conditional segmentation points account for a high proportion in the segmentation granularity constraint table.

[0033] The segmentation granularity constraint table is used to generate a segmentation reference sequence. Effective segmentation points and conditional segmentation points are selected from the segmentation granularity constraint table, and the segmentation points are sorted in descending order according to the divisibility score. When the number of effective segmentation points in the segmentation granularity constraint table is sufficient, the effective segmentation points are preferentially selected to construct a segmentation scheme; when the number of effective segmentation points is insufficient, the conditional segmentation points in the constraint table are combined and evaluated. The sorted segmentation points are selected greedily, and the node with the highest score and no conflict with the selected segmentation points is selected to join the segmentation scheme. Segmentation point conflict refers to the situation that two segmentation points are too close to each other, resulting in a subtask granularity lower than the minimum granularity constraint. The conflict detection is realized by calculating the task size between adjacent segmentation points. The conditional segmentation points are processed specially. When the conditional segmentation points combined with adjacent effective segmentation points can meet the granularity constraint, the conditional segmentation points are included in the segmentation scheme. After the segmentation point selection is completed, the segmentation points are arranged in the logical order in the original task to form an ordered segmentation reference sequence. The segmentation reference sequence is stored in the form of an ordered list, and the list elements include the segmentation point number, the segmentation position description, and the granularity estimation value of the subtask after segmentation. After the large-scale network segment scanning task is segmented, the segmentation reference sequence contains multiple network segment boundary segmentation points, and each subtask can be allocated to different scanning nodes for parallel execution. After the composite vulnerability scanning task is segmented, the segmentation reference sequence contains vulnerability type segmentation points, and each subtask is independently scheduled according to the vulnerability type.

[0034] In step S120, the segmentation reference sequence is used to carry out task unit recombination processing to generate a standard task queue. The node state synchronization analysis is performed on the standard task queue to extract the node load identifier. The node load identifier is used to trigger the scheduling mode adaptation mechanism to generate a mode adaptation instruction. The mode adaptation instruction is used to establish a multi-node parallel scheduling link.

[0035] Specifically, the task unit reorganization processing is carried out according to the segmentation reference sequence to generate a standard task queue. The original task is cut according to the position of each segmentation point in the segmentation reference sequence, and the task content is extracted between adjacent segmentation points to form an independent task unit. Each task unit is standardized and encapsulated, and the encapsulated content includes task unit number, parent task number, scanning target subset, scanning parameter copy and execution constraint condition. The task unit number is generated by combining the parent task number and the serial number, which ensures that the task unit can be traced back to the original task. The scanning target subset is the target range responsible for scanning by the task unit, the scanning parameter copy inherits the parameter configuration of the original task, and the execution constraint condition records the dependency relationship and execution order requirement of the task unit with other units. The encapsulated task unit is subjected to format checking to ensure that each field is complete and conforms to the standard definition. The task units that pass the checking are sequentially added to the queue according to the order of the segmentation reference sequence to form a standard task queue. The standard task queue is stored in a first-in first-out queue structure, and each task unit in the queue is independent and has the ability to be independently scheduled and executed. Each subnet segment scanning unit formed by segmentation of the asset discovery task enters the standard task queue, and the queue order reflects the address order of the subnet segment; each type of scanning unit formed by segmentation of the vulnerability scanning task enters the standard task queue, and the queue order reflects the logical order of vulnerability detection.

[0036] The node state synchronization analysis extracts the node load identifier. The total number of task units in the specification task queue and the resource demand are analyzed, and the range of scanning execution nodes that need to participate in scheduling is determined according to the size of the task queue. The list of currently available scanning execution nodes is obtained, and a state query request is sent to each node, including CPU utilization, memory occupancy, network bandwidth usage and current task queue depth. Each node receives the request, collects local resource usage, and returns the collected data as a state response message. The collected node state responses are analyzed, and the resource utilization indicators are extracted and normalized. The comprehensive load index of the node is calculated by integrating the resource utilization of each dimension, L=W1×Lcpu+W2×Lmem+W3×Lnet+W4×Lqueue, where Lcpu is the CPU utilization, Lmem is the memory occupancy, Lnet is the network bandwidth usage, Lqueue is the task queue depth load rate (the ratio of the current queue task number to the maximum queue capacity), and W1, W2, W3, W4 are weight coefficients. The total workload of the specification task queue is matched and analyzed with the remaining capacity of each node, and the ability of each node to undertake task units in the task queue is evaluated. The comprehensive load index of each node is encoded as a node load identifier, which is represented in the form of a combination of node number and load level. The load level is divided into five levels according to the comprehensive load index: idle, light load, medium load, heavy load and full load. The specification task queue containing 200 task units needs to match a sufficient number of idle or light load nodes. The node load identifier shows that there are 8 idle nodes and 5 light load nodes in the cluster, which have the capacity to undertake all task units.

[0037] In some embodiments, the generation of mode adaptation instructions by the scheduling mode adaptation mechanism triggered according to the node load identifier includes: performing load gradient analysis on the node load identifier to generate a node classification list; performing load level division according to the node classification list to generate a high load level and a low load level; using the low load level to perform scheduling weight proportioning on the high load level to form a differentiated scheduling configuration; and generating mode adaptation instructions according to the differentiated scheduling configuration.

[0038] The node load identifier is subjected to load gradient analysis to generate a node ranking list. The node load identifiers of all scan execution nodes are collected, and the load grades and comprehensive load indexes are extracted from the identifiers. The comprehensive load indexes of the nodes are sorted according to the numerical values, and the load index difference between adjacent nodes is calculated. The difference sequence reflects the load gradient distribution of the node group. The statistical characteristics of the load gradient distribution are analyzed, and the mean and variance of the difference sequence are calculated. The gradient mean reflects the average level of the load difference between the nodes, and the gradient variance reflects the balance degree of the load distribution. The nodes are ranked according to the comprehensive load indexes in the node load identifiers and the load gradient. The ranking rule comprehensively considers the absolute load level and the relative load position of the nodes. The nodes with low absolute load level and in the gradient trough are evaluated as high priority, and the nodes with high absolute load level and in the gradient peak are evaluated as low priority. The ranking results of the nodes are organized in the form of a list to form a node ranking list. The node ranking list takes the node number as the index and records the load grade, comprehensive load index, gradient position and priority score of each node. Some nodes in the scan cluster are executing large-scale port scanning tasks and are in a heavy load state. The node load identifier shows that the load grade of these nodes is heavy load, and these nodes obtain a lower priority score in the node ranking list.

[0039] According to the node ranking list, load level division is performed to generate a high load level and a low load level. The priority score distribution of each node in the node ranking list is analyzed, and a clustering method is used to divide the nodes into different load levels. The node set in the node ranking list is subjected to K-means clustering with the priority score as the feature, and the number of clusters is set to 2. The nodes are divided into two clusters. The cluster with a higher score mean corresponds to a node group with lower load, which is marked as a low load level; and the cluster with a lower score mean corresponds to a node group with higher load, which is marked as a high load level. The boundary of the clustering result is optimized. When a node in the node ranking list is close to the centers of the two clusters, the load change trend of the node is determined to decide its attribution. The node with an upward load trend is attributed to the high load level, and the node with a downward load trend is attributed to the low load level. The number and total capacity of the nodes included in the high load level and the low load level are counted. The nodes in the low load level are the priority targets for task scheduling, and the nodes in the high load level are the alternative targets or objects that need to be subjected to task migration. When the load distribution of the scan cluster is seriously uneven, the high load level includes a small number of overloaded nodes, and the low load level includes a large number of idle nodes. The task needs to be transferred from the high load level to the low load level through a scheduling strategy.

[0040] The low-load level is used to schedule the weight ratio of the high-load level to form a differentiated scheduling configuration. The total available capacity of the low-load level is calculated, and the available capacity is the sum of the residual processing capacity of each node in the level. The total overload of the high-load level is calculated, and the overload is the sum of the task amount exceeding the ideal load level of each node in the level. According to the ratio of the available capacity of the low-load level to the overload of the high-load level, the feasibility of load balancing is determined. If the ratio is greater than 1, it means that the low-load level has enough capacity to undertake the overload task. The scheduling weight of each node in the low-load level is allocated, and the scheduling weight is proportional to the residual processing capacity of the node. The greater the residual capacity, the higher the task allocation weight of the node. The task migration ratio of each node in the high-load level is set, and the migration ratio is proportional to the overload degree of the node. The more serious the overload, the more tasks are migrated out. The scheduling weight, migration ratio and target load level of each node are organized into a configuration record to form a differentiated scheduling configuration. The differentiated scheduling configuration is stored in the form of a configuration table, and each row in the table corresponds to each node. The columns include node number, belonging level, scheduling weight, migration ratio and target load index. The low-load level of a certain scanning cluster contains 6 idle nodes, and the high-load level contains 2 overloaded nodes. The differentiated scheduling configuration allocates a higher receiving weight to the idle nodes and sets a 30% task migration ratio for the overloaded nodes.

[0041] Mode adaptation instructions are generated according to the differentiated scheduling configuration. The configuration parameters of each node in the differentiated scheduling configuration are analyzed, and corresponding scheduling control instructions are generated according to the parameters. Task receiving instructions are generated for the nodes of the low-load level, and the instruction content includes node address, receiving weight and maximum receiving task number. Task migration instructions are generated for the nodes of the high-load level, and the instruction content includes node address, migration ratio and migration task selection rule. Load monitoring instructions are generated for all nodes, and the instruction content includes monitoring period, reporting threshold and alarm condition. Various instructions are organized into an instruction set according to node number and instruction type, and mode adaptation instructions are formed according to the complete content of the differentiated scheduling configuration. The mode adaptation instructions are stored in the form of an instruction queue, and the queue elements include instruction type, target node, instruction parameters and execution priority. A node in the vulnerability scanning cluster is overloaded due to the execution of a high-risk vulnerability deep scan, and the differentiated scheduling configuration marks the node as high-load. The generated mode adaptation instructions guide the node to pause receiving new tasks and migrate part of the to-be-executed tasks to idle nodes.

[0042] The multi-node parallel scheduling link is established via mode adaptation instructions. The mode adaptation instructions are issued to each scanning execution node, and the node updates the local scheduling configuration parameters after receiving the instructions. The task receiving channel is opened for the low-load level node, and the task receiving rate is controlled according to the receiving weight in the instruction. The task migration out process is started for the high-load level node, and the tasks are sent from the local queue to the receiving node according to the migration out rule in the instruction. The communication link between the scheduler and each node is established, the communication link uses a long connection mode to maintain a session state, and supports task distribution, state synchronization and instruction issuing. The task migration link is established between the nodes, and the migration link supports direct transmission of task data. The communication link between the scheduler and each node and the migration link between the nodes are uniformly managed to form a multi-node parallel scheduling link. After the multi-node parallel scheduling link is established, the task units in the task queue are distributed to each scanning node for parallel execution through the scheduling link. Hundreds of subnetwork segment scanning units of large-scale asset discovery tasks are simultaneously distributed to multiple scanning nodes through the multi-node parallel scheduling link, each node executes the scanning task in parallel, and the overall scanning time is greatly shortened.

[0043] In step S130, the load coordination analysis is performed on the multi-node parallel scheduling link to generate a node balance index, the multi-node coordination balance processing is performed according to the node balance index to form a coordination scheduling space, the task fragmentation distribution is implemented on the coordination scheduling space to generate a multi-node parallel execution sequence.

[0044] Specifically, the load coordination relationship of the multi-node parallel scheduling link is analyzed, and the coordination relationship strength is quantified by counting the task migration frequency and migration success rate between nodes, and the coordination coefficient C = a x Fmig + b x Rsuc, wherein Fmig is the normalized task migration frequency, Rsuc is the task migration success rate, a and b are weight coefficients and a + b = 1. The deviation degree of the real-time load level and the average load of the cluster is integrated to calculate the balance state score of each node, and the balance state score S = 1 - | Li - Lavg |, wherein Li is the comprehensive load index of node i, Lavg is the average value of the comprehensive load index of all nodes in the cluster, the balance state score ranges from 0 to 1, and the higher the node load is close to the average level of the cluster, the higher the score is. According to the historical load trend, the load change direction of each node is determined, and when the recent load shows an upward trend, it is marked as rising, when it shows a downward trend, it is marked as falling, and when it remains stable, it is marked as stable. The balance state score, load trend direction and coordination relationship coefficient of each node are packaged into a node balance index. The node balance index is stored in a structured record form, each node corresponds to a record, and the record fields include node number, balance score, trend mark and coordination coefficient. During the execution of the port scanning task, some nodes in the multi-node parallel scheduling link are accumulated due to slow response of the scanning target, and the balance score of these nodes in the node balance index is low and the trend mark shows that the load is rising.

[0045] In some embodiments, the multi-node coordination balancing processing according to the node balance index forms a cooperative scheduling space, including: constructing a node load difference matrix according to the node balance index; performing clustering division on the node load difference matrix to generate node cooperation groups; establishing inter-group task flow transfer rules for the node cooperation groups; and integrating the groups according to the inter-group task flow transfer rules to form a cooperative scheduling space.

[0046] According to the node balance index, a node load difference matrix is constructed. The node balance index records of each scanning execution node are read, and the balance score and the synergy coefficient are extracted from the node balance index. The load difference degree of any two nodes is calculated, and the load difference degree is defined as the weighted combination of the absolute value of the difference between the balance scores of the two nodes and the synergy coefficient. The difference degree D = |Si-Sj| x (1-Cij), where Si and Sj are the balance scores of the two nodes in the node balance index, and Cij is the synergy coefficient between the two nodes. The higher the synergy coefficient, the smaller the difference degree. A square matrix is constructed with the node number as the row and column index and the load difference degree as the element value, forming a node load difference matrix. The node load difference matrix is a symmetric matrix with zero diagonal elements. The smaller the matrix element value, the closer the load state of the corresponding two nodes and the tighter the synergy relationship. The element value is mapped to the interval of 0 to 1 after normalization processing, which is convenient for subsequent clustering analysis. The overall distribution characteristics of the difference matrix are analyzed, and the mean and variance of the matrix elements are calculated. The mean reflects the average load difference level of the cluster, and the variance reflects the dispersion degree of the load distribution. The nodes in the vulnerability scanning cluster that execute the same type of scanning tasks have a higher synergy coefficient, and the corresponding element value in the node load difference matrix is smaller. The nodes that execute different types of scanning tasks have a lower synergy coefficient, and the corresponding element value in the difference matrix is larger.

[0047] The node load difference matrix is executed to generate node cooperation groups. The node set is subjected to hierarchical clustering analysis with the node load difference matrix as the distance measure. The hierarchical clustering adopts the agglomerative method, and each node forms a class in the initial state. In the iteration process, the two classes with the closest distance in the difference matrix are merged into a new class, and the inter-class average distance method is used for distance calculation. The termination condition of clustering is set as the maximum distance within the class being not more than a preset threshold, and the threshold is determined according to the cluster size and management granularity requirements. After clustering, a plurality of node clusters are formed, and the nodes in each cluster have a smaller difference degree in the node load difference matrix. Each cluster corresponds to a node cooperation group. The node cooperation groups are labeled with attributes, including group number, member node list, group average load, and group total capacity. The group average load is the mean of the balance scores of the nodes in the group, and the group total capacity is the sum of the available processing capacities of the nodes in the group. The node cooperation groups are sorted according to the group average load. The group with lower load has more task carrying space, and the group with higher load needs to migrate tasks outward. In the web application scanning cluster, the nodes that execute static page crawling are grouped into one node cooperation group, and the nodes that execute dynamic interface testing are grouped into another node cooperation group. The task types within each group are similar, which is convenient for collaborative scheduling.

[0048] Exemplarily, the node cooperation group establishes an inter-group task flow transfer rule, including: performing boundary node identification on the node cooperation group to obtain an inter-group connection node set; analyzing cross-group communication cost according to the inter-group connection node set to generate a cost evaluation value; setting a task flow transfer direction constraint according to the cost evaluation value, wherein when the cost evaluation value is lower than a preset cost threshold, bidirectional flow transfer is allowed; when the cost evaluation value is higher than the preset cost threshold, unidirectional flow transfer is limited; and establishing an inter-group task flow transfer rule according to the task flow transfer direction constraint.

[0049] The node cooperation group performs boundary node identification to obtain an inter-group connection node set. The connection relationship between the member nodes of each node cooperation group and the members of other groups is analyzed. The boundary node is defined as a node that has a direct communication link with other node cooperation groups, and the boundary node undertakes the transfer responsibility of cross-group task flow. The adjacent nodes of each node in the multi-node parallel scheduling link are detected, and when the adjacent nodes of a certain node contain members of other node cooperation groups, the node is marked as a boundary node. The boundary nodes identified by each node cooperation group are summarized, and the target group number connected by the boundary nodes is recorded to form an inter-group connection node set. The inter-group connection node set is organized around the boundary nodes, each boundary node records its own group, the target group list connected, and the link bandwidth with each target group. When there are multiple boundary nodes between two node cooperation groups, all boundary nodes and their respective link attributes are recorded in the inter-group connection node set. In a distributed scanning cluster, nodes in the same machine room are usually grouped into a node cooperation group, and the nodes at both ends of the high-speed link across the machine room become members of the inter-group connection node set and undertake the task scheduling transfer of the machine room.

[0050] The cost evaluation value is generated according to inter-group connection node set analysis of cross-group communication cost. For each boundary node in the inter-group connection node set, the resource consumption of cross-group communication is analyzed. The cross-group communication cost is composed of three parts: transmission delay, bandwidth occupation and processing overhead. The transmission delay is the network round-trip time of task data from the source group to the target group, and the actual delay value is measured by sending a probe packet to the target group. The bandwidth occupation is the consumption proportion of cross-group task transmission to link bandwidth, which is calculated according to the task data volume and the total link bandwidth. The processing overhead is the CPU time consumed by the boundary node for task forwarding, including data unpacking, routing decision and repackaging. For the cross-group communication between any two node collaboration groups, the boundary nodes connecting the two groups in the inter-group connection node set are selected, and the cost evaluation value of the group pair is obtained by comprehensively calculating the transmission delay, bandwidth occupation and processing overhead. The cost evaluation value is represented in numerical form, and the larger the value is, the higher the cross-group communication cost is. The cost evaluation values of all group pairs are organized in the form of a cost matrix, and the matrix row and column correspond to the group number, and the element is the communication cost of the corresponding group pair. The cost evaluation values between different node collaboration groups in the same machine room are low, and the cost evaluation values between groups across machine rooms or regions are high.

[0051] The task flow direction constraint is set according to the cost evaluation value. The cost evaluation values of each group pair in the cost matrix are read and compared with the preset cost threshold. The preset cost threshold is determined according to the overall communication capability of the cluster and the task real-time requirement. If the threshold is too low, it will limit too much cross-group flow to cause the load balancing effect to decline, and if the threshold is too high, it will allow high-cost flow to cause waste of communication resources. For the group pair with a cost evaluation value lower than the threshold, the bidirectional flow permission is set, which means that the task can flow from any group to another group, and the flow direction is dynamically determined according to the real-time load difference. For the group pair with a cost evaluation value higher than the threshold, the unidirectional flow permission is set, and the unidirectional flow direction points from the high-load group to the low-load group, and the reverse flow is prohibited to avoid unnecessary high-cost transmission. For the group pair with a cost evaluation value far exceeding the threshold, the flow is prohibited, and direct task migration is not performed between the two groups, and if necessary, the task is transferred through an intermediate group. The flow permission of each group pair is recorded as the task flow direction constraint, and the content of the task flow direction constraint includes the source group, the target group, the flow direction and the flow condition. The cost evaluation value between two node collaboration groups connected by high-bandwidth interconnection is low, and the task flow direction constraint is set to allow bidirectional flow; the cost evaluation value between two groups connected by a wide area network is high, and the task flow direction constraint is set to allow unidirectional flow only from the heavy-load group to the light-load group.

[0052] The inter-group task flow transfer rules are established according to the task flow transfer direction constraints. The task flow transfer direction constraints of all group pairs are summarized to construct a complete cross-group task flow transfer rule system. For the group pairs allowing bidirectional transfer in the task flow transfer direction constraints, symmetric rules are established, and the rule triggering condition is that the load difference between the two groups exceeds the balance threshold, the transfer direction is from the high-load group to the low-load group, and the transfer amount is proportional to the load difference. For the group pairs limiting unidirectional transfer in the task flow transfer direction constraints, asymmetric rules are established, and the rule is only triggered when the high-load group is the source group, and the transfer amount is determined according to the overload degree of the source group and the remaining capacity of the target group. For the group pairs prohibited from transferring in the task flow transfer direction constraints, no direct rules are established, but feasible transit paths are recorded for emergency use. Priorities are set for each transfer rule, and the priority of the low-cost transfer rule is higher than that of the high-cost rule, and the priority of the intra-group transfer rule is higher than that of the cross-group rule. All transfer rules are organized in the form of a rule library to form inter-group task flow transfer rules. The inter-group task flow transfer rules are stored in the form of a rule entry list, and each entry contains the rule number, the source group, the target group, the triggering condition, the transfer amount calculation formula, and the priority. In the vulnerability scanning cluster, the inter-group task flow transfer rules between the low-risk vulnerability scanning group and the high-risk vulnerability scanning group are set to trigger transfer when the load difference exceeds 20%, and the transfer amount is half of the difference.

[0053] The collaborative scheduling space is formed by integrating groups according to the inter-group task flow transfer rules. The logical connections between the node collaboration groups are established based on the inter-group task flow transfer rules, and the logical connections reflect the task flow transfer accessibility and transfer cost between groups. The grouping association graph is constructed by taking the node collaboration groups as nodes and the transfer rules in the inter-group task flow transfer rules as edges, the weight of the edge in the graph is the cost evaluation value of the corresponding rule, and the direction attribute of the edge identifies the transfer direction constraint. The connectivity of the grouping association graph is analyzed to ensure that there is a direct or indirect task flow transfer path between any two groups, and if there is an isolated group, the inter-group task flow transfer rules are adjusted to establish a connection. The capacity information and load state information of each group are superimposed on the basis of the grouping association graph to form a comprehensive scheduling view containing static topology and dynamic state. The grouping association graph, the member node list of each group, the group capacity, the group load, and the inter-group task flow transfer rules are uniformly encapsulated to form a collaborative scheduling space. The collaborative scheduling space is stored in a composite data structure, which supports querying the group to which any node belongs, the transferable target of any group, and the transfer cost of any group pair. The collaborative scheduling space of the large-scale scanning cluster contains multiple node collaboration groups, and each group is connected through the inter-group task flow transfer rules. The scheduler makes global task allocation decisions based on the collaborative scheduling space.

[0054] In some embodiments, the task partition distribution on the cooperative scheduling space generates a multi-node parallel execution sequence, including: identifying the task capacity that each node can carry according to the cooperative scheduling space; determining the partition granularity to generate a task partition set according to the task capacity that each node can carry; performing node mapping binding on the task partition set to generate a partition node correspondence table; and generating a multi-node parallel execution sequence according to the partition node correspondence table.

[0055] The task capacity that each node can carry is identified according to the cooperative scheduling space. The cooperative group of each node and the list of member nodes thereof are obtained from the cooperative scheduling space, and the current load state and hardware configuration parameters of each node are queried. The hardware configuration parameters include the number of CPU cores, the memory capacity, the network bandwidth, and the storage space. The theoretical maximum processing capacity of the node is calculated according to the hardware configuration, and the maximum processing capacity is measured by the number of scanning targets that can be processed per unit time. The current load state of the node recorded in the cooperative scheduling space is queried, and the remaining processing capacity of the node is calculated in combination with the load state, and the remaining processing capacity is equal to the maximum processing capacity minus the current occupied capacity. Considering the safety margin of task scheduling, the remaining processing capacity is multiplied by a safety factor to obtain the task capacity that each node can carry, and the safety factor is usually set to 0.8 to reserve space for responding to sudden loads. The task capacity that each node can carry is recorded to a capacity table, and the capacity table takes the node number as the index, and the fields include the node number, the group to which the node belongs, the maximum capacity, the current load, and the task capacity that the node can carry. The task capacity that the member nodes in each cooperative group in the cooperative scheduling space can carry is summarized to obtain a group-level capacity, which is used as a reference for cross-group task allocation. High-performance scanning nodes are configured with multi-core CPUs and large-capacity memories, and the task capacity that they can carry is large and can undertake large-scale scanning tasks; lightweight scanning nodes are configured with limited capacity, and the task capacity that they can carry is small and are suitable for undertaking small-scale or low-complexity tasks.

[0056] For example, the task partition set is generated according to the task capacity that each node can carry, including: identifying the scanning granularity based on the task capacity that each node can carry to generate coarse-grained scanning blocks and fine-grained scanning blocks; cutting the boundaries of the coarse-grained scanning blocks using the fine-grained scanning blocks to form standard scanning partitions; capacity-adapting the standard scanning partitions according to the task capacity that each node can carry to generate node-matching partitions; and sequencing the node-matching partitions to build a task partition set.

[0057] The coarse-grained and fine-grained scanning blocks are generated based on the scanning granularity recognition of the bearable task capacity. The distribution characteristics of the bearable task capacity of each node in the capacity table are counted, and the maximum, minimum and median of the capacity are calculated. The division standards of coarse-grained and fine-grained are determined according to the distribution of the bearable task capacity. The coarse-grained corresponds to the carrying capacity of the large-capacity node, and the fine-grained corresponds to the carrying capacity of the small-capacity node. The task units to be distributed in the standard task queue are read, and the workload size of each task unit is analyzed. The workload size is comprehensively evaluated according to the number of scanning targets, the complexity of scanning type and the estimated execution time. The task units with workload size exceeding the coarse-grained threshold are marked as coarse-grained scanning blocks, the task units with workload size lower than the fine-grained threshold are marked as fine-grained scanning blocks, and the task units with workload size between the two are kept as they are for subsequent processing. The coarse-grained scanning blocks need to be further divided to adapt to the bearable task capacity of each node, and the fine-grained scanning blocks can be directly allocated to small-capacity nodes or combined with other tasks to be allocated to large-capacity nodes. Large-scale network segment scanning tasks usually form coarse-grained scanning blocks, and single-host deep scanning tasks usually form fine-grained scanning blocks.

[0058] The coarse-grained scanning blocks are cut by the fine-grained scanning blocks to form standard scanning fragments. The average workload size of the fine-grained scanning block is analyzed and used as the reference unit for fragmentation. Each coarse-grained scanning block is divided according to the reference unit, and the cutting method is selected according to the scanning task type. Network segment scanning tasks are divided according to IP address range, and large network segments in coarse-grained scanning blocks are divided into multiple small network segments. Port scanning tasks are divided according to port range, and full port scanning is divided into multiple port segment scanning. Vulnerability scanning tasks are divided according to vulnerability type, and comprehensive vulnerability scanning is divided into multiple single-type vulnerability scanning. During the cutting process, the workload of each fragment is ensured to be close to the reference unit of the fine-grained scanning block, and a certain range of deviation is allowed to maintain the integrity of the task logic. The task fragments generated by cutting are standardized and packaged, and the packaging content includes fragment number, parent task number, scanning range and estimated workload. The task fragments after standardized packaging are standard scanning fragments. Standard scanning fragments have similar workload size, which is convenient for balanced allocation among different capacity nodes. The original fine-grained scanning blocks are also packaged into the standard scanning fragment format and managed uniformly with the fragments generated by cutting.

[0059] The standard scan fragments are capacity adapted according to the bearable task capacity to generate node matching fragments. The number of bearable standard scan fragments is planned according to the bearable task capacity of each node in the capacity table, and the number of fragments is calculated by dividing the bearable task capacity of the node by the estimated workload of a single standard scan fragment. The calculation result is rounded down to ensure that it does not exceed the node carrying capacity. The node adaptation evaluation is performed on each standard scan fragment, and the evaluation indexes include the matching degree of the fragment workload and the node bearable task capacity, and the matching degree of the scan type of the fragment and the node expertise. Fragments with high matching degree are preferentially assigned to corresponding nodes, and fragments with high matching degree can fully exert the processing advantage of the node. The standard scan fragments that have completed the adaptation evaluation are marked with the recommended assignment node, and the marked fragments are called node matching fragments. The node matching fragments add the recommended node number and matching score fields to the standard scan fragments. The scan nodes deployed with a dedicated vulnerability library have high matching degree with the node matching fragments of the vulnerability scan type, and the nodes with sufficient network bandwidth have high matching degree with the node matching fragments of the large-flow port scan type.

[0060] The node matching fragments are sequenced to construct a task fragment set. The fragments are grouped according to the recommended node number of the node matching fragments, and the recommended fragments of the same node are grouped into the same group. The node matching fragments in each group are sorted according to the execution priority, and the priority is determined by the parent task priority and the fragment dependency relationship. Fragments without pre-dependence and with high parent task priority are placed in the front row. The node matching fragments between different groups are globally sorted, and the sorting basis is the current load state of the recommended node. The recommended fragments of the low-load node are preferentially entered into the execution queue to balance the cluster load as soon as possible. All the sorted node matching fragments are organized into an ordered list to form a task fragment set. The task fragment set is stored in a priority queue structure, and the queue elements include fragment number, recommended node, execution priority and parent task information. The fragments of the urgent security scan task are sorted in the front of the task fragment set and are preferentially executed, and the fragments of the regular inspection scan task are sorted in the back and are executed when the resources are idle.

[0061] The node mapping binding generates a slice node correspondence table. The nodes in the task slice set are processed in priority order, the slice is matched, and the final binding relationship between the slice and the recommended node is confirmed. Check if the real-time loadable task capacity of the recommended node still meets the slice demand. If the recommended node capacity is sufficient, the binding relationship is confirmed. If the recommended node capacity is insufficient, the reallocation process is triggered. The reallocation process queries other nodes in the same group as the recommended node from the collaborative scheduling space, selects a node with sufficient capacity and a suboptimal matching score as the replacement allocation target. All slices in the task slice set are traversed to complete binding confirmation, and the final allocation node of the confirmed slice is recorded. The correspondence between the slice number and the node number is written into the slice node correspondence table. The slice node correspondence table takes the slice number as the primary key, and the fields include the slice number, the allocation node, the allocation time, and the estimated completion time. The estimated completion time is estimated according to the slice workload and the node processing rate for reference of task progress monitoring. At the same time, a node index view of the slice node correspondence table is established to support quick query of all slices undertaken by a node according to the node number. The task slice set of a certain scanning cluster contains 150 slices, and the slice node correspondence table records the correspondence between each slice and 12 scanning nodes after node mapping binding. Each node undertakes about 12 slices on average.

[0062] A multi-node parallel execution sequence is generated according to the slice node correspondence table. The slice records in the slice node correspondence table are grouped and summarized according to the allocation node, and all the slices undertaken by each node are summarized. The slices of each node are sorted according to the execution priority to form a node-level execution queue. The execution queues of each node are arranged in parallel to form a multi-column parallel execution plan view. The execution plan view is time-aligned to identify slices that can be started simultaneously in the execution queue of each node in the slice node correspondence table, and the simultaneously started slices are marked as the same execution batch. All slices are arranged according to the execution batch, and the slices in the same batch are executed in parallel, and different batches are executed in sequence. The slice sequence arranged in batches is organized into a multi-node parallel execution sequence. The multi-node parallel execution sequence is stored in a two-dimensional structure, with rows corresponding to execution batches and columns corresponding to execution nodes, and elements being slice numbers executed by the node in the batch. The multi-node parallel execution sequence supports batch task delivery by the scheduler, and each batch is delivered after waiting for the nodes to complete the previous batch before delivering the next batch, realizing collaborative parallel execution of multiple nodes. After large-scale asset discovery tasks are sliced, the 150 slices mapped by the slice node correspondence table form a multi-node parallel execution sequence, containing 13 execution batches, and 12 nodes in each batch simultaneously scan different network segments, and the batches are sequentially advanced until all scanning targets are covered.

[0063] Step S140, perform execution progress tracking on the multi-node parallel execution sequence to identify task retention nodes, construct a bottleneck resolution path diagram according to the task retention nodes, and perform task dynamic migration according to the bottleneck resolution path diagram to generate an equalized execution configuration.

[0064] Specifically, the execution progress of the multi-node parallel execution sequence is tracked to identify task retention nodes. The multi-node parallel execution sequence records the tasks assumed by each node in each execution batch and the estimated completion time. An execution progress monitoring mechanism is established, and a progress query request is periodically sent to each scanning execution node. The query period is set according to the timeliness requirement of the task. Each node returns the currently executing shard number, the number of completed shards, and the remaining depth of the task queue upon receiving the query request. The actual execution progress of each node is compared with the estimated progress in the multi-node parallel execution sequence, and a progress deviation rate is calculated. The progress deviation rate is defined as the ratio of the difference between the actual completion time and the estimated completion time in the multi-node parallel execution sequence to the estimated execution time. A positive value indicates progress lag, and a negative value indicates progress advance. A lag alarm threshold is set, and if the progress deviation rate of a node exceeds the threshold, it is determined that the node has a task retention phenomenon. The nodes determined to have retention are marked, and their node numbers, retention shard lists, and lag durations are recorded to form a task retention node set. The nodes in the task retention node set are arranged in descending order of lag duration, and the nodes with more serious lag are arranged at the front.

[0065] In some embodiments, constructing a bottleneck resolution path diagram according to the task retention nodes comprises: performing retention cause diagnosis on the task retention nodes to identify bottleneck type labels; performing bottleneck influence domain expansion analysis according to the bottleneck type labels to generate a bottleneck influence range; performing resolution priority sorting on the bottleneck influence range to determine a resolution starting anchor point; and constructing a bottleneck resolution path diagram according to the resolution starting anchor point to plan a relief path layer by layer.

[0066] The task retention node performs retention reason diagnosis to identify the bottleneck type label. The detailed resource usage state and task execution log of each retention node in the task retention node set are collected. The resource usage state includes CPU core utilization, memory usage and swap area occupation, network transmission rate and packet loss rate, disk read / write rate and queue depth. The task execution log records the start time of each shard, the time consumption of the key stage and the abnormal events. Based on the resource state and the execution log, the bottleneck reason of the task retention node is diagnosed, and the diagnosis rules cover five types of bottlenecks, including computing bottleneck, memory bottleneck, network bottleneck, storage bottleneck and external dependency bottleneck. When the CPU utilization rate is continuously higher than the threshold value and the task is in the calculation intensive stage, it is diagnosed as a computing bottleneck. When the memory usage rate is close to the upper limit and the swap is frequently triggered, it is diagnosed as a memory bottleneck. When the network packet loss rate increases or the bandwidth is full, it is diagnosed as a network bottleneck. When the disk queue depth continuously increases, it is diagnosed as a storage bottleneck. When the task log shows that the waiting time for the response of the external service is too long, it is diagnosed as an external dependency bottleneck. The bottleneck type label is labeled for each task retention node, and the bottleneck type label is formed. The bottleneck type label is stored in the form of key-value pair, the key is the node number, and the value is the bottleneck type code and confidence score. The nodes performing large-scale network segment survival detection are often labeled with network bottleneck in the bottleneck type label, and the nodes performing deep vulnerability verification are often labeled with computing bottleneck in the bottleneck type label.

[0067] According to the bottleneck type label, the bottleneck influence domain expansion analysis is performed to generate the bottleneck influence range. The bottleneck type label of each task retention node is read, and the potential influence of the bottleneck type label on the surrounding nodes and associated tasks is analyzed. The influence propagation path of the bottleneck is calculated, and the propagation path extends along the task dependency relationship and the node cooperation relationship. When the bottleneck type label shows that a node has a network bottleneck, other nodes sharing the network outlet with the node may be affected by bandwidth competition, and these nodes are included in the influence domain. When a node has a task retention, the subsequent tasks that depend on the output results of the node will be blocked, and the execution nodes of these tasks are included in the influence domain. The influence domain of each task retention node is expanded and searched, and the search depth is dynamically adjusted according to the severity of the bottleneck recorded in the bottleneck type label. The search depth of the severe bottleneck is deeper. All affected nodes and tasks searched are summarized, and their affected degree and influence path are labeled to form the bottleneck influence range. The bottleneck influence range is stored in a graph structure, in which the nodes represent the affected scanning nodes or tasks, and the edges represent the influence propagation path. The node attributes include the influence degree score and the number of hops from the bottleneck source. In a distributed vulnerability scanning cluster, when the aggregation node has a storage bottleneck, all working nodes that report the scanning results to the aggregation node enter the bottleneck influence range, forming a fan-shaped influence domain centered on the aggregation node.

[0068] The bottleneck influence range is prioritized to determine the resolution starting anchor point. The urgency of each affected node and task in the bottleneck influence range is analyzed, and the urgency is comprehensively evaluated by task priority, lag time, and influence range. The node with high task priority has high resolution urgency, the node with long lag time has high resolution urgency, and the node with wide influence range has high resolution urgency. The comprehensive urgency score of each affected node in the bottleneck influence range is calculated, and the scoring formula is U = W1 × P + W2 × T / Tmax + W3 × N / Nmax, where P is the normalized value of task priority, T is the lag time, Tmax is the maximum allowed lag time, N is the number of affected downstream nodes, Nmax is the total number of nodes in the cluster, W1, W2, and W3 are weight coefficients and sum to 1. The nodes in the bottleneck influence range are arranged in descending order according to the comprehensive urgency score, and the node with the highest score is the priority processing object of the resolution work. Select the top-ranked nodes from the ranking results as the resolution starting anchor points, and the number of selected nodes is determined according to the mobilizable resolution resources. The resolution starting anchor point is the entry position of the bottleneck resolution work, and the maximum resolution benefit can be obtained by starting task migration or resource allocation from the resolution starting anchor point. When there are multiple independent influence sub-domains in the bottleneck influence range, resolution starting anchor points are selected in each sub-domain to ensure that the bottlenecks in each sub-domain are processed in parallel. The stranded node where the emergency safety scanning task is located is usually selected as the resolution starting anchor point, and its execution progress is prioritized to be restored.

[0069] The bottleneck resolution path graph is constructed by planning the relief path layer by layer according to the resolution starting anchor point. The target node that can undertake the migrated task is searched in the collaborative scheduling space starting from each resolution starting anchor point. The target node needs to meet three conditions: sufficient capacity, complementary bottleneck type, and acceptable communication cost. Sufficient capacity means that the target node can accommodate the task to be migrated, complementary bottleneck type means that the target node does not have the same type of bottleneck as the resolution starting anchor point, and acceptable communication cost means that the task migration cost between the resolution starting anchor point and the target node is below a threshold. The relief path is established from the resolution starting anchor point to the target node that meets the conditions, and the path direction points from the bottleneck node to the idle node. When the direct relief path does not exist, the indirect relief path via intermediate nodes is searched, and the hop count of the indirect path does not exceed the set upper limit. The relief paths planned for each resolution starting anchor point are summarized, and there may be shared nodes or intersections between the paths. The intersection is detected and coordinated. All relief paths are integrated into a unified graph structure to form the bottleneck resolution path graph. The bottleneck resolution path graph is stored in the form of a directed graph, where the nodes represent scanning execution nodes, and the edges represent task relief directions. The edge attributes include the amount of migratable tasks and the migration priority. The bottleneck resolution path graph plans the relief paths from the three anchor points to five idle nodes, forming a multi-source and multi-sink relief network.

[0070] The task dynamic migration is performed according to the bottleneck resolution path diagram to generate the balanced execution configuration. The task migration operation is performed according to the priority order of each relief path in the bottleneck resolution path diagram. The path with the highest priority is processed first. The task to be migrated is selected from the task queue of the starting node of the path in the bottleneck resolution path diagram. The rule is to prefer the task that has not started execution to reduce the migration overhead. The selected task is removed from the source node queue and transmitted to the target node via the relief path specified in the bottleneck resolution path diagram. The target node receives the task and adds it to the local execution queue. The allocation relationship in the shard node table is updated for the migrated task. The allocation node is modified to the target node. The load change of each node after migration is monitored. When the source node load decreases to the normal range or the target node load approaches the upper limit, the migration of the path is stopped. Each relief path in the bottleneck resolution path diagram is processed in turn until the retention state of all task retention nodes is eliminated or there is no more task to be migrated. After the migration is completed, the load distribution and task allocation of each node are recalculated. The updated node load state, task allocation relationship and scheduling parameters are integrated into the balanced execution configuration. The balanced execution configuration is stored in the form of a configuration file, which includes the target load level of each node, the task receiving limit and the priority adjustment parameter.

[0071] In step S150, the cluster resource state collection is carried out according to the balanced execution configuration to generate a node load topology matrix. The optimal distribution time slot is determined by performing low-load interval matching on the node load topology matrix. The task balanced distribution instruction set is generated according to the optimal distribution time slot and the node load topology matrix.

[0072] The cluster resource state collection is carried out according to the balanced execution configuration to generate a node load topology matrix. Based on the node list in the balanced execution configuration, a resource state collection request is initiated to each scanning execution node. The collection content includes the static configuration information and the dynamic running state of the node. The static configuration information includes the number of CPU cores, the total memory, the network bandwidth and the storage capacity. The dynamic running state includes the real-time utilization rate and the historical utilization rate curve of each resource. Referring to the target load level of each node recorded in the balanced execution configuration, the collected data is structured and arranged. A matrix structure is constructed with node number as row index and resource type and time dimension as column index. At the same time, the topology connection information between nodes is embedded in the matrix. The topology connection describes the network reachability and communication bandwidth between nodes. The resource state matrix and the topology connection information are integrated to form a node load topology matrix. The node load topology matrix is stored in the form of a three-dimensional array. The first dimension corresponds to the node number, the second dimension corresponds to the resource type, and the third dimension corresponds to the time slice. The adjacency matrix describes the node topology. After the resource state collection of a cluster containing 20 scanning nodes, the node load topology matrix records the resource utilization rate snapshot of each node every minute in the past 24 hours.

[0073] In some embodiments, the low-load interval matching is performed on the node-bearing topology matrix to determine the preferred distribution time slot, including: performing time slicing on the node-bearing topology matrix to obtain a load snapshot of each time interval; calculating a node idle rate based on the load snapshot of each time interval to generate a time interval idle rate sequence; performing continuity detection on the time interval idle rate sequence to identify a low-load candidate time interval; and performing a duration evaluation based on the low-load candidate time interval to determine the preferred distribution time slot.

[0074] The time slicing is performed on the node-bearing topology matrix to obtain a load snapshot of each time interval. The time dimension of the node-bearing topology matrix is sliced according to a set time granularity, which is determined according to the accuracy requirement of task scheduling, and is usually set to minute or second level. Each time slice corresponds to a two-dimensional section of the node-bearing topology matrix at that time, and the section contains the utilization rates of various resources of all nodes at that time. The section data of each time slice is stored independently to form a time-ordered time interval load snapshot sequence. The time interval load snapshot is indexed by snapshot number and time label, and the content is the resource utilization rate vector of each node at that time. Statistical features are calculated for the time interval load snapshot, including the average load, maximum load, minimum load, and load variance of all nodes in each snapshot. The average load reflects the overall busy degree of the cluster at that time, and the load variance reflects the balance of the load distribution among the nodes. The statistical features are attached to the corresponding time interval load snapshot record to facilitate subsequent time interval screening and comparison. The time interval load snapshot of the night time interval usually shows a lower average load, and the snapshot of the working time interval shows a higher average load. The snapshot distribution pattern also differs between weekends and weekdays.

[0075] The node idle rate is calculated based on the load snapshot of each time interval to generate a time interval idle rate sequence. For each time interval load snapshot, the resource idle degree of each node at that time is calculated. The node idle rate is defined as the proportion of the remaining available resources to the total resources, and the calculation formula is idle rate = 1 - comprehensive utilization rate, where the comprehensive utilization rate is the weighted average of CPU, memory, network, and storage utilization rates. The idle rate of each node in the time interval load snapshot is calculated, and the average idle rate of all nodes at the same time is obtained to get the cluster-level idle rate. The cluster-level idle rate at each time is extracted by traversing all time interval load snapshots, and a time-ordered time interval idle rate sequence is formed. The time interval idle rate sequence is stored in the form of a time sequence, and the sequence element contains the time label and the corresponding cluster idle rate value. The fluctuation characteristics of the time interval idle rate sequence are analyzed, and the mean, variance, maximum, and minimum of the sequence are calculated. The change curve of the time interval idle rate sequence is drawn, and the peak of the curve corresponds to the time interval of the cluster idle, and the valley corresponds to the time interval of the cluster busy. The timing inspection scanning task is usually arranged to be executed in the high idle time interval shown in the time interval idle rate sequence to avoid resource competition with other tasks during the business peak period.

[0076] The continuity detection of the period idle rate sequence identifies low-load candidate periods. An idle rate threshold is set, which is determined according to the resource requirement of task execution and the service level agreement of the cluster. The period idle rate sequence is scanned, and the time points with idle rate higher than the threshold are marked as low-load times. The marked low-load times in the period idle rate sequence are analyzed for continuity, and the low-load times that are continuous in time are grouped into the same section. The continuity determination rule is that the interval between two adjacent low-load times does not exceed the maximum allowed gap, and the short high-load within the gap is considered as fluctuation noise and ignored. The start time, end time and duration of each continuous section are recorded, and the duration is equal to the end time minus the start time. The sections with duration longer than the minimum duration threshold are screened as low-load candidate periods, and the minimum duration threshold is determined according to the execution time of typical tasks to ensure that the low-load candidate period can accommodate complete task execution. The low-load candidate periods are stored in the form of a section list, and the list elements include section number, start time, end time, duration and average idle rate in the section. The lunch break and early morning period of weekdays are often identified as low-load candidate periods, and the whole day of weekends may form a continuous low-load candidate period.

[0077] Based on the low-load candidate periods, the duration evaluation is implemented to determine the preferred distribution time slots. The duration and idle rate level of each low-load candidate period are analyzed to evaluate its suitability for carrying task distribution. The suitability is comprehensively evaluated by two indicators of duration adequacy and idle rate stability. The duration adequacy reflects whether the low-load candidate period can accommodate the execution of the task to be distributed, and the calculation formula is adequacy = period duration / estimated execution time of the task, and the adequacy greater than 1 indicates that the period is sufficient to accommodate the task. The idle rate stability reflects the fluctuation degree of the idle rate in the low-load candidate period, and the calculation formula is stability = 1 - idle rate standard deviation / idle rate mean, and the higher the stability indicates that the resource supply in the period is more stable. The adequacy and stability are combined to calculate the suitability score of each low-load candidate period, and the score formula is suitability = a x adequacy + b x stability, where a and b are weight coefficients. The low-load candidate periods are ranked in descending order according to the suitability score, and the top several periods with the highest score are selected as the preferred distribution time slots. The preferred distribution time slots are stored in the form of a time slot list, and the list elements include time slot number, start and end time, suitability score and recommended distribution task type. Large-scale asset discovery scanning is suitable for allocation to preferred distribution time slots with long duration, and fast vulnerability verification scanning can be allocated to preferred distribution time slots with short duration but high idle rate.

[0078] The task balanced distribution instruction set is generated according to the preferred distribution time slot and the node carrying topology matrix. The preferred distribution time slot list is traversed, and the expected idle resource amount of each node in the time slot is obtained by querying the node carrying topology matrix for each time slot. The task type that can be distributed is determined according to the duration and resource characteristics of the preferred distribution time slot. The time slot with a long duration is suitable for distributing large-scale scanning tasks, and the time slot with a high idle rate is suitable for distributing computing-intensive tasks. The task distribution in each preferred distribution time slot is refined to a specific node, and the task is distributed to the node with the highest resource matching degree according to the node idle resource recorded in the node carrying topology matrix. The distribution time of the task is accurately timed, and the distribution time is set in the preheating window before the start time of the preferred distribution time slot. The distribution target node, distribution time and execution parameter of each task are organized into a distribution instruction, and the instruction format includes instruction number, target node address, distribution time and execution constraint condition. All distribution instructions are summarized to form a task balanced distribution instruction set. The task balanced distribution instruction set is stored in the form of an instruction queue, and the queue is sorted by distribution time. A certain scanning cluster generates a task balanced distribution instruction set according to three preferred distribution time slots and a node carrying topology matrix, wherein the morning time slot is matched to eight low-load nodes to perform network segment scanning, and the noon time slot is matched to four idle nodes to perform vulnerability verification.

[0079] In order to perform the network scanning task intelligent segmentation and load balancing method corresponding to the above-mentioned method embodiment, the corresponding functions and technical effects are realized. Referring to Figure 2 , Figure 2 The structure block diagram of the network scanning task intelligent segmentation and load balancing system 200 provided by the embodiment of the application is shown. For ease of illustration, only the part related to the embodiment is shown. The network scanning task intelligent segmentation and load balancing system 200 provided by the embodiment of the application comprises:

[0080] The task acquisition module 201 is configured to acquire a network scanning task request to be processed, perform task integrity verification on the network scanning task request to identify an abnormal task segment, and perform task feature analysis on the abnormal task segment to generate a segmentation reference sequence;

[0081] The scheduling establishment module 202 is configured to perform task unit reorganization processing on the segmentation reference sequence to generate a standard task queue, perform node state synchronization analysis on the standard task queue to extract a node load identifier, trigger a scheduling mode adaptation mechanism according to the node load identifier to generate a mode adaptation instruction, and establish a multi-node parallel scheduling link via the mode adaptation instruction;

[0082] The coordination balancing module 203 is configured to perform load coordination analysis on the multi-node parallel scheduling link to generate a node balancing index, perform multi-node coordination balancing processing according to the node balancing index to form a coordinated scheduling space, perform task fragmentation distribution on the coordinated scheduling space to generate a multi-node parallel execution sequence, and the like.

[0083] The feedback correction module 204 is configured to perform execution progress tracking on the multi-node parallel execution sequence to identify a task stagnation node, construct a bottleneck resolution path diagram according to the task stagnation node, perform task dynamic migration according to the bottleneck resolution path diagram to generate an equalized execution configuration, and the like.

[0084] The distributed execution module 205 is configured to perform cluster resource state collection according to the equalized execution configuration to generate a node carrying topology matrix, perform low-load interval matching on the node carrying topology matrix to determine an optimal distribution time slot, and generate a task equalization distribution instruction set according to the optimal distribution time slot and the node carrying topology matrix.

[0085] The network scanning task intelligent segmentation and load balancing system 200 described above can implement a network scanning task intelligent segmentation and load balancing method described above. The optional items in the method embodiments described above are also applicable to the present embodiment, and will not be described in detail here. The remaining content of the present embodiment can be referred to the content of the method embodiments described above, and will not be described in detail in the present embodiment.

[0086] The above embodiments are not an exhaustive enumeration based on the present application, and there can be a plurality of other embodiments not listed. Any replacement and improvement made without violating the concept of the present application is within the protection scope of the present application.

Claims

1. A method for intelligent segmentation and load balancing of network scanning tasks, characterized in that, include: Obtain network scanning task requests to be processed, perform task integrity verification on the network scanning task requests to identify abnormal task fragments, and perform task feature parsing based on the abnormal task fragments to generate a segmentation benchmark sequence. Based on the segmentation benchmark sequence, task unit reorganization processing is carried out to generate a standardized task queue. The node status synchronous parsing of the standardized task queue is used to extract node load identifiers. The node status synchronous parsing refers to the process of collecting the resource utilization rate of each scanning execution node, performing weighted calculation based on the resource utilization rate to obtain a comprehensive load index reflecting the overall load of the node, classifying the load level according to the comprehensive load index and encoding it to form a node load identifier. Based on the node load identifier, the scheduling mode adaptation mechanism is triggered to generate mode adaptation instructions, and a multi-node parallel scheduling link is established through the mode adaptation instructions. Load coordination analysis is performed on the multi-node parallel scheduling link to generate node balancing indicators. Multi-node collaborative balancing processing is performed based on the node balancing indicators to form a collaborative scheduling space. Task fragmentation and distribution are performed on the collaborative scheduling space to generate a multi-node parallel execution sequence. The execution progress of the multi-node parallel execution sequence is tracked to identify task stagnation nodes. A bottleneck resolution path graph is constructed based on the task stagnation nodes. Based on the bottleneck resolution path graph, the execution tasks are dynamically migrated to generate a balanced execution configuration. Based on the balanced execution configuration, cluster resource status is collected to generate a node carrying topology matrix. Low-load interval matching is performed on the node carrying topology matrix to determine the preferred distribution time slot. Based on the preferred distribution time slot and the node carrying topology matrix, a task balanced distribution instruction set is generated.

2. The method according to claim 1, characterized in that, The step of generating a segmentation benchmark sequence by parsing the task features based on the abnormal task fragment includes: Extract a task dependency graph based on the abnormal task fragments; Breakpoint detection is performed on the task dependency graph to identify the set of separable nodes; Construct a segmentation granularity constraint table based on the set of divisible nodes; A segmentation reference sequence is generated based on the segmentation granularity constraint table.

3. The method according to claim 1, characterized in that, The step of generating mode adaptation instructions based on the node load identifier triggering the scheduling mode adaptation mechanism includes: Perform load gradient analysis on the node load identifiers to generate a node hierarchy list; The load hierarchy is divided into high-load and low-load levels based on the node hierarchy list. The low-load level is used to allocate scheduling weights to the high-load level to form a differentiated scheduling configuration; Generate mode adaptation instructions based on the differentiated scheduling configuration.

4. The method according to claim 1, characterized in that, The step of performing multi-node collaborative balancing processing to form a collaborative scheduling space based on the node balancing index includes: Construct a node load difference matrix based on the node balancing index; Clustering is performed on the node load difference matrix to generate node cooperation groups; Establish inter-group task flow rules for the node collaboration groups; Based on the inter-group task flow rules, groups are integrated to form a collaborative scheduling space.

5. The method according to claim 1, characterized in that, The step of distributing tasks in the collaborative scheduling space to generate a multi-node parallel execution sequence includes: The task capacity that each node can carry is identified based on the collaborative scheduling space; Based on the stated task capacity, the granularity of the sharding is determined to generate a set of task shards; Execute node mapping and binding on the task shard set to generate a shard node lookup table; A multi-node parallel execution sequence is generated based on the shard node lookup table.

6. The method according to claim 1, characterized in that, The step of constructing a bottleneck resolution path graph based on the task delay nodes includes: Perform a bottleneck type labeling and cause diagnosis on the task-delayed nodes; Based on the bottleneck type label, perform bottleneck impact domain expansion analysis to generate the bottleneck impact range; Prioritize the elimination of the bottleneck's impact range to determine the starting anchor point for elimination; Based on the aforementioned initial anchor point, a bottleneck resolution path diagram is constructed by planning and guiding the path layer by layer.

7. The method according to claim 1, characterized in that, The step of performing low-load interval matching on the node-bearing topology matrix to determine the preferred distribution time slot includes: Perform time-series slicing on the node-bearing topology matrix to obtain load snapshots for each time period; Based on the load snapshots of each time period, the node idle rate is calculated to generate a time period idle rate sequence; The idle rate sequence of the aforementioned time periods is continuously detected to identify candidate low-load time periods; The optimal distribution time slot is determined based on the duration evaluation of the low-load candidate period.

8. The method according to claim 4, characterized in that, The establishment of inter-group task flow rules for the node collaboration groups includes: The node collaboration groups are used to perform boundary node identification to obtain the set of inter-group connection nodes; Based on the set of inter-group connection nodes, analyze the cross-group communication cost to generate a cost evaluation value; Based on the cost evaluation value, the task flow direction constraint is set, wherein when the cost evaluation value is lower than the preset cost threshold, bidirectional flow is allowed; when the cost evaluation value is higher than the preset cost threshold, unidirectional flow is restricted. Establish inter-group task flow rules based on the aforementioned task flow direction constraints.

9. The method according to claim 5, characterized in that, The step of determining the granularity of the sharding and generating a set of task shards based on the capacity to handle the task includes: Based on the stated task capacity, scan granularity is identified to generate coarse-grained scan blocks and fine-grained scan blocks; The fine-grained scanning block is used to cut the coarse-grained scanning block at its boundaries to form standard scanning slices; The standard scan fragments are adapted to the capacity of the task to generate node matching fragments. The node matching segments are sequenced and arranged to construct a task segment set.

10. A network scanning task intelligent segmentation and load balancing system, characterized in that, include: The task acquisition module is used to acquire network scanning task requests to be processed, perform task integrity verification on the network scanning task requests to identify abnormal task fragments, and perform task feature parsing based on the abnormal task fragments to generate a segmentation benchmark sequence. The scheduling establishment module is used to perform task unit reorganization processing to generate a standardized task queue based on the segmentation benchmark sequence, and to extract node load identifiers by synchronously parsing the node status of the standardized task queue. The synchronous parsing of node status refers to the process of collecting the resource utilization rate of each scanning execution node, performing weighted calculation based on the resource utilization rate to obtain a comprehensive load index reflecting the overall load of the node, classifying the load level according to the comprehensive load index and encoding it to form a node load identifier, triggering the scheduling mode adaptation mechanism to generate mode adaptation instructions based on the node load identifiers, and establishing a multi-node parallel scheduling link through the mode adaptation instructions. The collaborative balancing module is used to perform load collaborative analysis on the multi-node parallel scheduling link to generate node balancing indicators, perform multi-node collaborative balancing processing based on the node balancing indicators to form a collaborative scheduling space, and implement task fragmentation and distribution in the collaborative scheduling space to generate a multi-node parallel execution sequence. The feedback correction module is used to track the execution progress of the multi-node parallel execution sequence, identify task stagnation nodes, construct a bottleneck resolution path graph based on the task stagnation nodes, and dynamically migrate tasks to generate a balanced execution configuration based on the bottleneck resolution path graph. The distribution and execution module is used to collect cluster resource status data and generate a node bearer topology matrix based on the balanced execution configuration, perform low-load interval matching on the node bearer topology matrix to determine the preferred distribution time slot, and generate a task balanced distribution instruction set based on the preferred distribution time slot and the node bearer topology matrix.

Images