A performance optimization method of a distributed data analysis system

Abstract

This invention relates to the field of data processing technology and discloses a performance optimization method for a distributed data analysis system. This method addresses the problem in traditional methods of lacking coordinated control over logical execution plans, runtime resource status, and cross-node data transmission status. First, it collects the running status, node load status, and cross-node data transmission status of each execution stage and compares them with a preset execution baseline to analyze state deviations during job execution and generate coordinated correction information. It then coordinates the stage progression order, node acceptance relationships, and transmission order to improve resource allocation and task scheduling, reduce operational fluctuations, and ensure the stability and processing efficiency of distributed data analysis jobs.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a performance optimization method for a distributed data analysis system. Background Technology

[0002] With the development of big data processing, distributed computing, and multi-node collaborative analysis technologies, distributed data analysis systems have been widely applied in scenarios such as the collection, cleaning, statistics, mining, and result output of massive amounts of data. Existing technologies typically improve processing efficiency through task splitting, parallel execution, parameter configuration, or local scheduling. For example, the published invention patent application CN102479225B discloses a distributed data analysis and processing method and system, which distributes the database host load and improves data analysis and processing efficiency by splitting scripts, grouping tasks, load-sharing servers, parallel execution, and result feedback. The published invention patent application CN107948227B discloses a performance optimization method and apparatus for a distributed system platform, which improves the performance of the distributed system platform by parsing the partition number configuration information in the configuration file and updating the platform partition number. The above solutions optimize from the perspectives of task splitting and scheduling and partition parameter adjustment, respectively, and can improve the local processing efficiency of the system to a certain extent. However, existing solutions only optimize one aspect of the query plan, runtime parameters, task scheduling, or caching strategy. Optimization is largely based on static settings at task initiation, scheduling schemes, or local resource states, lacking comprehensive control over the logical execution plan, runtime resource states, and cross-node data transmission status. During distributed data analysis job execution, factors such as data skew, node load, network transmission status, and even the amount of intermediate results transmitted across nodes can change over time. This makes optimization decisions based on a single aspect unsuitable for the operating environment, leading to a mismatch between optimization decisions and job execution status. Consequently, some tasks may remain stalled for extended periods, long-tail tasks cannot be promptly eliminated, and cross-node data transmission becomes problematic. The increasing overhead of data transmission between nodes leads to significant fluctuations in overall job completion latency. Furthermore, when the execution plan cannot respond promptly to changes in the running status, local optimization may only improve the processing effect of a certain link without improving the overall system performance. Additionally, data transmission burden may increase after partition adjustments, and congestion at hot nodes may be further aggravated after task scheduling. Therefore, a performance optimization method for distributed data analysis systems is needed to address the lack of adjustment mechanisms between logical execution plans, runtime resource states, and cross-node data transmission states in existing technologies, which leads to dynamic mismatch in optimization decisions. This method aims to improve the stability of overall job execution, reduce fluctuations in job completion latency, and enhance the overall performance optimization effect of distributed data analysis systems. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a performance optimization method for distributed data analysis systems, solving the problem of the lack of coordinated control over logical execution plans, runtime resource status, and cross-node data transmission status in traditional methods.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A performance optimization method for a distributed data analysis system includes: S1: Receives data analysis jobs to be executed and establishes an execution baseline based on the logical execution plan, stage dependencies, and initial resource allocation relationships; S2: During the execution of the data analysis job, the running status, node load status and cross-node data transmission status of each execution stage are collected according to the preset sampling period, and corresponding stage status information is generated. S3: Determine whether there is a mismatch between the logical execution plan, runtime resource status and cross-node data transmission status based on the stage status information, and generate linkage correction information when a mismatch is determined, and determine the effective period of the linkage correction information. S4: Based on the effective period, coordinate the order of execution phases, partition access relationships, node acceptance relationships, and cross-node data transmission order; S5: Verify the execution status after the collaborative adjustment, and maintain the current adjustment relationship or perform downgrade or rollback processing based on the verification results.

[0005] Preferably, the system receives data analysis jobs to be executed and establishes an execution baseline based on the logical execution plan, stage dependencies, and initial resource allocation, including: Data analysis jobs are accessed and parsed to generate job access records and stage relationship tables; Based on the phase relationship table, establish phase dependency mapping, node group acceptance mapping, node quota boundaries, and estimated cross-node transmission relationships, and form an execution baseline record.

[0006] Preferably, during the execution of the data analysis job, the running status, node load status, and cross-node data transmission status of each execution stage are collected according to a preset sampling period, including: The sampling period is determined based on the execution baseline record, and the progress status, node carrying status and transmission association status are collected at each sampling time. The current state segment is determined based on the input arrival rate, output completion rate, and stage progress speed.

[0007] Preferably, the corresponding stage status information is generated, including: The stage operation records, node load records, and transmission status records at the same sampling time are merged to generate stage status information; The phase status information contains a set of key nodes, a set of hotspot partitions, a node load summary, and a transmission status summary. And perform consistency verification on the stage status information.

[0008] Preferably, the determination of whether there is a mismatch between the logical execution plan, runtime resource status, and cross-node data transmission status based on the stage status information includes: The stage status information is mapped to the execution baseline record to establish a mismatch candidate window; Based on the mismatch candidate window, plan mismatch candidates, resource mismatch candidates, and transmission mismatch candidates are generated; Mismatch determination records are generated based on the relationships, persistence status, and impact levels among the mismatch candidates.

[0009] Preferably, when a mismatch is determined, linkage correction information is generated, and the effective period of the linkage correction information is determined, including: Based on the mismatch determination record, linkage correction information is generated. The linkage correction information contains a set of correction items, effective start conditions, effective end conditions, maximum duration of action, verification items, and rollback trigger conditions. The effective period of the linkage correction information is determined by combining the state segment boundary and the sampling period boundary; The legality of the linked correction information is also checked.

[0010] Preferably, the execution phase's advancement order, partition access relationship, node acceptance relationship, and cross-node data transmission order are coordinated and adjusted based on the effective period, including: Load the linkage correction information before execution to form a correction execution order and an adjustment object mapping table; Implement quota freezing, link slot reservation, and batch version locking; The process involves adjusting the phased access relationships, switching the node acceptance relationships, and rearranging the transmission order in a fixed sequence. Control the scope of impact, the amount of effect, and the boundaries of anomalies in the adjustment; It also generates collaborative adjustment execution records.

[0011] Preferably, the execution status after collaborative adjustment is verified, including: Establish a verification window based on collaborative adjustment execution records; Within the verification window, the actual implementation of the adjustment items and the convergence of key state relationships are verified. Verification result records are generated based on the verification results. The verification result records contain the action generation efficiency, the proportion of ineffective items, the convergence judgment results, the processing conclusions, and the newly added anomaly levels.

[0012] Preferably, based on the verification results, the current adjustment relationship is maintained or a downgrade or rollback is performed, including: The maintenance path, downgrade path, or rollback path will be determined based on the verification results record; Generate corresponding maintenance records, downgrade version records, or rollback records; Switch job status according to the status transition rules; The current valid version will be used as the version reference for subsequent running status collection and mismatch determination.

[0013] Compared with existing technologies, this invention provides a performance optimization method for a distributed data analysis system, which has the following beneficial effects: 1. This invention establishes an execution baseline during the job access phase and continuously collects phase progress status, node load status, and cross-node data transmission status during job operation. It jointly determines deviations in the logical execution plan, runtime resource status, and cross-node data transmission status, and generates linkage correction information with effective time periods based on mismatches. It coordinately adjusts the phase progress order, partition access relationship, node acceptance relationship, and cross-node data transmission order. Combined with execution status verification, maintenance, degradation, and rollback, it ensures that optimization decisions match the actual operating status, reduces the continued retention of long-tail tasks, the continuous increase in load on hot nodes, and the continuous backlog of cross-node data transmission, avoids mutual obstruction of local optimizations, and ultimately improves the stability of distributed data analysis job execution and reduces job completion latency fluctuations. This solves the problem of the lack of coordinated linkage control of logical execution plan, runtime resource status, and cross-node data transmission status in traditional methods.

[0014] 2. This invention continuously collects the stage operation status, node load status, and cross-node data transmission status during the data analysis job operation process, compares them with the execution baseline, and identifies state deviations during the job operation process. After discovering deviations, it generates linkage correction information to achieve coordinated adjustment of various links such as stage advancement, node acceptance, and data transmission. Through pre-positioning, quota freezing, effective time period control, and legality check constraint adjustment process, combined with the verification of the adjusted execution status, it reduces resource conflicts, transmission backlog, and task delays, improves the sustainability of distributed data analysis jobs under complex operating conditions, and ultimately improves the stability and reliability of overall data processing. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a performance optimization method for a distributed data analysis system according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: Figure 1 A performance optimization method for distributed data analysis systems is presented, including: S1: Receives data analysis jobs to be executed, establishes an execution baseline based on the logical execution plan, stage dependencies, and initial resource allocation, specifically as follows: After receiving the data analysis job to be executed, the system first performs access parsing and generates an execution baseline record, and then proceeds to the subsequent running status collection process. The execution baseline record includes at least the execution structure, stage dependency boundaries, resource boundaries, and transmission expectations, which are used to unify the reference standards for subsequent stage status collection, mismatch judgment, linkage correction, and execution status verification.

[0018] In practice, the input information of the data analysis job is first read and a job access record is generated. The job access record is stored in a structured record format, and its input data includes at least job header information and execution relationship information. The job header information includes at least job number, submission time, business priority, target completion deadline, input dataset identifier set, and logical execution plan identifier. When the access information does not directly carry the logical execution plan version number, a unique plan version identifier can be generated based on the job definition information. The job definition information includes at least stage set, stage dependency relationship, input partition definition, and output partition definition. The execution relationship information includes at least stage... Fields include segment set, stage sequence dependency relationship, input partition set for each stage, output partition set for each stage, and initial node candidate set; the job number is used for unified association of status information, correction information and verification information throughout subsequent stages, and can be a globally unique code with a length of 16 to 32 bits, preferably 24 bits; the lower limit of this range is determined based on the uniqueness requirements under conditions of multi-job concurrency, batch retry and version reconstruction, and the upper limit is determined based on the record indexing overhead and cross-node transmission burden control requirements; the 24-bit code can be formed, for example, by combining the job identifier segment, time period segment, batch segment and version segment, so as to ensure uniqueness while taking into account record storage and transmission overhead; The business priority can be selected from level 1 to level 5, where level 1 represents the highest priority and level 5 represents the lowest priority. This classification method is a commonly used discrete priority representation method in this field, which facilitates unified judgment in subsequent resource contention and delayed access scenarios. The target completion time limit is used to represent the longest execution time allowed for the job, and its value can be selected from 30 seconds to 28800 seconds. For example, interactive analysis scenarios can be selected from 30 seconds to 300 seconds, near real-time statistics scenarios can be selected from 300 seconds to 1800 seconds, and offline batch processing scenarios can be selected from 1800 seconds to 28800 seconds. The above range is determined according to the common execution time distribution of different types of data analysis jobs. The lower limit corresponds to the shortest acceptable response time for interactive analysis, and the upper limit corresponds to the common completion time boundary within a single offline processing cycle. If the target completion time limit is not clearly defined in the job access stage, it will be difficult to uniformly determine the advancement boundary, conservative access boundary, and correction effective boundary later. After completing the job access record, the logical execution plan is decomposed into stages, transforming the original execution plan from a task sequence description into a stage relationship table that can express the stage connection relationship, data processing boundary, and progress boundary. The stage relationship table includes at least the following fields: stage number, set of preceding stage numbers, set of subsequent stage numbers, stage input scope, stage output scope, initial parallelism level of the stage, and critical path candidate identifier. The stage numbers are sequentially encoded from 1 to N, where N is determined according to the job decomposition granularity and can be selected from 3 to 50. The lower limit of this range corresponds to the three basic processing stages: input processing, core analysis, and result output. When the number of stages is less than 3, it is difficult to distinguish the above three stages simultaneously. The processing stage; the upper limit of this range is determined based on the granularity of stage decomposition within a single job and the complexity control requirements of cross-stage scheduling; when the number of stages is significantly higher than 50, the cross-stage scheduling relationship and state association relationship increase significantly, and the overhead of execution baseline maintenance and the complexity of subsequent state comparison increase significantly; the stage input caliber should at least include fields such as the input partition identifier range, the upstream stage to which the partition belongs, and the valid input conditions; the stage output caliber should at least include fields such as the output partition identifier range, the output target stage, and the output completion judgment conditions; the stage input caliber and stage output caliber are used to define the data boundaries of each stage and serve as a unified caliber for subsequent statistical input arrival ratio and output completion ratio; The initial parallelism level of a phase indicates the number of task batches or partition batches allowed to proceed simultaneously under the initial plan. Its value can range from 2 to 128, preferably from 8 to 64. This range is determined comprehensively based on the current number of nodes, the number of task batches a single node can stably handle, and the size of the phase's input partition. If the value is too low, it will be difficult to reflect the parallelism of distributed processing and will not be conducive to identifying the actual expansion pressure. If the value is too high, it will introduce excessive resource pre-occupancy assumptions during the baseline establishment phase and increase the burden of ineffective scheduling. The critical path candidate identifier is used to mark phases that are more likely to dominate the overall completion latency. It can be based on the phase dependency depth, the size of the input partition, and the average performance of that phase in historical execution records. The processing time level is jointly determined, whereby the average processing time level can be divided based on the average completion time range of similar stages in historical operations. When a stage is located in the last third of the dependency chain, or when the number of remaining dependency layers from the stage to the termination stage is higher than the median value of all stages in the operation, and the number of input partitions is more than 1.5 times the median value of input partitions in all stages in the operation, it can be marked as a candidate stage of the critical path. The 1.5 times threshold is used to distinguish input scales that are significantly higher than the median level of the overall stage load. When the threshold is lower than 1.5 times, it is easy to misidentify regular fluctuation stages as candidate stages of the critical path. When the threshold is significantly higher than 1.5 times, it may miss the early identification of high-risk stages. After the stage relationship table is generated, a stage dependency mapping is further established. The stage dependency mapping includes at least the stage sequence connection relationship, the prohibition of out-of-bounds relationship, and the conditions for lifting out-of-bounds conditions. The prohibition of out-of-bounds relationship is used to constrain subsequent stages to participate in execution only in a conservative access state when the preceding stage has not yet met the preset advancement conditions. The preset advancement conditions can be determined as follows: when the input arrival ratio of the preceding stage is less than 70%, or when the preceding stage has not yet formed a stable state, its subsequent stages maintain a conservative access state; when the input arrival ratio reaches or exceeds 70%, and the advancement rhythm fluctuation of the preceding stage does not exceed 15% within 2 to 4 consecutive sampling periods, its subsequent stages lift the conservative access restriction. The above 70% threshold is used to distinguish whether the input relationship has initially converged, and the 15% threshold is used to distinguish whether the advancement rhythm has entered the observable stable range. If the business emphasizes timeliness, the input arrival ratio threshold can be lowered to 60%; if the business emphasizes stability, the input arrival ratio threshold can be raised to 80%. By establishing the prohibition of out-of-bounds relationship in advance, a clear stage advancement boundary can be provided for subsequent plan mismatch identification. After establishing the stage dependency mapping, the initial resource allocation relationship is determined. The initial resource allocation does not use an average node allocation method; instead, candidate nodes are first grouped according to their resource fluctuation levels, and then the initial transition mapping from stage to node group is determined based on the execution nature of different stages. Resource fluctuation levels are defined based on the fluctuation range of node usage, memory usage, and transmission queuing within a preset statistical period, which can be selected from 3 to 30 minutes. If a node's usage fluctuation does not exceed 15%, memory usage fluctuation does not exceed 10%, and transmission queuing fluctuation does not exceed 12% within the last 5 consecutive sampling windows, it is assigned to the stable node group. If any of these fluctuations... Nodes fluctuating between 1 and 2 times the corresponding threshold are grouped into the transitional node group; if any fluctuation exceeds 2 times the corresponding threshold, they are grouped into the high-fluctuation node group. These thresholds are determined based on the stable operating fluctuation range of the nodes within the continuous sampling window. The computational resource usage fluctuation threshold is relatively relaxed because computational resources typically have strong local recovery capabilities; the memory usage fluctuation threshold is relatively tightened because memory fluctuations more easily and directly affect the continuity of stage acceptance; the transmission queuing fluctuation threshold is set to the middle range because transmission congestion usually has a cross-node amplification effect. This grouping method allows the initial acceptance relationship to match the node stability, providing a unified basis for subsequent resource acceptance and dynamic adjustments. In the initial resource allocation relationship, the upstream data processing stage is preferentially mapped to the stable node group and the transition node group to promote the convergence of input relationships; the intermediate aggregation stage configures the basic acceptance quota in the stable node group and reserves the elastic acceptance quota in the transition node group; the final result output stage is preferentially mapped to the stable node group to reduce the propagation of fluctuations in the completion stage; the value of the basic acceptance quota can be selected from 20% to 40% of the initial parallel level of the stage, preferably 30%; the lower limit of this range is determined according to the basic progress capability of the stage. If it is lower than 20%, it may be difficult for some stages to form a stable initial progress; the upper limit of this range is determined according to the subsequent resource redistribution margin. If it is higher than 40%, the acceptance space available for linkage adjustment will be compressed; When establishing a node assignment mapping, node quota boundaries are set synchronously. Node quota boundaries include at least the following fields: upper bound for computation quota, upper bound for memory usage, and upper bound for transmission concurrency. The upper bound for computation quota represents the maximum stable computational usage percentage that the current node can sustainably provide to the data analysis job; its value can be selected from 60% to 85%, preferably 70% to 80%. The upper bound for memory usage represents the maximum percentage of available memory that the data analysis job can occupy on the node; its value can be selected from 65% to 85%, preferably 72% to 80%. The upper bound for transmission concurrency represents the maximum percentage of stable transmission concurrency that the data analysis job can occupy on the node; its value can be selected from 40% to 75%, preferably 50% to 65%. The upper bound for stable transmission concurrency can... The number of transmission tasks a node is allowed to simultaneously handle is determined based on the fact that no congestion level increases on the link within a preset statistical period. The above range is determined comprehensively based on the need for nodes to reserve system overhead, instantaneous fluctuation buffers, and subsequent adjustment margins under stable operating conditions. The upper limit of the computation quota is higher than the upper limit of the transmission concurrency because computational resources usually have stronger local recovery capabilities, while transmission concurrency is more likely to amplify cross-node congestion. The upper limit of memory usage is controlled within 85% to avoid affecting the stability of phase handling when memory usage approaches its limit. If the upper limit of the quota is set too high, there will be insufficient remaining handling capacity available for adjustment when mismatches are identified later. If the upper limit of the quota is set too low, the execution baseline will be too conservative and will be difficult to reflect the phase advancement boundary under normal operating conditions. After completing the stage relationship table, stage dependency mapping, and initial resource allocation relationship, an execution baseline record is generated. The execution baseline record serves as the baseline record for the current job, and its content includes at least the following fields: baseline number, job number, stage relationship table version, stage dependency mapping version, initial stage-to-node group acceptance mapping table, node quota boundary table, estimated cross-node transmission relationship table, and baseline effective time. The baseline number uses a version encoding format associated with the job number; for example, a 2- to 4-digit version number can be appended to the job number to distinguish different baseline versions formed when the same job is re-accessed, rescheduled, rolled back, or abnormally rebuilt. The lower limit of the version number's length is used to meet the differentiation requirements when the same job is rebuilt multiple times and rolled back, while the upper limit is used to control the additional overhead that the identifier length brings to record indexing and transmission processing. The estimated cross-node transmission relationship table records fields such as the expected transmission direction, source node group set, target node group set, and expected transmission batch number range between adjacent stages. The expected transmission batch number range is determined based on the degree of partition merging, splitting, and reassembly between adjacent stages: when the output of an upstream single partition mainly corresponds to the reception of a downstream single partition, it can be regarded as a one-to-one correspondence transmission, and the batch number is close to the number of input partitions; when multiple upstream partitions merge into a single downstream partition, it can be regarded as batch aggregation, and the lower limit of the batch number can be selected as 0.8 times the number of input partitions; when the output of a single upstream partition needs to enter multiple downstream nodes or multiple downstream partitions, it can be regarded as batch splitting or multi-target forwarding. The upper limit for the number of batches can be selected as 1.5 times the number of input partitions; the lower limit of 0.8 times is used to cover the common situation where there is partial batch merging in adjacent stages but no large-scale convergence, and the upper limit of 1.5 times is used to cover the common situation of single batch splitting and limited multi-target forwarding; when the expected number of transmission batches exceeds this range, it usually indicates that the stage structure or transmission mode has changed significantly, and the execution baseline should be reassessed; for example, when the number of input partitions is 120, the expected transmission batch number range can be selected as 96 to 180; this range is used to provide a basis for comparison for subsequent judgment on whether the actual transmission relationship deviates from the execution baseline, rather than requiring the actual transmission batch number to be equal to a certain fixed value; After the baseline record is generated, it is registered as the current valid baseline version and the access state is changed to the pending observation state. The pending observation state corresponds to a job that has the conditions for collecting running status but the stage status has not been formed. Subsequent stage status collection, mismatch judgment, and linkage correction correspond to the current valid baseline number. When the baseline number is invalid, the baseline version is replaced, or the baseline version is inconsistent, the current processing is stopped, and the valid baseline version is verified, updated, or restored first. Anomaly and boundary handling can be performed according to the following rules: When the received data analysis job lacks a logical execution plan identifier and a unique plan version identifier cannot be generated based on the job definition information, no execution baseline is generated, and the job is placed in a pending completion state, while recording information such as missing fields; when there is a circular reference in the stage dependency relationship, it is marked as an illegal dependency anomaly, and the establishment of the execution baseline is terminated; when at least one critical path candidate stage cannot obtain a basic acceptance quota in the stable node group, the job is placed in a delayed access state, and the execution baseline is re-attempted after the resources meet the minimum acceptance conditions; when the partition identifier, partition quantity range, or the stage to which the partition belongs in the input partition set is inconsistent with the stage input caliber definition, it is marked as an input caliber inconsistency anomaly, and subsequent processing is stopped; the retry period in the delayed access state can be selected from 30 seconds to 300 seconds, preferably 60 seconds to 120 seconds; the lower limit of this range is determined to avoid too many invalid retries, and the upper limit is determined to control the waiting time of high-priority jobs, thereby balancing retry efficiency and access timeliness.

[0019] S2: During the execution of the data analysis job, the running status, node load status, and cross-node data transmission status of each execution stage are collected according to the preset sampling period, and corresponding stage status information is generated. The specific implementation is as follows: The execution baseline records generated above serve as input. After data analysis, the runtime status acquisition mechanism is activated. The runtime status is collected for stage progress status, node load status, and cross-node data transmission status. The stage relationships, stage dependency boundaries, node acceptance boundaries, and estimated cross-node transmission relationships in the execution baseline are used as comparison criteria. The runtime data generated within the same sampling period are merged to generate stage status information for subsequent mismatch determination. In practice, the status acquisition period is first initialized based on the execution baseline record. The execution baseline record includes at least the job number, target completion time limit, total number of stages, and stage dependency mapping. The sampling period is used to represent the time interval between two adjacent status acquisitions and to unify the observation granularity of stage advancement, node occupancy, and transmission backlog. The sampling period can be selected from 3 seconds to 15 seconds. For interactive jobs with a target completion time limit of less than 300 seconds, the sampling period can be selected from 3 seconds to 5 seconds. For batch jobs with a target completion time limit of more than 600 seconds, the sampling period can be selected from 5 seconds to 12 seconds. For batch jobs with a target completion time limit of more than 72 seconds, the sampling period can be selected from 5 seconds to 12 seconds. Long jobs of 00 seconds can be selected from 10 to 15 seconds; the above range is determined based on the common execution duration distribution and problem propagation speed of different types of data analysis jobs; the lower limit is used to avoid significant diffusion of stage advancement and transmission backlog between two adjacent sampling points, which may lead to failure to identify them in time, while the upper limit is used to avoid excessive amplification of short-term jitter, instantaneous backlog, or single batch surges; after the job enters execution, the sampling period corresponding to the current baseline version is fixed; the sampling period is reset only when the job enters the rollback reconstruction state or the baseline reconstruction state, thereby ensuring that the time caliber of the same round of state collection is consistent; At the start of each sampling period, the running status of each execution stage is first collected. The running status is temporarily stored in the form of a stage running raw record. This record contains at least the following fields: baseline number, job number, stage number, sampling time, number of input partitions received, number of output partitions, number of ready tasks, number of tasks in progress, number of blocked tasks, and stage progress speed. The number of input partitions received indicates the number of partitions that have entered the current stage's input scope and meet the valid input conditions at the current sampling time. The number of output partitions indicates the number of partitions that have been processed in the current stage and entered the output scope of the subsequent stage. The number of ready tasks indicates the number of tasks that meet the execution prerequisites but have not yet been connected to the node. The number of tasks in progress indicates the number of tasks that are being allocated execution resources and are in the processing state. The number of blocked tasks indicates the number of tasks that cannot continue to progress due to lack of input, incomplete transmission, or insufficient node acceptance conditions. The stage progress rate is used to represent the degree of change in task progress within an adjacent sampling period. Its value can be selected as the ratio of the number of newly completed tasks in the current sampling period to the number of newly completed tasks in the previous sampling period. When the number of newly completed tasks in the previous sampling period is 0, the stage progress rate can be recorded as a preset default value, or it can be marked as a sudden increase based on the number of newly completed tasks in the current sampling period. The reason for using this recording method is that simply recording the number of tasks is difficult to reflect the change in stage processing speed, while the stage progress rate can provide a unified basis for subsequent judgment on whether the stage progress tends to be stable. After obtaining the original records of the phase operation, the current state segment is further determined. The state segment serves as the boundary basis for subsequent mismatch determination and linkage correction, and can be divided into the initial segment, expansion segment, stable segment, convergence segment, and termination segment. The determination of the state segment includes at least the input arrival ratio, output completion ratio, and stage advancement speed change amplitude. The input arrival ratio represents the proportion of the number of input partitions received in the current stage to the expected number of input partitions in the execution baseline for that stage. The output completion ratio represents the proportion of the number of output partitions in the current stage to the expected number of output partitions for that stage. The stage advancement speed change amplitude represents the degree of change in the stage advancement speed within two consecutive sampling periods. The specific judgment rules can be set as follows: when the input arrival rate is below 30%, it is judged as the initial stage; when the input arrival rate is between 30% and 70%, and the number of ready tasks and the number of tasks in execution increase for two consecutive sampling periods, or both increase compared to the previous sampling period, it is judged as the expansion stage; when the input arrival rate is above 70%, and the change in the stage's progress speed does not exceed 15%, it is judged as the stable stage; when the output completion rate exceeds 75%, and the number of blocked tasks decreases for at least one consecutive sampling period, it is judged as the convergence stage; when the output completion rate exceeds 95%, and the remaining task quantity is less than 20% of the initial parallel level of this stage, it is judged as the termination stage; the above thresholds are based on... Typical change boundaries during the phased advancement process are defined as follows: 30% is used to characterize the early state where input has not yet reached a large-scale access level; 70% is used to characterize the input relationship has basically converged; 75% is used to characterize the phase entering a clear closing interval; 95% is used to characterize the near-completion state; and 20% is used to characterize the remaining task volume entering the tail processing range. A 15% threshold for the change in phase advancement speed is used to distinguish whether the phase advancement has entered a stable interval. This threshold can be adjusted to 10% to 20% depending on the node size. It can be appropriately relaxed when the number of nodes is small, and appropriately tightened when the number of nodes is large and the parallel fluctuation is strong. This range is determined based on the natural fluctuation amplitude of the phase advancement speed under different node sizes. After completing the phase operation status collection and status segment determination, the node load status is collected synchronously. The node load status collection object can be selected as the set of nodes currently undertaking the tasks of this data analysis job phase. Irrelevant nodes are not fully sampled to control the status collection overhead. The original node load status record includes at least the following fields: node number, node group to which the node belongs, computation usage ratio, memory usage ratio, buffer usage ratio, transmission concurrency usage ratio, number of pending tasks, and node fluctuation level. The computation usage ratio indicates the proportion of the computational usage provided by the current node for this job to its stable output computing capacity. The memory usage ratio indicates the proportion of the effective memory used by the job on the node to the node's available memory. The buffer usage ratio indicates the proportion of the intermediate result temporary storage area between phases to the reserved buffer space. The transmission concurrency usage ratio indicates the proportion of the number of data transmission tasks currently in the sending or receiving state to the node's maximum stable transmission task number. The maximum number of stable transmission tasks per node can be pre-determined under the condition that the link congestion level does not increase within 10 consecutive sampling periods. The observation length of 10 consecutive sampling periods is used to cover the combined effects of short-term link fluctuations and phase switching fluctuations. When the observation length is less than 10 sampling periods, it is easy to mistake short-term idle time for stable transmission capacity; when it is significantly more than 10 sampling periods, it will reduce the adaptability to real-time operating conditions. The node fluctuation level follows the classification logic of stable node group, transition node group, and high fluctuation node group, and allows dynamic correction during operation. If a transition node experiences high fluctuations within 5 consecutive sampling periods... If the fluctuation in internal computing usage does not exceed 10%, memory usage fluctuation does not exceed 8%, and transmission queuing fluctuation does not exceed 10%, the node can be temporarily upgraded to a stable level. If any fluctuation of a stable node exceeds twice the corresponding threshold within three consecutive sampling periods, it will be downgraded to a transitional level or a high fluctuation level. Five consecutive sampling periods are used for upgrading to avoid false upgrades caused by short-term stability. Three consecutive sampling periods are used for downgrading to reflect operational risks in a timely manner when a node shows significant instability. The memory fluctuation threshold is lower than the computing and transmission thresholds because memory fluctuations are more likely to directly affect the continuity of stage transitions. After collecting node load status data, cross-node data transmission status data is collected. The cross-node data transmission status data is collected based on the estimated cross-node transmission relationship table in the execution baseline. The data collection objects are the source stage, target stage, source node, and target node that actually have transmission associations within the current sampling period. The original transmission status record includes at least the following fields: transmission batch number, source stage number, target stage number, source node number, target node number, number of data blocks to be sent, number of data blocks to be received, sending queue length, receiving queue length, number of completed transmission batches, current transmission order level, and link congestion level. The transmission batch number can be a composite encoding form of stage number plus sequence number to locate the specific backlog position along the transmission link. The statistical caliber of sending queue length and receiving queue length is fixed when the job is accessed. It can be selected as batch number caliber or data block number caliber. It is preferred to use batch number caliber to directly identify whether the key batch has been moved forward based on the transmission order. Link congestion levels are classified into four levels: low, medium, high, and critical. A low level occurs when neither the transmit queue length nor the receive queue length increases significantly within three consecutive sampling periods. A medium level occurs when at least one of the two increases continuously with an increase of less than 30%. A high level occurs when at least one of the two increases by 30% to 80%. A critical level occurs when the increase exceeds 80% within two consecutive sampling periods, or when critical path batches wait continuously for more than two sampling periods. These thresholds are determined based on the evolution of transmission backlog from normal fluctuations to a spreading state, and then to a significant congestion state. A 30% threshold is used to identify the beginning of transmission backlog spreading, and an 80% threshold is used to identify that the link is approaching significant congestion. A critical path batch waiting continuously for more than two sampling periods indicates that the wait is sufficient to propagate the congestion downstream, and therefore it is treated as a critical state. After collecting data on stage operation status, node load status, and cross-node data transmission status, the three types of raw records are merged at the same sampling time to generate stage status information. The stage status information is organized in the form of a stage status record set, and its record content includes at least the following fields: record number, baseline number, job number, stage number, status segment number, sampling time, stage input arrival ratio, stage output completion ratio, number of ready tasks, number of tasks in execution, number of blocked tasks, set of key nodes, set of hotspot partitions, node load summary, transmission status summary, and stage progress level. The set of key nodes represents the set of nodes that simultaneously meet the following conditions within the current sampling period: node load is higher than the average level of the current receiving nodes, and they actually receive the candidate stage tasks of the critical path. Among them, a node load higher than the average level of the current receiving nodes can be determined by any one of the following: computational usage ratio, memory usage ratio, and transmission concurrency usage ratio being higher than the corresponding average value, or by the comprehensive load index formed by the three being higher than the average value. A hotspot partition set refers to a set of partitions in the current sampling period where at least one of the following—the number of tasks in the queue, the number of data blocks to be sent, or the number of data blocks to be received—is higher than the median level for the same period. The threshold for determining this set can be selected as 1.5 to 2.5 times the median level, preferably 1.8 to 2.2 times. This range is determined based on the partition load distribution characteristics under different scales of operations: when the threshold is below 1.5 times, it is easy to misidentify normal load fluctuations as hotspot partitions; when the threshold is above 2.5 times, it may delay the identification of persistent hotspot chains. The preferred range is used to strike a balance between identification timeliness and misjudgment control. To avoid a single sampling anomaly directly triggering subsequent mismatch determination, consistency verification is performed after the stage state information is generated. The consistency verification objects include at least the stage input arrival rate, node computational occupancy rate, and transmission queue length. If the stage input arrival rate in the current sampling period changes by more than 40% compared to the previous sampling period, and the node load summary and transmission state summary do not change in the same direction, the corresponding stage state record is marked as pending confirmation and will not be included in the formal mismatch determination set. If a node's computational occupancy rate increases by more than 30% in a single sampling period, but falls back to the original fluctuation range in the next sampling period... If the change is recorded as an instantaneous fluctuation, the node fluctuation level will not be updated. If the sending queue length of a certain transmission batch suddenly increases to more than twice the original value within a sampling period, while the receiving queue length does not change synchronously and recovers in the next sampling period, the batch will be marked as transient congestion, and a high-level link congestion state will not be triggered based on this. Among them, the 40% threshold is used to identify abnormal input jumps that are inconsistent with the node load state and transmission state, and the 30% threshold is used to identify sudden increases in node occupancy that exceed the normal fluctuation range. Both are used to eliminate short-term noise and avoid using single-point sampling anomalies directly as the basis for mismatch judgment. Time and resource constraints can be further set as follows: each sampling, merging, and generation of stage status information should be completed within the current sampling period, preferably not exceeding 40% of the sampling period; this proportion is used to reserve processing time for the next round of sampling, status merging, and subsequent judgment within the current sampling period, and to avoid the status acquisition process from occupying the effective execution time of the job in reverse; for example, when the sampling period is 5 seconds, the status acquisition and processing process should preferably be completed within 2 seconds; the auxiliary overhead introduced by status acquisition and record processing should not exceed 5% of the stable computing capacity of the node undertaking the data analysis job, preferably not exceeding 3%; this range is determined based on the principle that status acquisition should not significantly change the operating status of the observed job; if the current job is in a high-priority state and the remaining time limit is less than 20% to 30% of the target completion time limit, the sampling summary frequency of non-critical path stages can be adjusted to be collected every other period, and full-cycle collection can be maintained for candidate stages of the critical path, so as to balance the acquisition accuracy and auxiliary overhead control; Regarding boundary condition handling, when a certain stage has not yet reached the starting segment condition, only a preheating record is generated; when a node loses connection within two consecutive sampling periods, it is marked as unavailable in the node load summary, and an intermittent anomaly flag is added to the corresponding stage status record; when the source stage of a certain transmission batch has entered the end segment, but the target stage has not yet received the corresponding data block, the transmission batch is marked as transmission pending verification status.

[0020] S3: Determine whether there is a mismatch between the logical execution plan, runtime resource status, and cross-node data transmission status based on the stage status information, and generate linkage correction information when a mismatch is determined, and determine the effective period of the linkage correction information. The specific implementation is as follows: The input data is merged to form a mismatch judgment input, which may include fields such as baseline number, job number, stage number, state segment number, number of consecutive sampling periods, stage input arrival ratio, stage output completion ratio, number of ready tasks, number of tasks in execution, number of blocked tasks, set of important nodes, set of hotspot partitions, node load summary, transmission status summary, and target completion time limit. By matching the stage status information with the stage relationship table version, stage dependency mapping version, stage initial acceptance mapping version, and estimated cross-node transmission relationship table in the execution baseline record, the baseline reference value and running offset value under the same stage and the same state segment can be obtained. The runtime offset is not a numerical difference, but rather a structural change in the current stage's advancement relationship, node acceptance relationship, and transmission order relationship relative to the execution baseline. It refers to structural changes in the direction, order, or proportion of the runtime relationship's correspondence with the execution baseline that continuously deviate from the boundary. For example, if a downstream stage is constrained by the input boundary of the preceding stage and only allows conservative access, and significant queue expansion has already occurred in the runtime state, this is considered a deviation in the advancement relationship. Similarly, if a critical path stage primarily accepts stable node groups in the execution baseline, but the proportion of highly volatile node groups continuously increases in the runtime state, this is considered a deviation in the node acceptance relationship. Furthermore, if a batch corresponding to a hotspot partition is in a priority transmission direction in the execution baseline, but continuously lags behind in the transmission queue in the runtime state, this is considered a deviation in the transmission order relationship. To ensure that mismatch determination is based on continuous change, a mismatch candidate window is established before the formal determination. The mismatch candidate window is used to aggregate stage state information belonging to the same stage and state segment within multiple consecutive sampling periods into a timing unit to be determined. The window length can be selected from 2 to 5 sampling periods, preferably 3 sampling periods. Specifically, for jobs with a target completion time of less than 300 seconds, 2 sampling periods are preferred; for jobs with a target completion time between 300 and 7200 seconds, 3 sampling periods are preferred; and for jobs with a target completion time greater than 7200 seconds, 4 to 5 sampling periods can be selected. The above ranges vary depending on the type of job. The completion time limit and problem propagation speed of the type of task are determined: when the window length is too short, transient jitter is easily misjudged as mismatch; when the window length is too long, the problem may have already spread significantly before the completion judgment, thereby reducing the timeliness of subsequent adjustments; the mismatch candidate window should include at least the following fields: window number, start sampling time, end sampling time, task number, stage number, status segment number, set of stage status record numbers included in the window, and summary set of key nodes, hotspot partitions and key transmission batches within the window; the window number can adopt a composite encoding form associated with the task number and stage number, so that the corresponding correction information can be called according to the window granularity later; After the mismatch candidate window is established, the following steps are performed sequentially: plan mismatch determination, resource mismatch determination, and transmission mismatch determination. The plan mismatch determination uses the stage dependency mapping as a reference to check whether the current stage's progress relationship breaks through the execution baseline boundary of the preceding stage. When the number of ready tasks in a downstream stage within the mismatch candidate window increases continuously, and the input arrival ratio of its preceding stage is continuously lower than a preset stability threshold, a plan mismatch candidate is determined to exist. The preset stability threshold can be selected from 70% to 85%, preferably 75%. This range is used to distinguish whether the input structure of the preceding stage has reached a level that supports the stable expansion of the subsequent stage. When the input arrival ratio is lower than 75%, it usually indicates that the input structure of the preceding stage has not yet fully converged. If the downstream stage has already shown significant expansion at this time, it means that the stage dependency boundary may have been broken in advance. If at the same time, the number of blocked tasks in the downstream stage continues to rise, or the stage progress level changes from stable to sluggish, then the plan mismatch candidate is marked as a higher-level candidate state. Resource mismatch determination is based on the initial acceptance mapping table from stage to node group and the node quota boundary table, checking whether the node acceptance relationship of the current stage deviates from the execution baseline. When the proportion of highly volatile nodes in the current acceptance nodes of the critical path stage is between 30% and 50%, preferably 40%, and the number of blocked tasks in this stage does not decrease within the mismatch candidate window, a resource mismatch candidate is determined to exist. The above range is determined based on the identification sensitivity and determination stability: when the proportion is less than 30%, it is usually insufficient to have a sustained impact on the overall progress; when the proportion reaches more than 40%, node acceptance instability has usually begun to affect the progress of the critical path stage; the upper limit of 50% is used to cover common situations where node acceptance deviates significantly from the execution baseline. Transmission mismatch determination uses the estimated cross-node transmission relationship table as a reference to check whether there is an unreasonable lag in the hotspot partition batch. When the set of hotspot partitions maintains a high degree of overlap within the mismatch candidate window, and the intersection ratio of three adjacent sampling periods reaches more than 60%, while the sending queue length or receiving queue length of the corresponding transmission batch continues to increase, a transmission mismatch candidate is determined to exist. The 60% threshold is used to distinguish between occasional hotspots and persistent hotspot chains. When the intersection ratio is low, the stability of the hotspot object is insufficient, making it difficult to indicate a persistent transmission deviation. When the intersection ratio reaches or exceeds 60%, it indicates that the same batch of hotspot partitions appears continuously within the continuous window, which has a stable basis as a transmission mismatch candidate. The condition of continuously increasing sending queue length or receiving queue length is used to associate hotspot identification with actual transmission backlog, avoiding the determination of transmission mismatch based solely on changes in heat intensity. After obtaining the planned mismatch candidates, resource mismatch candidates, and transmission mismatch candidates, the linkage relationship is identified to determine whether the current problem is a single mismatch or multiple mismatch couplings. The linkage relationship identification is performed according to preset branch conditions: when both the planned mismatch candidate and the transmission mismatch candidate are established, and the intersection ratio of the hot spot partition set or key node set associated with them reaches more than 30%, it is determined that planned transmission coupling exists; when both the resource mismatch candidate and the transmission mismatch candidate are established, and there is a node number overlap between the high fluctuation node set and the source node set with the rapidly growing transmission queue length, or the high fluctuation node is located in the source node set with the fastest growing transmission queue length, it is determined that resource transmission exists. Coupling; when the candidate for mismatch in planning, resources, and transmission is simultaneously established, and the progress level of the critical path stage decreases continuously for at least two sampling periods, it is determined to be a compound mismatch; the intersection ratio can be selected from 20% to 40%, preferably 30%; this range is used to distinguish between weak and strong correlations: when the intersection ratio is less than 20%, the correlation between multiple mismatches is usually insufficient; when the intersection ratio is greater than 40%, it often shows strong coupling; the preferred value of 30% is suitable for correlation identification under most conventional cluster sizes; the continuous decrease in the progress level of the critical path stage for at least two sampling periods is used to exclude the transient impact of single-period fluctuations and to indicate that the progress decline has become persistent; After completing the linkage identification, a mismatch judgment record is generated. The mismatch judgment record includes at least the following fields: mismatch number, job number, stage number, status segment number, mismatch type, mismatch level, set of associated nodes, set of associated partitions, set of associated transmission batches, start sampling time, number of duration periods, dominant problem marker, and processing priority. The mismatch level can be selected from level 1 to level 4, where level 1 represents mild and level 4 represents severe. The mismatch level is jointly determined based on persistence, scope of impact, and degree of erosion over time: when there is only a single mismatch candidate and the number of duration periods does not exceed 2 sampling periods, it is recorded as level 1. This threshold is used to distinguish between short-term deviation and persistent deviation. When there is a single mismatch that has affected the critical path, or when there are two types of mismatch coupling but the scope of impact is still limited to a single stage chain, it is recorded as level 2. When two types of mismatch coupling exist and cause a continuous increase in critical path blocking tasks, it is recorded as Level 3; when a compound mismatch occurs and the estimated completion time exceeds the target completion time by 10% to 20%, it is recorded as Level 4; the estimated completion time can be estimated based on the current stage progress speed, remaining task volume, and current transmission backlog status; the 10% to 20% range is used to identify a state that has significantly eroded the target completion time and has a risk of continuous spread; the dominant problem marker is used to determine the priority of starting from the progress relationship, the succession relationship, or the transmission relationship in subsequent corrections; the processing priority can be selected as high, medium, or low, where high priority can be taken when the mismatch level is Level 3 or 4, low priority can be taken when the mismatch level is Level 1, and medium or high priority can be determined by combining the business priority and the current status segment when the mismatch level is Level 2; After generating the mismatch determination record, the linked correction information is generated. The linked correction information includes at least the following fields: correction number, corresponding mismatch number, correction target stage, set of correction items, effective start condition, effective end condition, effective period, maximum duration, set of verification items, and rollback trigger condition. The set of correction items can be selected from one or more combinations of stage advancement order adjustment, partition access relationship adjustment, node acceptance relationship adjustment, and cross-node transmission order adjustment. To ensure that subsequent processing can be executed directly, boundary conditions are set in advance for the linked correction information: when the target stage has entered the end phase, only transmission order adjustment or node acceptance fine-tuning with a small impact range is allowed, and large-scale partition access re-adjustment is not initiated. When the transmission progress of a hotspot batch exceeds 50%, its transmission order will no longer be adjusted, but its subsequent reception priority will be adjusted first. The 50% threshold is used to distinguish the effective range of transmission order adjustment. When the transmission progress has exceeded half, adjusting the transmission order is likely to introduce new queue disturbances and reduce the adjustment benefits. When the critical path stage is still in the latter half of the expansion phase, a combination of advancing order convergence and node acceptance switching can be adopted. The maximum effective duration can be selected from 1 to 4 state segments, preferably 2 state segments. This range is used to ensure that the correction action has a clear effective range: if the effective duration is too short, the adjustment effect may not have been manifested yet; if the effective duration is too long, the old correction may continue to act on the changed problem environment. The effective period is used to determine the time boundary for the execution of linkage correction information. Its value is determined jointly by the state segment boundary and the sampling period boundary. When mismatch occurs in the expansion segment, the effective period is preferably from the second half of the current expansion segment to the first half of the next stable segment, so as to intervene before the problem continues to spread and avoid interrupting the normal progress that has not yet formed in the early stage of expansion. When mismatch occurs in the stable segment, the effective period can be selected as 2 to 4 consecutive sampling periods within the current stable segment. This range is determined according to the time required to observe the adjustment effect and the requirements for problem spread control. Less than 2 sampling periods are usually insufficient to observe the adjustment effect within the stable segment, while more than 4 sampling periods may delay the entry of the next round of correction. When mismatch occurs in the convergence segment, only correction entries with a small impact range are retained, and the effective period is narrowed to 1 to 2 sampling periods. This range is determined based on the characteristics of the small remaining task volume in the convergence segment and the fact that long-term correction is easy to reduce the adjustment benefits. Legality checks are performed after the linked correction information is generated. These checks include at least stage dependency legality checks, resource boundary legality checks, and transmission budget legality checks. Stage dependency legality checks ensure that the correction entry does not exceed the necessary boundaries of the next stage. Resource boundary legality checks ensure that the adjusted node connection relationship does not exceed the computation, memory, or transmission concurrency quota boundaries of the execution baseline. Transmission budget legality checks ensure that the adjusted transmission order does not exceed the execution baseline allowed for the total cross-node transmission volume during the current effective period. If any legality check fails, the linked correction information does not enter the pending execution state, is returned to the correction candidate set, and is recorded as a restricted correction attempt. If the same correction information fails two consecutive legality checks, it is recorded as a restricted correction attempt and will not be retried within the current state segment. The setting of two consecutive failures is used to distinguish between occasional boundary conflicts and unexecutable states, avoiding invalid corrections repeatedly occupying processing resources. The generated mismatch judgment record and linked correction information record serve as input for collaborative adjustment.

[0021] S4: Based on the effective period, the execution phase's advancement order, partition access relationships, node acceptance relationships, and cross-node data transmission order are coordinated and adjusted. Specifically, the implementation is as follows: Before execution, the linked correction information is loaded to form a correction execution order. The correction execution order includes at least the following fields: correction number, corresponding mismatch number, target stage set, correction item set, effective start condition, effective end condition, effective period, maximum duration, verification item set, and rollback trigger condition. Based on this, an adjustment object mapping table is generated to map correction items to specific stages, partition batches, node groups, and transmission batches. The adjustment object mapping table includes at least the stage number, the set of batch numbers to be adjusted, the original node acceptance set, the candidate target node set, the original transmission order identifier, the target transmission order identifier, and... Adjustment priority and other fields; the adjustment priority is determined jointly based on the mismatch level and the target completion deadline: when the mismatch level is level 3 or 4 and the remaining percentage of the target completion deadline is less than 40%, the adjustment priority is high; when the mismatch level is level 2 and the current stage is in a stable phase, the adjustment priority is medium; when the mismatch level is level 1 and the impact is limited to non-critical path stages, the adjustment priority is low; the 40% threshold is used to distinguish whether the remaining adjustment time is significantly limited; when it is below this percentage, the correction window is significantly shortened, and adjustments corresponding to higher-level mismatches should be prioritized for execution; Before the formal adjustment, a pre-placement process is performed. This pre-placement process reserves resources, transmission locations, and batch version boundaries for the upcoming adjustment action. The pre-placement duration is calculated based on the sampling period, and can be selected from 1 to 3 sampling periods, with 2 sampling periods being preferred. If the remaining percentage of the current target completion deadline is less than 25% of the total deadline, the pre-placement duration can be 1 sampling period. This threshold is used to ensure that pre-placement does not consume too much effective adjustment time in time-sensitive scenarios. If the adjustment object involves more than 2 stages, more than 2 node groups, or multiple key transmission batches, the pre-placement duration can be 2 to 3 sampling periods. The sample period is designed to cover the reserved operation time required for parallel switching of multiple objects. During the pre-positioning period, the quota of the target node group to receive the new task is frozen. That is, during the current pre-positioning period, other low-priority jobs are no longer allowed to occupy the computing quota, memory quota and transmission concurrency quota reserved in this node group for this adjustment. The reservation threshold here can be determined based on the sum of the computing quota requirement, memory quota requirement and transmission concurrency requirement corresponding to the correction execution order, or it can be taken as 100% to 120% of the quota expected to be occupied by the corresponding adjustment object during the effective period, in order to cover normal adjustment needs and short-term fluctuation margin. After the quota is frozen, slot reservations are made for the target links to receive the new transmission batches. The number of slots reserved can be 10% to 35% of the current link's stable transmission concurrency limit, preferably 20% to 25%. This range is used to balance the entry positions required for the forward movement of critical batches and the basic channel requirements of non-critical transmission batches. If it is less than 10%, it is difficult to ensure the stable entry of critical batches. If it is more than 35%, it is easy to overcrowd non-critical transmission capacity. Subsequently, version locking is performed on the partition batches and transmission batches to be adjusted. Version locking only applies to objects that have not yet entered the execution state or the sending state. Objects that have entered the execution state or have been sent more than halfway are no longer included in the version locking range to avoid reverse interruption of objects that have formed output chains. Version locking is released when the adjustment is completed, a rollback is triggered, or the adjustment is canceled. If, during the pre-reservation period, the available quota of the target node group is lower than the reserved threshold, the target link slot is occupied by a higher priority job, or a version conflict occurs in the batch to be adjusted, the current correction execution order will not enter the formal effective state, but will remain in the pending execution state and will be retried in the next effective period. After pre-positioning is approved, the phase advancement relationship is adjusted first. This adjustment prioritizes critical path phases and high-risk phases, achieving resource rebalancing between phases through differentiated advancement boundaries. When the dominant issue marker in the linkage correction information includes plan mismatch, and a backlog of ready tasks and blocked tasks appears in the downstream phase before the preceding phase reaches its stable advancement boundary, a conservative advancement rule is applied to that downstream phase. The conservative advancement rule limits the upper limit of new access batches within the current effective period. Its value can be selected from 40% to 80% of the initial parallel level of the phase in the execution baseline, preferably 60% to 70%. This range is used to balance expansion suppression and minimum advancement capacity preservation. Below 40%, phase advancement capacity is prone to insufficiency; above 80%, it is premature. Expansion is difficult to effectively suppress; for example, if the initial parallelism level of a certain stage is 20, the upper limit of the new access batch can be selected as 12 to 14; if the current problem is mainly manifested as the slow progress of the critical path stage, while the non-critical path stage continues to occupy node resources and transmission positions, then a yield rule is applied to the non-critical path stage; the yield rule compresses the access rate of new tasks in the non-critical path stage to 50% to 75% of the original progress rate, preferably 60%; this range is used to release the resources and transmission channels required by the critical path, while avoiding excessive compression that would create new progress waste; after the stage progress relationship is adjusted, a progress adjustment record is generated, which includes at least the stage number, the access limit before adjustment, the access limit after adjustment, the effective start time, and the failure condition; After adjusting the phased advancement relationships, the partition access relationships are adjusted accordingly. This adjustment applies to new partition batches that have not yet entered the execution state and queued but not yet started batches, without altering partition objects that have already entered the execution state and formed processing chains. Each partition batch includes at least the following fields: batch number, original partition number, current mapped node, candidate backup node group, batch size level, hotspot marker, applicable scope of status segment, and reordering marker. Batch size levels can be divided into three levels based on data volume: small, medium, and large. For example, 64 MB to 256 MB is a small batch, 256 MB... Batch sizes from 100 bytes to 1024 megabytes are classified as medium batches, and those larger than 1024 megabytes are classified as large batches. If the current main cause of the correction includes both transmission backlog and partition hotspots, then large batches that have not yet started will be split and accessed first. A single large batch will be broken down into multiple small and medium batches and distributed and mapped to candidate backup node groups, so that the hotspot load will be shifted from centralized backlog to multi-point acceptance. The number of new batches after splitting can be selected as 2 to 5 times the number of original batches, preferably 2 to 3 times. This range is used to balance the distribution effect and scheduling burden. When it is less than 2 times, the distribution effect is usually not obvious. When it is more than 5 times, the scheduling and merging overhead can easily increase significantly. If the current problem mainly manifests as a small number of high-hotspot partitions waiting at the end of the critical path, then the access order of the hotspot partition batches should be moved forward first, while keeping the order of the ordinary partition batches unchanged. The adjustment of the partition access relationship should meet two boundary conditions at the same time: first, the partition batch affiliation chain that has already formed the output should not be changed; second, the target node should not exceed the quota boundary defined in the execution baseline after the adjustment. After the adjustment is completed, a partition access change table should be generated. The partition access change table should include at least the following fields: batch number, original access order, new access order, original mapping node set, new mapping node set, effective time, and corresponding correction number. After adjusting the partition access relationship, the node acceptance relationship is switched. The node acceptance relationship adjustment is based on the stage nature, node fluctuation level, current status segment, and target completion time limit, and tasks are not directly migrated to any idle node. Critical path stages are preferentially mapped to stable node groups. When the remaining basic acceptance quota of stable node groups is insufficient, nodes with load fluctuations below the threshold for the most recent three consecutive sampling periods are selected from the transition node group as supplements. The threshold can be selected as follows: computational usage fluctuation not exceeding 12%, memory usage fluctuation not exceeding 10%, and transmission queuing fluctuation not exceeding 15%. The three consecutive sampling periods are used to exclude misselection caused by accidental stability in a single period. The computational usage fluctuation threshold is higher than the memory usage fluctuation threshold because the local recovery capability of short-term fluctuations in computing resources is stronger. The transmission queuing fluctuation threshold is taken as a relatively intermediate value because link queuing fluctuations have a faster impact on cross-node collaboration. Non-critical path stages are preferentially mapped to transition node groups. High-fluctuation node groups are only allowed temporary access when the execution baseline allows it and the remaining proportion of the target completion time limit is higher than 50%. The 50% threshold is used to ensure that high-fluctuation nodes are only used as constrained buffer resources when there is sufficient remaining processing time. The node acceptance relationship adjustment adopts an incremental switching method. Instead of replacing the entire chain of nodes that are already running stably, the newly added batches are switched first, and then the load change of the original accepting nodes is observed in the next sampling period. If the computing usage ratio, memory usage ratio, or number of pending tasks of the original accepting nodes decreases by more than 10% compared with before the switch, and the number of pending tasks of the target node does not increase for two consecutive sampling periods, the switch range is allowed to be expanded; otherwise, the current switch boundary is maintained. The 10% threshold is used to exclude the interference of slight fluctuations on the judgment of the switch effect. After the node acceptance relationship is adjusted, a node acceptance change table is formed. The node acceptance change table includes at least the following fields: change number, stage number, original accepting node set, new accepting node set, effective start time, effective end time, rollback version number, and number of this switch batch. After the phase advancement relationship, partition access relationship, and node acceptance relationship are adjusted, the cross-node data transmission order is adjusted. The transmission order adjustment aims to prioritize critical path batches, hotspot batches, and batches that directly unblock downstream traffic, rather than simply increasing the total concurrency. Each transmission batch includes at least a transmission timing level field, which can be selected as normal, priority, or urgent. Transmission batches on the critical path whose arrival can directly unblock downstream traffic are promoted to urgent. Transmission batches directly corresponding to hotspot partitions and whose transmission queues have not decreased for two consecutive sampling periods are promoted to priority. The condition of two consecutive sampling periods is used to exclude interference from single-period queue fluctuations on the identification of hotspot batches. Non-critical transmission batches remain at the normal level. During the transmission order adjustment, urgent and priority batches are reserved in the pre-positioning phase. Priority is given to entering the slots for emergency and priority batches. Normal batches can be delayed by 1 to 3 sampling cycles. The delay time corresponds to the link congestion level; the higher the link congestion level, the longer the delay for normal batches. The delay should not exceed 3 sampling cycles to prevent non-critical transmission backlogs from further becoming new sources of mismatch. If a link has reached a critical congestion level, only emergency and priority batches are allowed to enter that link. Normal batches are transferred to backup links or kept in a pending state and reassessed in the next effective period. Backup links can be selected based on the estimated cross-node transmission relationship table in the execution baseline and the current link congestion level. If the target node's receiving queue corresponding to an emergency batch reaches critical congestion during the adjustment period, the batch will be switched from transmission priority to reception priority, i.e., the source end release is temporarily suspended, the target end backlog is cleared first, and then transmission is resumed. The coordination adjustment order must remain unchanged at least, namely, the forward pre-positioning, the forward phase advancement relationship adjustment, the forward phase access relationship adjustment, the forward node acceptance relationship switching, and the forward node transmission order adjustment; if transmission is carried out in advance before the forward phase advancement boundary has converged, the downstream stage may not be able to accept it after the critical batch is moved forward; if the forward node acceptance relationship is switched before the forward node acceptance relationship is adjusted, the forward node may not be able to process the batch that should be processed first. To control new disturbances introduced by coordinated adjustments, two boundary parameters can be set: the maximum adjustment impact area and the minimum adjustment effective amount. The maximum adjustment impact area represents the upper limit of the number of stages, nodes, and batches that can be rewritten simultaneously within a single effective period. Its value can be selected as 10% to 40% of the total number of stages, 10% to 35% of the total number of receiving nodes, and 15% to 45% of the total number of batches to be adjusted, preferably 20%, 25%, and 30%, respectively. The above range is used to prevent the correction action from becoming too large a coverage and becoming a new source of disturbance. The upper limit of the number of stages is lower than the upper limit of the number of batches because the impact area of ​​stage-level rewriting is larger than that of batch-level rewriting, and the upper limit of the number of nodes is lower than that of the number of batches. The upper limit is to prioritize controlling the spread of disturbances at the resource layer; the minimum effective adjustment amount indicates the minimum number of key objects that this coordinated adjustment must successfully affect. Its preferred values ​​are no less than 1 key stage, no less than 2 key nodes, and no less than 1 group of key transmission batches. Here, 1 group of key transmission batches can refer to the set of associated transmission batches from the same source stage to the same target stage. No less than 2 key nodes are used to avoid misjudging a single node local adjustment as having formed a coordinated effect; if the minimum effective adjustment amount is not reached after execution, the current adjustment is considered to have not formed an effective effect, does not enter the formal effective state, but returns to the correction candidate pool, waiting for the next effective period to be rematched; This section outputs the collaborative adjustment execution record, which must include the adjustment number, corresponding correction number, effective start time, effective end time, adjusted stage progress sequence table, adjusted partition access relationship table, adjusted node acceptance relationship table, adjusted transmission sequence table, and explanation of ineffective items. The explanation of ineffective items is used to mark correction items that were not implemented due to insufficient quota, version conflict, boundary restrictions, or priority overlay. After the collaborative adjustment execution record is generated, the job status changes from the "determined to be adjusted" status to the "adjustment pending verification" status. The generated collaborative adjustment execution record serves as the input basis for subsequent execution status verification. Anomalies and boundary handling can be performed according to the following rules: If a node discontinuity anomaly occurs during the node acceptance relationship switchover process, the unfinished switchover action will be stopped and the original acceptance relationship will be maintained; If it is found that the batch to be adjusted has been pre-occupied by a higher priority job during the partition access relationship adjustment process, the batch will retain its original access relationship; If the target link congestion level suddenly increases and the backup link becomes unavailable during the transmission order reordering process, the ordinary level batch will be returned to the waiting state, and the emergency level batch will be retained first; If an adjustment item causes the target node's computational usage ratio, memory usage ratio, or transmission concurrency usage ratio to reach more than 95% of the upper limit of the execution baseline quota during the effective process, the item will enter a forced contraction state, maintaining only the currently effective range; The 95% threshold is used to retain necessary safety margins when approaching the quota boundary to prevent the adjustment scope from continuing to expand.

[0022] S5: Verify the execution status after the collaborative adjustment, and based on the verification results, maintain the current adjustment relationship or perform downgrade or rollback processing. Specifically, the implementation is as follows: A verification window is established based on the actual effective start time in the collaborative adjustment execution record. The verification window is used to receive continuous stage status records after the collaborative adjustment takes effect. Its length can be selected from 2 to 6 sampling periods, preferably 3 to 4 sampling periods. For interactive jobs with short target completion time limits, 2 or 3 sampling periods can be selected. For batch processing jobs with long target completion time limits and large stage spans, 4 to 6 sampling periods can be selected. The above range is determined according to the job timeliness requirements and the observation needs of the adjustment results: if the window is too short, the transient fluctuations after the adjustment may affect the judgment; if the window is too long, the decision to maintain, downgrade, or rollback may lag behind the changes in the problem. The start time of the verification window is preferably set to the next sampling period after the actual effective start time of the collaborative adjustment, so as to avoid including the process of the transition between the old and new relationships at the moment of adjustment switching in the formal judgment. The verification window includes at least the following fields: window number, corresponding adjustment number, start sampling time, end sampling time, set of stage status record numbers included in the window, set of key nodes in the window, set of hot spot partitions in the window, and set of key transmission batches in the window. After the verification window is established, the action effectiveness verification is performed first. The action effectiveness verification is used to confirm whether the adjustment actions recorded as effective in the collaborative adjustment execution record have been actually reflected in the running status. The action effectiveness verification includes at least the following: phase advancement verification, partition access verification, node acceptance verification, and transmission order verification. The phase advancement verification compares the phase advancement sequence table in the collaborative adjustment execution record with the new access rhythm of the corresponding phase in the verification window to confirm whether the conservative advancement rule or yielding rule has changed the expansion speed of the relevant phase. If a yielding rule is applied to a non-critical path phase, the number of new access batches in that phase should decrease compared to before the adjustment in the verification window. The decrease ratio can be selected as 10% to 50% of the original advancement rate, preferably not less than 20%. This range is used to distinguish between slight fluctuations and identifiable advancement changes: when it is less than 10%, it is difficult to distinguish from normal fluctuations; when it is more than 50%, it usually belongs to a strong suppression scenario; preferably not less than 20%, it can balance identification sensitivity and judgment stability. Partition access verification compares the partition access relationship table with the actual batch entry order in the verification window to confirm whether the moved hotspot partition batches have entered the new access order, and whether the split large batches have been started in different target node groups according to the split results; node acceptance verification compares the node acceptance change table with the actual acceptance node distribution of the target stage in the verification window to confirm whether the new acceptance node has received the target stage batches, and whether the original acceptance node has exited the new acceptance as expected; transmission sequence verification compares the adjusted transmission timing table with the batch order in the sending queue and receiving queue in the verification window to confirm whether priority batches and urgent batches have been moved forward; if any adjustment item is not reflected in the running status in the verification window, the item is marked as an ineffective item and written to the ineffective list; the ineffective list includes at least the item number, item type, corresponding stage, reason for not being reflected, and impact scope level; After the action effectiveness verification is completed, the problem convergence verification continues. Problem convergence verification is used to determine whether the adjusted critical state relationship has changed towards convergence. Its judgment content includes at least signals such as changes in mismatch level, changes in blocked tasks in the critical path stage, changes in the overlap of hot spot partition sets, changes in the sending queue and receiving queue, and changes in the deviation of the target completion time limit. The change in mismatch level is compared by re-mapping the running status in the current verification window to the mismatch level caliber in the mismatch judgment stage. It is judged whether the original high-level mismatch has decreased compared with the corresponding mismatch record before the coordinated adjustment, or at least not continued to increase, even if it has not decreased. The change in blocked tasks in the critical path stage is used to determine whether the critical path is still deteriorating. If the number of blocked tasks in the critical path stage does not increase for two consecutive sampling periods in the verification window, or the increase is less than 10%, it can be recorded as the blockage stabilizing. The condition of two consecutive sampling periods is used to exclude the interference of single-period fluctuations on the convergence judgment, and the 10% threshold is used to distinguish between slight fluctuations and continuous growth. Changes in the overlap of hotspot partition sets are used to determine whether a persistent hotspot chain has been broken up. This can be achieved by comparing the hotspot partition set within the current verification window with the hotspot partition set in the mismatch window before collaborative adjustment. If the overlap decreases by 20% to 50%, preferably by 30%, the hotspot concentration is considered to have weakened. This range distinguishes between slight changes and effective convergence; a value below 20% is usually insufficient to indicate that the hotspot chain has been effectively broken up, while a value above 50% often indicates strong convergence. Changes in the sending and receiving queues are used to determine whether the transmission backlog has entered a controllable range. A controllable range does not require immediate recovery to the execution baseline state, but rather requires that the sending and receiving queue lengths no longer increase for two consecutive sampling periods within the verification window, or that the increase is less than 10%. Changes in the deviation from the target completion time limit are used to determine whether the overall completion risk has stopped deteriorating. If the deviation of the expected completion time from the target completion time no longer increases within the verification window, it is considered that the time limit deviation has stabilized. The expected completion time can be estimated based on the current progress speed, remaining task volume, and current transmission backlog status. After the action effectiveness verification and issue convergence verification are completed, a verification result record is generated. The verification result record includes at least the following fields: verification number, corresponding adjustment number, verification window number, action generation efficiency, proportion of ineffective items, convergence judgment result, new anomaly level, processing conclusion, and conclusion effective time. Action generation efficiency represents the proportion of adjustment items actually reflected in the verification window to the total number of adjustment items that should be effective. Its value can be selected from 70% to 90%, preferably not less than 80%. This range is used to distinguish between the state where most adjustments have been implemented and the state where only a few items have taken effect: when it is less than 70%, the manifestation of adjustment items in the running state is usually insufficient to support the maintenance judgment; when it is greater than 90%, it can be regarded as most key items have entered the actual effective state. The proportion of ineffective items represents the proportion of the number of ineffective items to the number of key adjustment items. The processing conclusions can be divided into three categories: maintaining the current adjustment relationship, performing downgrading processing, and performing rollback processing. When the action efficiency is not less than 80%, the original mismatch level decreases, or although it does not decrease, it at least does not continue to worsen, and no new level 3 or 4 mismatches appear, the processing conclusion is recorded as maintaining the current adjustment relationship. If the action has partially taken effect, but the problem convergence is limited, or some items have a suppressive effect while some items introduce new mild to moderate abnormal effects, the processing conclusion is recorded as performing downgrading processing. If the proportion of ineffective items exceeds a preset threshold, or although the action has taken effect, the original mismatch level continues to rise, or new severe mismatches are introduced, the processing conclusion is recorded as performing rollback processing. The threshold for the proportion of ineffective items can be set to 20% to 50%, preferably 30%. This range is used to distinguish between local ineffectiveness and overall insufficient implementation: when it is below 20%, it can usually still be observed by maintaining or partially shrinking; when it is above 50%, it indicates that the current adjustment relationship is not fully implemented. The preferred value of 30% is suitable for the boundary judgment of maintenance, downgrading, and rollback in most common scenarios. When the processing conclusion points to a downgrade, the process enters the downgrade path. Downgrade processing aims to reduce the impact of current adjustments while maintaining continuous operation, retaining valid entries and revoking those with significant side effects. Downgrade processing can be executed in a fixed order: first, revoking partition splitting and access actions with a large impact; then, revoking overly strong suppression rules on non-critical path stages; subsequently, assessing whether node acceptance relationships are limited to the currently successfully switched node groups; and finally, retaining the transmission priorities and necessary node acceptance relationships related to the critical path. This order is used because partition splitting and access actions typically affect many objects, and continuously retaining them can easily amplify queuing chains; persistent overly strong suppression rules on non-critical path stages may create new progress waste; and excessively rapid expansion of node acceptance relationships may also increase buffer pressure. After downgrading, a downgrade version record is generated. The downgrade version record includes at least the following fields: downgrade number, original adjustment number, set of retained items, set of revoked items, downgrade start time, and next verification window number. The next verification window after downgrading can be selected as 1 to 3 sampling periods, preferably 2 sampling periods, in order to quickly determine whether the adjustment relationship after contraction tends to stabilize. If no one of the following is found within two consecutive verification windows after downgrading: blockage stabilization, weakening of hotspot concentration, or deviation from stabilization in time limit, then the local reduction is stopped and the process is rolled back. The setting of two consecutive verification windows is used to distinguish between short-term fluctuations and a state of no continuous improvement. When the processing conclusion selects rollback, the rollback path is entered. Rollback is based on the previous stable version and execution baseline, systematically restoring completed relationship changes. Rollback targets include at least incomplete partition access relationships, unstable node acceptance relationships, unsent transmission order changes, and non-critical path stage progression restrictions. Completed partition batches, sent transmission batches, and output stage results are not considered for rollback to avoid secondary disturbances. The preferred rollback order is to first restore stage progression restrictions, then transmission order, then node acceptance relationships, and finally partition access relationships. The reason for the order is that if the partition access relationship is restored first while the transmission order is still under adjustment, the restored batch will still have an unreasonable transmission rhythm. If the node acceptance relationship is restored first while the stage advancement restriction is not restored, the original acceptance node will again be occupied by too many non-critical path stages. During the rollback process, the rollback number and corresponding adjustment number include the rollback object set, the version number before rollback, the version number after rollback, the rollback start time, the rollback completion time, the rollback reason, etc. After the job rollback is completed, it enters a re-observation state where the rolled-back version is the current valid version, and uses this version as a reference for subsequent running status collection and mismatch judgment. To clarify the state transition relationships after performing status verification, the operation states are defined as verification state, maintenance state, degraded state, rollback state, and re-observation state. Verification state refers to the current operation being in the verification window or during the verification result generation stage; maintenance state refers to the current adjustment relationship still being valid; degraded state refers to the current adjustment relationship being reduced to a version with a smaller impact; rollback state refers to the current adjustment relationship being restored to the previous version; and re-observation state refers to the current version entering the subsequent observation stage. The state transition rules are as follows: when the verification result meets the maintenance condition, the operation transitions from verification state to maintenance state; when the verification result meets the degraded condition, the operation transitions from verification state to degraded state; when the verification result meets the rollback condition, the operation transitions from verification state to rollback state; when the current version after maintenance, degrade, or rollback is stable in the next verification window, the operation transitions from the corresponding state to re-observation state. When a new mismatch candidate appears again in the re-observation state, the mismatch determination procedure is initiated. This section sets time and resource constraints. Regarding time constraints, verification judgments should be completed within one sampling period after the verification window ends, preferably within 50% of the sampling period after the verification window ends. For example, when the sampling period is 6 seconds, the formal processing conclusion should ideally be formed within 3 seconds to avoid delays in maintenance, downgrade, or rollback decisions affecting the next round of sampling and judgment. Regarding resource constraints, the auxiliary resource overhead introduced by verification calculations, version switching, and status record organization should not exceed 5% of the current job's stable-state resource usage, preferably not exceeding 3%. When node resources reach more than 95% of the upper limit of the execution baseline quota, verification should be completed using existing status record reuse and logical comparison methods, without adding additional high-frequency detection actions. The 5% and 3% ranges are used to control additional verification overhead and avoid significant disturbances to the observed job during the verification process. The 95% threshold is used to characterize that node resources are close to the execution baseline safety boundary; at this point, further high-frequency detection can easily amplify resource fluctuations. Boundary and anomaly handling can be performed according to the following rules: When more than 50% of the sampled data is missing within the verification window, a maintenance decision should not be made directly, but rather a downgrade should be prioritized, or the current version should be maintained and a re-observation state should be entered; the 50% threshold is used to distinguish between local data loss and insufficient overall verification basis, and a stable maintenance judgment should not be made directly when this proportion is exceeded; if no one of the following occurs within two consecutive verification windows after downgrading: blockage stabilization, weakening of hotspot concentration, or deviation from stabilization within the time limit, a forced rollback should be triggered; after rolling back to the previous stable version, if the same type of level 3 or higher data still occurs within two consecutive verification windows... When a Level 4 mismatch occurs, the mismatch mode is recorded as a restricted mode, and the same adjustment combination is prohibited from being reused within the remaining execution cycle of the current job. The conditions of two consecutive verification windows are used to distinguish between a single abnormal rollback and a persistent failure mode. The restricted mode record includes at least the following fields: mode number, corresponding mismatch type, prohibited adjustment item combination, number of triggers, and failure conditions. Failure conditions may include that no corresponding mismatch has occurred in several consecutive verification windows, or that the current job has entered a new baseline version. Verification result records, maintenance records, downgrade version records, or rollback records are output as version references for subsequent processing.

[0023] The technical solution of this embodiment takes the joint analysis of daily transaction details, user behavior logs, and inventory flow on an e-commerce platform at night as an example. In the job access phase, an execution baseline is first defined based on the logical execution plan, stage dependencies, and node resource status. The input / output boundaries, node capacity, and estimated data transmission relationships for each stage are defined. During the job's operation, the progress status of each stage, node load status, and cross-node data transmission status are collected according to the sampling period, generating stage status information. When a certain aggregation stage experiences hotspot partition backlog, increased critical node load, and gradual disappearance of cross-node transmission, it is determined whether there is a mismatch in the logical execution plan, runtime resource status, and cross-node data transmission status. Linkage correction information and effective periods for each stage are generated. Then, based on pre-positioning, the stage progress order, hotspot partition access order, node capacity, and critical batch transmission order are adjusted collaboratively. After the adjustment takes effect, a verification window is opened to check whether each adjustment item has actually been adjusted, whether the original mismatch has begun to converge, and whether new anomalies have occurred. Based on the verification results, the original adjustment relationship is maintained, or downgrade or rollback processing is performed, enabling the job to complete analysis and processing relatively stably even under conditions of node load fluctuations and data skew.

[0024] It should be noted that this invention can be deployed on the device itself to realize embedded applications, or it can run on a PC or other terminal with a user interface, thereby meeting various hardware environments and usage requirements.

[0025] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented in whole or in part by a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions of the embodiments of this application are implemented in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted wirelessly or wiredly from one website, computer, server, or data center to another website, computer, server, or data center. Wired methods include optical fiber, twisted pair, coaxial cable, etc. Wireless methods include infrared, microwave, etc. Available media include any available media that can be accessed by a computer or data storage devices such as servers and data centers that contain one or more sets of available media. Available media can be magnetic media (floppy disks, hard disks, magnetic tapes), optical media (DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0026] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0027] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A performance optimization method for a distributed data analysis system, characterized in that, include: S1: Receives data analysis jobs to be executed and establishes an execution baseline based on the logical execution plan, stage dependencies, and initial resource allocation relationships; S2: During the execution of the data analysis job, the running status, node load status and cross-node data transmission status of each execution stage are collected according to the preset sampling period, and corresponding stage status information is generated. S3: Determine whether there is a mismatch between the logical execution plan, runtime resource status and cross-node data transmission status based on the stage status information, and generate linkage correction information when a mismatch is determined, and determine the effective period of the linkage correction information. S4: Based on the effective period, coordinate the order of execution phases, partition access relationships, node acceptance relationships, and cross-node data transmission order; S5: Verify the execution status after the collaborative adjustment, and maintain the current adjustment relationship or perform downgrade or rollback processing based on the verification results.

2. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, Receive data analysis jobs to be executed, and establish an execution baseline based on the logical execution plan, stage dependencies, and initial resource allocation relationships, including: Data analysis jobs are accessed and parsed to generate job access records and stage relationship tables; Based on the phase relationship table, establish phase dependency mapping, node group acceptance mapping, node quota boundaries, and estimated cross-node transmission relationships, and form an execution baseline record.

3. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, During the execution of the data analysis job, the running status, node load status, and cross-node data transmission status of each execution stage are collected according to a preset sampling period, including: The sampling period is determined based on the execution baseline record, and the progress status, node carrying status and transmission association status are collected at each sampling time. The current state segment is determined based on the input arrival rate, output completion rate, and stage progress speed.

4. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, Generate the corresponding stage status information, including: The stage operation records, node load records, and transmission status records at the same sampling time are merged to generate stage status information; The phase status information contains a set of key nodes, a set of hotspot partitions, a node load summary, and a transmission status summary. And perform consistency verification on the stage status information.

5. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, Based on the stage status information, determine whether there is a mismatch between the logic execution plan, runtime resource status, and cross-node data transmission status, including: The stage status information is mapped to the execution baseline record to establish a mismatch candidate window; Based on the mismatch candidate window, plan mismatch candidates, resource mismatch candidates, and transmission mismatch candidates are generated; Mismatch determination records are generated based on the relationships, persistence status, and impact levels among the mismatch candidates.

6. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, When a mismatch is detected, linkage correction information is generated, and the effective period of the linkage correction information is determined, including: Based on the mismatch determination record, linkage correction information is generated. The linkage correction information contains a set of correction items, effective start conditions, effective end conditions, maximum duration of action, verification items, and rollback trigger conditions. The effective period of the linkage correction information is determined by combining the state segment boundary and the sampling period boundary; The legality of the linked correction information is also checked.

7. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, Based on the effective period, the execution phase's advancement order, partition access relationships, node succession relationships, and cross-node data transmission order will be coordinated and adjusted, including: Load the linkage correction information before execution to form a correction execution order and an adjustment object mapping table; Implement quota freezing, link slot reservation, and batch version locking; The process involves adjusting the phased access relationships, switching the node acceptance relationships, and rearranging the transmission order in a fixed sequence. Control the scope of impact, the amount of effect, and the boundaries of anomalies in the adjustment; It also generates collaborative adjustment execution records.

8. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, Verify the execution status after the collaborative adjustment, including: Establish a verification window based on collaborative adjustment execution records; Within the verification window, the actual implementation of the adjustment items and the convergence of key state relationships are verified. Verification result records are generated based on the verification results. The verification result records contain the action generation efficiency, the proportion of ineffective items, the convergence judgment results, the processing conclusions, and the newly added anomaly levels.

9. The performance optimization method for a distributed data analysis system according to claim 1, characterized in that, Depending on the verification results, the current adjustment relationship may be maintained, or a downgrade or rollback may be implemented, including: The maintenance path, downgrade path, or rollback path will be determined based on the verification results record; Generate corresponding maintenance records, downgrade version records, or rollback records; Switch job status according to the status transition rules; The current valid version will be used as the version reference for subsequent running status collection and mismatch determination.

Images