A method and device for cross-supercomputing center big data workflow collaborative computing

Abstract

The application provides a cross-supercomputing center big data workflow collaborative computing method and device, relates to the technical field of distributed computing and workflow scheduling, and comprises the following steps: obtaining a workflow model to be executed, the workflow model defining a plurality of computing tasks and data dependency relationships between the tasks; obtaining resource state information, data distribution information and historical execution statistical information of a plurality of supercomputing centers; constructing a cost prediction model for predicting task execution time consumption and data transmission time consumption based on the workflow model and the historical execution statistical information; and distributing the plurality of computing tasks to corresponding supercomputing centers based on the resource state information, the data distribution information and the cost prediction model, and generating a scheduling scheme containing task scheduling instructions and data pre-scheduling instructions. The application provides a reliable basis for prospective scheduling decision, thereby avoiding resource waiting or invalid pre-transmission caused by inaccurate estimation.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and workflow scheduling technology, and more specifically, to a method and apparatus for collaborative computing of big data workflows across supercomputing centers. Background Technology

[0002] With the rapid growth in data scale and computational complexity in fields such as scientific research, engineering simulation, and artificial intelligence training, the storage and computing power of a single supercomputing center can no longer meet the needs of ultra-large-scale workloads. Joint execution of big data workflows across multiple heterogeneous supercomputing centers has become an important development trend. Such workflows are usually modeled using directed acyclic graphs, where nodes represent tasks that carry out computational logic, edges represent data dependencies between tasks and large-scale datasets that need to be transferred across nodes, and when executed across centers, the efficiency of wide area network data transmission between centers directly determines the overall completion time of the workflow.

[0003] In existing technologies, the common approach for workflow scheduling across supercomputing centers is as follows: First, obtain the availability status of computing resources and the distribution location of the required datasets in each supercomputing center. Then, statically allocate each computing task in the workflow to different centers based on strategies such as proximity or load balancing. For tasks with data dependencies that are allocated to different centers, after the upstream source task is completed, the scheduling system establishes a connection between the source supercomputing center and the target supercomputing center and transmits all the output data generated by the source task to the target center. Once all the data arrives, the downstream task is triggered to start execution. At the same time, some systems introduce task execution time and data transmission time estimation based on experience or simple models to assist in generating the above scheduling scheme.

[0004] However, the existing technical solutions mentioned above still have some shortcomings: since supercomputing centers are mostly interconnected via wide area networks or cross-domain leased lines, the network bandwidth is relatively limited and the latency is high. The waiting time generated by the full data transmission causes the downstream tasks of the target center to be idle for a long time, resulting in a significant increase in the overall completion time of the workflow. This scheduling mode relies on the serial triggering logic of "source task completes - full data transfer - downstream task starts", which fails to take advantage of the pipeline overlap characteristics of upstream tasks being able to advance downstream processing after generating some data. The deep parallel space between computing and cross-center data transmission is wasted. In addition, there is a lack of accurate prediction methods for task execution progress, data output rate and network dynamics. The scheduling decision is difficult to determine the optimal data pre-transmission time and task coordination rhythm, and cannot fundamentally alleviate the performance bottleneck caused by cross-center data dependence.

[0005] Therefore, we have made improvements to this by proposing a collaborative computing method and device for big data workflows across supercomputing centers. Summary of the Invention

[0006] The purpose of this invention is to address the current limitations in the overall workflow completion time. This is because interconnections between supercomputing centers are often via wide area networks or dedicated cross-domain lines, resulting in relatively limited network bandwidth and high latency. The waiting time caused by full data transmission leaves downstream tasks at the target center idle for extended periods. Furthermore, the current scheduling model relies on a sequential triggering logic of "source task complete—full data transfer—downstream task start," failing to leverage the pipeline overlap characteristic where upstream tasks can advance downstream processing after generating partial data. This wastes the deep parallel space between computation and cross-center data transmission. In addition, the lack of precise prediction methods for task execution progress, data output rate, and network dynamics makes it difficult to determine the optimal data pre-transmission timing and task coordination rhythm, thus failing to fundamentally alleviate the performance bottleneck caused by cross-center data dependency.

[0007] To achieve the above-mentioned objectives, this invention provides a collaborative computing method and apparatus for big data workflows across supercomputing centers, in order to improve the aforementioned problems.

[0008] The application is as follows:

[0009] A collaborative computing method for big data workflows across supercomputing centers is proposed, which obtains a workflow model to be executed, wherein the workflow model defines multiple computing tasks and the data dependencies between tasks.

[0010] Obtain resource status information, data distribution information, and historical execution statistics from multiple supercomputing centers;

[0011] Based on the workflow model and the historical execution statistics, a cost prediction model is constructed to predict task execution time and data transmission time.

[0012] Based on the resource status information, the data distribution information, and the cost prediction model, the multiple computing tasks are allocated to corresponding supercomputing centers, generating a scheduling scheme that includes task scheduling instructions and data pre-scheduling instructions; wherein, for source tasks and target tasks that have data dependencies and are allocated to different supercomputing centers, the data pre-scheduling instructions are used to transmit at least a portion of the data required by the target task from the source supercomputing center to the target supercomputing center before the source task is completed.

[0013] According to the scheduling scheme, the multiple supercomputing centers coordinate to execute the computing tasks and implement the data pre-scheduling instructions.

[0014] The preferred technical solution of this application, which involves obtaining resource status information, data distribution information, and historical execution statistics information of multiple supercomputing centers, includes:

[0015] Through monitoring agents deployed in various supercomputing centers, the utilization rates of computing resources, storage resources, network bandwidth status, and the storage location and copy information of datasets are periodically collected; and

[0016] Extract the task execution time, data output rate, and cross-center data transmission throughput of completed workflow instances from the execution log as historical execution statistics.

[0017] The preferred technical solution of this application describes a cost prediction model for predicting task execution time and data transmission time, comprising:

[0018] A regression model is used to predict task execution time, with task type, input data size, and available computing resources of the allocated supercomputing center as input features.

[0019] Using the amount of data to be transmitted and the available bandwidth of the current cross-center network as input features, the data transmission time is predicted.

[0020] As a preferred technical solution of this application, the generation of a scheduling scheme including task scheduling instructions and data pre-scheduling instructions includes:

[0021] With the goal of minimizing the estimated total completion time of the workflow, the scheduling scheme is obtained by searching the allocation mapping of tasks to supercomputing centers and the start timing of pre-scheduling of data across centers;

[0022] The data pre-scheduling instruction includes a pre-transmission start threshold, which indicates that data pre-transmission is triggered when the output data generation amount of the source task reaches a preset condition.

[0023] The implementation of the data pre-scheduling instruction, as a preferred technical solution of this application, includes:

[0024] During the execution of the source task, monitor the amount of output data generated by the source task;

[0025] When the amount of output data reaches the pre-transmission start threshold, data transmission from the source supercomputing center to the target supercomputing center is initiated. The data transmission is used to transmit the currently generated portion of the output data.

[0026] As a preferred technical solution of this application, the process of coordinating the multiple supercomputing centers to execute the computing task and implement the data pre-scheduling instructions according to the scheduling scheme further includes:

[0027] Obtain real-time resource status and task execution progress of each supercomputing center;

[0028] When the deviation between the actual execution progress and the predicted progress of the cost prediction model exceeds a preset threshold, the computational tasks that have not yet started execution are reallocated based on the updated resource status information and the cost prediction model, and the corresponding data pre-scheduling instructions are updated.

[0029] The preferred technical solution in this application includes the following data transmission process across supercomputing centers:

[0030] The data to be transmitted is fragmented.

[0031] The fragmented data is transmitted through multiple parallel transmission channels;

[0032] The fragmented data is reassembled at the receiving end.

[0033] The data transmission process across supercomputing centers, as a preferred technical solution of this application, further includes:

[0034] After fragmentation and before transmission, the fragmented data is compressed and encrypted.

[0035] Before reassembly at the receiving end, the fragmented data is decrypted and decompressed.

[0036] The workflow model described in the preferred technical solution of this application is a directed acyclic graph, in which nodes represent computational tasks, edges represent data dependencies and the identifiers of the datasets being passed.

[0037] The coordinated execution of the computing tasks by the multiple supercomputing centers includes: the central scheduling node issuing task execution instructions to each supercomputing center, and each supercomputing center executing local computing tasks according to the instructions; when cross-supercomputing center data transmission is required, a data transmission connection is directly established between the two supercomputing centers involved.

[0038] A cross-supercomputing center big data workflow collaborative computing device, used in the aforementioned cross-supercomputing center big data workflow collaborative computing method, includes:

[0039] The workflow acquisition unit is used to acquire the workflow model to be executed, wherein the workflow model defines multiple computing tasks and the data dependencies between the tasks;

[0040] The information acquisition unit is used to acquire resource status information, data distribution information, and historical execution statistics information from multiple supercomputing centers.

[0041] The model building unit is used to build a cost prediction model for predicting task execution time and data transmission time based on the workflow model and the historical execution statistics.

[0042] The scheduling generation unit is used to allocate the multiple computing tasks to corresponding supercomputing centers based on the resource status information, the data distribution information, and the cost prediction model, and to generate a scheduling scheme that includes task scheduling instructions and data pre-scheduling instructions; wherein, for source tasks and target tasks that have data dependencies and are allocated to different supercomputing centers, the data pre-scheduling instructions are used to transmit at least a portion of the data required by the target task from the source supercomputing center to the target supercomputing center in advance before the source task is completed;

[0043] The collaborative execution unit is used to coordinate the multiple supercomputing centers to execute the computing tasks and implement the data pre-scheduling instructions according to the scheduling scheme.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] In the scheme of this application:

[0046] 1. By acquiring historical execution statistics and constructing a cost prediction model based on the workflow model to predict task execution time and data transmission time, the scheduling system's estimation of task running status and network transmission status is improved from static experience judgment to dynamic and accurate prediction based on historical data. This provides a reliable basis for forward-looking scheduling decisions and avoids resource waiting or invalid pre-transmission caused by inaccurate estimation.

[0047] 2. By generating a scheduling scheme that includes data pre-scheduling instructions, and by pre-transferring at least a portion of the data required by the target task from the source supercomputing center to the target supercomputing center before the source task is completed, the serial logic of "source task completes - full data transfer - downstream task starts" is broken. As a result, the data output process of the source task and the cross-center data transmission process can be carried out in parallel and overlapped, and the remaining computation of the source task can be performed synchronously with the data transmission.

[0048] 3. The scheduling scheme aims to minimize the estimated total completion time of the workflow. While searching for task allocation mapping, it determines the start time of pre-scheduling for each cross-center data. In particular, it sets a pre-transfer start threshold that matches the predicted output of the cost prediction model. This ensures that the pre-transfer operation starts at the right time, neither too early (wasting network bandwidth and remote storage) nor too late (pipeline stall). It incorporates the progress of the source task's output data with the actual time taken for cross-center network transmission into a unified coordination, thereby enabling the deep pipeline overlap of computation and transmission to be achieved in the optimal state.

[0049] 4. During collaborative execution, by acquiring the real-time resource status and task execution progress of each supercomputing center, and comparing the deviation between the actual execution progress and the predicted progress of the cost prediction model, when the deviation exceeds a preset threshold, the computing tasks that have not yet started execution are reallocated and the corresponding data pre-scheduling instructions are updated. This feature transforms the entire scheduling system from a one-time static planning to a closed-loop adaptive dynamic adjustment system, which can resist the uncertainty in actual operation and maintain the prediction-pre-scheduling-execution-rescheduling technical link in a highly efficient state, further ensuring and improving the stability of the overall workflow execution efficiency. Attached Figure Description

[0050] Figure 1 A flowchart illustrating a collaborative computing method for big data workflows across supercomputing centers provided in this application;

[0051] Figure 2 A schematic diagram of the structure of a cross-supercomputing center big data workflow collaborative computing device provided in this application;

[0052] Figure 3 A directed acyclic diagram illustrating a collaborative computing method for big data workflows across supercomputing centers provided in this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0054] The present invention will be further described below with reference to embodiments.

[0055] Example: Refer to Figures 1 to 3 As shown in the figure, this embodiment provides a collaborative computing method and device for big data workflows across supercomputing centers. Taking a gene sequencing data analysis workflow across three supercomputing centers as an example, the three supercomputing centers are center A deployed in location A, center B deployed in location B, and center C deployed in location C. They are interconnected through cross-domain dedicated lines. Each center is equipped with a monitoring agent to collect resource status, data distribution, and task execution logs.

[0056] Step 1: The central scheduling node obtains the workflow model to be executed. The workflow model defines multiple computing tasks and the data dependencies between tasks.

[0057] Preferably, the central scheduling node receives the gene sequencing analysis workflow description file submitted by the user. This file is represented by a directed acyclic graph (DAG), which contains five nodes corresponding to five computational tasks: T1 (data preprocessing), T2 (sequence alignment), T3 (variant detection), T4 (functional annotation), and T5 (result aggregation). The edges in the graph define the data dependencies between tasks and the identifiers of the datasets to be transferred: T1→T2 transfers dataset DS1 (estimated output size approximately 500GB), T2→T3 transfers dataset DS2 (estimated output size approximately 300GB), T2→T4 transfers dataset DS3 (estimated output size approximately 200GB), T3→T5 transfers dataset DS4 (estimated output size approximately 50GB), and T4→T5 transfers dataset DS5 (estimated output size approximately 80GB).

[0058] The mutation detection tool used in Task T3 and the function annotation tool used in Task T4 both support streaming input processing. They can start execution by receiving and processing data simultaneously even when the input data is not fully available. The central scheduling node parses the description file and extracts the task list and dependent topology.

[0059] By obtaining complete structural information of the workflow through the scheduling system, and obtaining key characteristics of whether downstream tasks support streaming processing, we can lay the foundation for subsequent cost prediction and pre-scheduling decisions.

[0060] Step 2: The central scheduling node obtains resource status information, data distribution information, and historical execution statistics information from multiple supercomputing centers through the monitoring agent.

[0061] The monitoring agents deployed in centers A, B, and C collect data every 30 seconds. This data includes the total number of CPU cores and the number of currently available cores, the total number of GPU cards and the number of currently available GPU cards, the total capacity and used capacity of the parallel file system, and the current available bandwidth of the dedicated line between the center and other centers. At the same time, the agents scan the storage location and copy information of each dataset in the file system to form data distribution information. The above information is reported to the central scheduling node in real time via a message queue.

[0062] Meanwhile, the central scheduling node extracts historical execution statistics of workflow instances completed in the past 90 days from the job execution log database of each supercomputing center. Specifically, this includes: the task type of each computing task (identified by "data preprocessing", "sequence alignment", "mutation detection", "functional annotation", and "result aggregation"), the input data size (GB) of each execution, the number of CPU cores and GPUs allocated, the execution time (seconds) of the task from start to finish, the output data production rate (MB / s) of the task in each time window, and the actual throughput (MB / s) and total transmission time (seconds) of each cross-center data transfer. A total of 1,276 historical task execution records and 438 historical cross-center data transfer records are extracted.

[0063] Multi-dimensional historical and real-time data provide ample data support for the training and scheduling decisions of the cost prediction model.

[0064] Step 3: Based on the workflow model and the historical execution statistics, the central scheduling node constructs a cost prediction model to predict the task execution time and data transmission time.

[0065] The central scheduling node uses the historical execution statistics extracted in step two to construct two regression prediction sub-models for the task types involved in the workflow model.

[0066] Sub-model one is a task execution time prediction model, employing the XGBoost regression algorithm. Input features include: one-hot encoding of the task type (dimension 5), input data size (continuous value, GB), number of available CPU cores in the allocated supercomputing center (continuous value), and number of available GPUs (continuous value). The output is the predicted task execution time, in seconds. Using 80% of historical samples as the training set, 5-fold cross-validation is performed, and the hyperparameter combination that minimizes the mean absolute percentage error is selected.

[0067] Sub-model two is a data transmission time prediction model, employing a linear regression algorithm. Input features include: the amount of data to be transmitted (continuous value, GB) and the currently available network bandwidth between the source and target centers (continuous value, MB / s). The output is the predicted data transmission time, in seconds.

[0068] The cost prediction model trained on historical data can dynamically and accurately predict the time consumption of each stage based on the current resource status and task scale, providing a precise cost assessment basis for subsequent search for the optimal scheduling scheme.

[0069] Step 4: Based on the resource status information, the data distribution information, and the cost prediction model, the central scheduling node allocates the multiple computing tasks to the corresponding supercomputing centers and generates a scheduling scheme that includes task scheduling instructions and data pre-scheduling instructions. Among them, for source tasks and target tasks that have data dependencies and are allocated to different supercomputing centers, the data pre-scheduling instructions are used to transmit at least a portion of the data required by the target task from the source supercomputing center to the target supercomputing center before the source task is completed.

[0070] Preferably, the central scheduling node searches for the optimal scheduling scheme with the goal of minimizing the estimated total completion time of the workflow. Preferably, the search process employs the simulated annealing algorithm, with the state variable being the allocation mapping vector from each task to the supercomputing center. ,in Indicates task The task is assigned to center A, B, or C. The initial solution is generated according to the principle of data proximity: the task is preferentially assigned to the center that has its input dataset; if there is no clearly defined nearest center, it is randomly assigned.

[0071] In each iteration, for the current mapping Generate new mappings by performing random perturbations For each mapping, the cost prediction model is invoked to calculate the estimated total completion time of the workflow. The calculation process is as follows:

[0072] Traverse the topological order of the directed acyclic graph to determine the earliest start time of each task. and earliest completion time For tasks that do not have cross-center data dependencies, ,in The predicted task execution time for sub-model 1; for source tasks with cross-center data dependencies. and target task (Right now This involves calculating the data pre-scheduling start timing for the dependent edge. Specifically, a pre-transmission start threshold is defined. This indicates that when the cumulative generation ratio of the source task's output data reaches... Data pretransmission is triggered at that time.

[0073] Preferred, Determined by the following formula:

[0074]

[0075] in, and These are the predicted execution times for the source task and the target task, respectively. The predicted cross-center data transmission time for sub-model 2 (based on the total amount of data output by the source task and the current available bandwidth) enables transmission to start as early as possible after the source task produces data, while avoiding premature transmission that would cause the remote end to wait idly.

[0076] After pre-transfer is initiated, the remaining data output from the source task and cross-center data transmission occur in parallel. This applies to target tasks that support streaming processing. It can start execution after receiving the pre-transmitted initial data block, and subsequent data is continuously input during the process; based on this, Earliest start time "The source task execution progress has reached..." The later of the "time of completion" and the "time of completion of the initial pre-transmission data block" is used. The entire graph is traversed in this manner, and finally, the largest value of the last task is taken. Used as the estimated total completion time for the workflow.

[0077] If new mapping If the estimated total completion time is less than the estimated total completion time of the current mapping, then accept. For the new current solution; otherwise, use probability. Accept, among which This is the time increment for completion. Temperature parameter. The initial value is 1000, the decay factor is 0.95, and the termination condition is no improvement after 500 consecutive iterations or It dropped below 0.1.

[0078] The optimal mapping is: T1→Center A, T2→Center A, T3→Center B, T4→Center C, T5→Center B. The scheduling scheme includes: issuing scheduling instructions for computational tasks T1 and T2 to Center A; issuing scheduling instructions for computational tasks T3 and T5 to Center B; issuing scheduling instructions for computational tasks T4 to Center C; and simultaneously, generating data pre-scheduling instructions for cross-center data dependency T2→T3: during the execution of T2 in Center A, when the generated proportion of the T2 output dataset DS2 reaches... At that time, DS2 transmission is initiated from center A to center B; in this example, it is calculated according to the formula. The threshold is 0.35, meaning it's triggered when the cumulative output data volume reaches 105GB. Similarly, for T2→T4, when the generated proportion of the T2 output dataset DS3 reaches 0.35, a transmission to center C is initiated. Regarding the dependencies T3→T5 and T4→T5, since T5 and T3 are both at center B, the data transmission from T4 to T5 is executed normally after T4 is completed.

[0079] By using a cost prediction model to guide task allocation and pre-scheduling timing, the timing of cross-center transmission initiation can be precisely matched with the data output progress of the source task, thus avoiding both premature transmission that wastes bandwidth and late transmission that causes pipeline stalls.

[0080] Step 5: The central scheduling node, according to the scheduling scheme, coordinates with the multiple supercomputing centers to execute the computing tasks and implement the data pre-scheduling instructions.

[0081] The central scheduling node sends task scheduling instructions and data pre-scheduling instructions to each center through a secure channel. Upon receiving the instructions, the job scheduler at each center starts the corresponding computation task.

[0082] Taking cross-center data pre-scheduling from T2 to T3 as an example, the monitoring agent of center A monitors the amount of data in the output directory of T2 once per second during the execution of T2. When the cumulative amount of generated data in DS2 reaches 105GB (i.e., 300GB × 35%), a data pre-transmission operation is triggered. Preferably, the data to be transmitted is fragmented, with the fragment size set to 64MB, resulting in approximately 1640 fragments. More preferably, each fragment is sequentially compressed using gzip (compression level set to 6) and encrypted using AES-256-GCM. Then, the fragmented data is transmitted to the data receiving service of center B through eight parallel TCP connection channels. At the receiving end, center B sequentially decrypts the fragments using AES-256-GCM and decompresses them using gzip, then reassembles them according to their sequence numbers and writes them to the local parallel file system. During and after the initial 105GB data transfer, the remaining 195GB of data continuously generated by T2 is appended to the center B in real time using the same fragmentation, compression, encryption, and multi-channel transmission methods until T2 finishes execution and all data has been transferred.

[0083] T3 is a streaming mutation detection task that is started after receiving the initial recombination data, and subsequent data streams are continuously fed into the task process.

[0084] By fragmenting, compression, and multi-channel parallel transmission, the throughput of a single cross-center data stream is improved and the total network transmission volume is reduced; encryption ensures the confidentiality of data in cross-domain transmission, and the combination of data pre-scheduling and streaming task processing realizes deep pipeline overlap of source task computation, cross-center data transmission, and downstream task computation.

[0085] Step 6: The central scheduling node obtains the real-time resource status and task execution progress of each supercomputing center. When the deviation between the actual execution progress and the predicted progress of the cost prediction model exceeds a preset threshold, the computing tasks that have not yet started execution are reallocated, and the corresponding data pre-scheduling instructions are updated.

[0086] During collaborative execution, the central scheduling node retrieves the current resource status and the completion percentage of each task from each monitoring agent every 60 seconds. Simultaneously, the scheduling node compares the measured completion percentage with the expected progress sequence of each task corresponding to the cost prediction model generated in step four when the scheduling plan is generated. Preferably, the deviation calculation formula is:

[0087]

[0088] The preset deviation threshold is 15%.

[0089] Suppose that at a certain point in the process, the number of available GPUs at center B drops from 8 to 4 due to a fault, significantly extending the estimated execution time of task T3, which has not yet started. The actual deviation between the measured progress and the predicted progress of T3 reaching 22%, exceeding the preset threshold, triggers dynamic rescheduling: task T3, which has not yet started, is reassigned from center B to center C, and the task scheduling instructions and data pre-scheduling instructions are recalculated based on the updated resource status (center C currently has 6 available GPUs) and the cost prediction model. After recalculation, the pre-transmission start threshold for T2→T3 is adjusted to 0.45, and the number of parallel channels is adjusted to 12 (due to the higher available bandwidth between center A and center C). Some of the transmitted data remains at center B but does not affect subsequent processes; since T3 has not yet started, the central scheduling node issues updated data pre-scheduling instructions to centers A and C, and subsequent data is directly transmitted from center A to center C according to the new threshold and the new number of channels.

[0090] Through real-time monitoring, deviation comparison, and dynamic rescheduling, the scheduling system has the ability to adaptively adjust to counteract fluctuations in resource status and prediction deviations during operation, ensuring the stability of the overall execution efficiency of the workflow.

[0091] Step 7: Workflow execution complete, central scheduling node summarizes results.

[0092] After all tasks in the directed acyclic graph are completed, the central scheduling node receives the task completion confirmation and result data storage location reported by each center. Task T5 produces the final aggregated result, and the scheduling node returns the result access path to the user.

[0093] By using prediction-driven pre-scheduling, streaming task overlap, and dynamic adaptive adjustment, the overall completion time of big data workflows across supercomputing centers has been significantly shortened, and the efficiency of cross-center resource utilization has been improved.

[0094] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0095] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A collaborative computing method for big data workflows across supercomputing centers, characterized in that, Includes the following steps: Obtain the workflow model to be executed, which defines multiple computational tasks and the data dependencies between the tasks; Obtain resource status information, data distribution information, and historical execution statistics from multiple supercomputing centers; Based on the workflow model and the historical execution statistics, a cost prediction model is constructed to predict task execution time and data transmission time. Based on the resource status information, the data distribution information, and the cost prediction model, the multiple computing tasks are allocated to corresponding supercomputing centers, generating a scheduling scheme that includes task scheduling instructions and data pre-scheduling instructions; wherein, for source tasks and target tasks that have data dependencies and are allocated to different supercomputing centers, the data pre-scheduling instructions are used to transmit at least a portion of the data required by the target task from the source supercomputing center to the target supercomputing center before the source task is completed. According to the scheduling scheme, the multiple supercomputing centers coordinate to execute the computing tasks and implement the data pre-scheduling instructions.

2. The collaborative computing method for big data workflows across supercomputing centers according to claim 1, characterized in that, The acquisition of resource status information, data distribution information, and historical execution statistics information from multiple supercomputing centers includes: Through monitoring agents deployed in various supercomputing centers, the utilization rates of computing resources, storage resources, network bandwidth status, and the storage location and copy information of datasets are periodically collected; and Extract the task execution time, data output rate, and cross-center data transmission throughput of completed workflow instances from the execution log as historical execution statistics.

3. The collaborative computing method for big data workflows across supercomputing centers according to claim 2, characterized in that, The construction of the cost prediction model for predicting task execution time and data transmission time includes: A regression model is used to predict task execution time, with task type, input data size, and available computing resources of the allocated supercomputing center as input features. Using the amount of data to be transmitted and the available bandwidth of the current cross-center network as input features, the data transmission time is predicted.

4. The collaborative computing method for big data workflows across supercomputing centers according to claim 3, characterized in that, The generation of a scheduling scheme that includes task scheduling instructions and data pre-scheduling instructions includes: With the goal of minimizing the estimated total completion time of the workflow, the scheduling scheme is obtained by searching the allocation mapping of tasks to supercomputing centers and the start timing of pre-scheduling of data across centers; The data pre-scheduling instruction includes a pre-transmission start threshold, which indicates that data pre-transmission is triggered when the output data generation amount of the source task reaches a preset condition.

5. The collaborative computing method for big data workflows across supercomputing centers according to claim 4, characterized in that, The implementation of the data pre-scheduling instruction includes: During the execution of the source task, monitor the amount of output data generated by the source task; When the amount of output data reaches the pre-transmission start threshold, data transmission from the source supercomputing center to the target supercomputing center is initiated. The data transmission is used to transmit the currently generated portion of the output data.

6. The collaborative computing method for big data workflows across supercomputing centers according to claim 5, characterized in that, The process of coordinating the multiple supercomputing centers to execute the computing task and implement the data pre-scheduling instructions according to the scheduling scheme further includes: Obtain real-time resource status and task execution progress of each supercomputing center; When the deviation between the actual execution progress and the predicted progress of the cost prediction model exceeds a preset threshold, the computational tasks that have not yet started execution are reallocated based on the updated resource status information and the cost prediction model, and the corresponding data pre-scheduling instructions are updated.

7. The collaborative computing method for big data workflows across supercomputing centers according to claim 6, characterized in that, The data transmission process across supercomputing centers includes: The data to be transmitted is fragmented. The fragmented data is transmitted through multiple parallel transmission channels; The fragmented data is reassembled at the receiving end.

8. The collaborative computing method for big data workflows across supercomputing centers according to claim 7, characterized in that, The data transmission process across supercomputing centers also includes: After fragmentation and before transmission, the fragmented data is compressed and encrypted. Before reassembly at the receiving end, the fragmented data is decrypted and decompressed.

9. A collaborative computing method for big data workflows across supercomputing centers according to claim 8, characterized in that, The workflow model is a directed acyclic graph, where nodes represent computational tasks, edges represent data dependencies and the identifiers of the datasets being passed. The coordinated execution of the computing tasks by the multiple supercomputing centers includes: the central scheduling node issuing task execution instructions to each supercomputing center, and each supercomputing center executing local computing tasks according to the instructions; when cross-supercomputing center data transmission is required, a data transmission connection is directly established between the two supercomputing centers involved.

10. A cross-supercomputing center big data workflow collaborative computing device, used in the cross-supercomputing center big data workflow collaborative computing method according to claim 9, characterized in that, include: The workflow acquisition unit is used to acquire the workflow model to be executed, wherein the workflow model defines multiple computing tasks and the data dependencies between the tasks; The information acquisition unit is used to acquire resource status information, data distribution information, and historical execution statistics information from multiple supercomputing centers. The model building unit is used to build a cost prediction model for predicting task execution time and data transmission time based on the workflow model and the historical execution statistics. The scheduling generation unit is used to allocate the multiple computing tasks to corresponding supercomputing centers based on the resource status information, the data distribution information, and the cost prediction model, and to generate a scheduling scheme that includes task scheduling instructions and data pre-scheduling instructions; wherein, for source tasks and target tasks that have data dependencies and are allocated to different supercomputing centers, the data pre-scheduling instructions are used to transmit at least a portion of the data required by the target task from the source supercomputing center to the target supercomputing center in advance before the source task is completed; The collaborative execution unit is used to coordinate the multiple supercomputing centers to execute the computing tasks and implement the data pre-scheduling instructions according to the scheduling scheme.

Images