A cross-domain distributed processing system and scheduling optimization method

Abstract

The present invention discloses a cross-domain distributed processing system and scheduling optimization method, specifically comprising the following steps: Step 1: A user submits a job through any data center client node in the Hadoop cross-domain distributed processing system; Step 2: The performance of each computing node in multiple data centers is calculated based on the computing node network load; Step 3: The task is assigned to any computing node that has a copy of the data required for the task or a computing node with the smallest network load; Step 4: The total execution time of the job is predicted; Step 5: The ratio of the completed workload of the job is calculated based on the predicted total execution time of the job; Step 6: The job is divided into multiple queues according to a threshold; Step 7: The job execution order is adjusted according to the dynamic priority of the job. Step 8: The layout of the data copies is corrected. Tasks are assigned to the optimal computing nodes in multiple data centers using data locality and dynamic job execution time prediction methods.

Description

Technical Field

[0001] The present invention relates to the technical field of processing systems, and in particular to a cross-domain distributed processing system and a scheduling optimization method. Background Art

[0002] With the accelerating innovation of technologies such as the internet, big data, cloud computing, artificial intelligence, and blockchain, and their increasing integration into all sectors of economic and social development, the digital economy is developing at an unprecedented pace, with a wider reach and profound impact. In the context of "Eastern Data and Western Computing," an increasing number of data centers are springing up, showing a polarized trend: small, micro data centers at the edge, and large, ultra-large data centers. Users desire a logically unified resource space for faster management, access, deployment, and scheduling. However, physically, data and physical resources reside in multiple, dispersed data centers. How to better connect these multiple data centers and achieve global resource management, scheduling, and interoperability has become a critical technical challenge that urgently needs to be addressed.

[0003] Traditional distributed processing systems mostly perform data collection and statistical analysis in the same data center, and do not optimize the layout of data copies. This consumes a large amount of data center resources and data transmission traffic. At the same time, the priority of jobs can only be run according to the priority label when the job is submitted, and cannot be dynamically adjusted based on the job's operation information, which is insufficient in ensuring the fairness of cluster jobs.

[0004] The publication number is CN115357401A, and the patent name is "A method and system for task scheduling and visualization based on multiple data centers". It discloses a method and system for task scheduling and visualization based on multiple data centers, which can support resource scheduling of multiple tasks in multiple data centers and ensure that tasks can be matched to qualified data hosts and executed visually within the deadline. Specifically, it includes: parsing the attribute parameters of the task to be processed, and determining the target resource parameters based on the attribute parameters; obtaining the current performance parameters of each data host in multiple data centers; determining the assignable data host of the task to be processed based on the target resource parameters; sending the task to be processed to the assignable data host for processing; and visually displaying the task currently being processed by each data host through a topology diagram. The system includes a central server and multiple data centers, each data center is configured with multiple data hosts, and each data host includes multiple virtual machines. The present invention can support resource scheduling of multiple tasks in multiple data centers and ensure that tasks can be matched to qualified data hosts and executed visually within the deadline.

[0005] The publication number is CN112637263A, and the patent name is "A method, system and storage medium for optimizing and improving resources in multiple data centers". It discloses a method, system and storage medium for optimizing and improving resources in multiple data centers. By applying resource scheduling rules set by experts, a method combining global scheduling and edge scheduling is adopted when using resources. Finally, a resource consistency graph database analysis method is used to achieve resource consistency and utilization analysis. This method innovatively proposes a method for optimizing and improving resources in multiple data centers, which is suitable for the scenario of optimizing and improving resources in multiple centers. It uses the RDF resource description framework and the TLGM data model to build a multi-data center resource graph library, uses a global scheduler and an edge scheduler to collaboratively process computing requests, and uses a data linkage state data model, scheduling rules, and a probability calculation matrix to transform resource consistency and resource utilization issues into graph queries. It uses original graph reprojection, subgraph merging technology, and efficient load balancing to achieve graph queries. Through the above technologies, optimization and improvement of resources in multiple data centers are achieved.

[0006] The technical problems with the above two patent documents are: global resource management, layout optimization of data copies, and dynamic adjustment of job priorities, resulting in a lack of a good global unified management and access strategy for resources in multiple data centers. At the same time, the lack of optimized layout of data copies results in a large amount of resource and data transmission traffic consumption in the data center, and there is no dynamic adjustment of job priorities, which lacks job fairness. Summary of the Invention

[0007] The main technical problems addressed by this invention are the lack of a well-defined, globally unified management and access strategy for resources across multiple data centers. The lack of optimized data replication layout results in significant data center resource and data transmission traffic consumption. Furthermore, the lack of dynamic job priority adjustment results in a lack of job fairness. This invention provides a cross-domain distributed processing system and scheduling optimization method.

[0008] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a cross-domain distributed processing system and scheduling optimization method based on the metadata management function and resource management function of Hadoop3.x is transformed to form a new Hadoop cross-domain distributed processing system for unified management, access and scheduling of dispersed data resources and physical resources in multiple data centers. The Hadoop cross-domain distributed processing system aggregates the data resources and physical resources of multiple data centers from three levels: local, local, and wide, to form a global unified namespace, providing a global view of all resources in the data center to support the aggregated management and unified access of data resources and physical resources across data centers. Users can submit jobs through any data center client node in the Hadoop cross-domain distributed processing system, and through the global data management function and global resource scheduling function, tasks are assigned to the optimal computing nodes in multiple data centers. Under the condition of ensuring job fairness, the overall running time of jobs across data centers is reduced, and optimal scheduling across multiple data centers is achieved.

[0009] The specific steps include:

[0010] Step 1: The user submits a job through any data center client node in the Hadoop cross-domain distributed processing system. After the user submits the job, the global resource manager allocates resources for the job to a computing node in the data center and starts the job manager.

[0011] Step 2: The global resource manager creates a job manager to manage jobs. The job manager initializes the jobs and divides them into tasks. The scheduler obtains global data distribution and global resource information and calculates the performance of each compute node in multiple data centers based on the compute node network load.

[0012] Step 3: For the initial task, choose to assign the task to any node that has a copy of the data required by the task or a node with the smallest network load;

[0013] Step 4: Build a multivariate linear regression model based on the initial task execution logs of the computing nodes in each data center to predict the task execution time, thereby predicting the total execution time of the job;

[0014] Step 5: During the job execution, calculate the ratio of the completed workload by monitoring the job execution time and combining it with the predicted total execution time.

[0015] Step 6: Based on the job completed workload ratio, the jobs are divided into multiple queues according to the threshold. The higher the job completed workload ratio, the lower the priority.

[0016] Step 7: Adjust the job execution order according to the dynamic priority of the job and continue from step 2 until all tasks are completed.

[0017] Step 8: After the job is completed, collect the access popularity of the data used in this calculation, and modify the layout of the data replicas by comprehensively considering the computing power and load conditions of each data center node to reduce the data transmission overhead when executing subsequent tasks.

[0018] Preferably, in step 3:

[0019] (1) The network load of the computing node is:

[0020] In a multi-data center scenario, due to the existence of cross-data center network transmission, the network bandwidth of the multi-data center is set to WD, as shown below:

[0021]

[0022] The network load of a computing node in a Hadoop cross-domain distributed processing system is determined by the amount of data received by the node within a time interval and the available bandwidth. Due to the problem of cross-data center transmission, the available bandwidth between data centers must be considered. The calculation formula is as follows:

[0023]

[0024] in, represents the amount of data received by computing node n during the t-th time interval, represents the available bandwidth of computing node n in the tth time interval, represents the available bandwidth of data center d in the tth time interval, Indicates the cumulative amount of data to be processed by computing node n after the end of the tth time interval. The larger the value, the greater the network load of computing node n;

[0025] (2) The real-time performance of the computing node is:

[0026] The formula for calculating node real-time performance is as follows:

[0027]

[0028] in is the real-time performance of node n at time t, and the weight of the node resource influencing factors β1+β2+β3+β4+β5=1; Indicates the real-time number of available CPU cores of node n at time t, represents the real-time memory available at node n at time t, represents the real-time disk read and write rate of node n at time t, Indicates the real-time disk availability of node n at time t, Represents the real-time network load of node n at time t, which is the above The weights of factors affecting the real-time performance of computing nodes are calculated using the AHP hierarchical analysis method.

[0029] Preferably, step 5 is: the computing node analyzer collects five parameters of the computing node, namely, CPU rate, CPU usage, memory usage, network bandwidth occupancy, and whether the task is local, based on the completed tasks, and uses a multiple linear regression model to predict the task execution time, as follows:

[0030] 3.1 Collect information on completed tasks on the computing nodes, including five parameters: CPU rate, CPU usage, memory usage, network bandwidth usage, and whether the data is local.

[0031] 3.2 Use the collected data to build a multiple linear regression prediction model. The model is as follows:

[0032] t=f(cf,cu,ru,wu,lt)

[0033] Where t represents the predicted execution time of a unit task on a compute node, cf represents the CPU rate of the compute node executing the task, cu represents the CPU usage of the compute node executing the task, ru represents the memory usage of the compute node executing the task, wu represents the network bandwidth occupancy of the compute node executing the task, and lt represents whether the data replica of the task is on the compute node. The linear regression prediction model matrix is ​​as follows:

[0034]

[0035] Based on the collected data, the value of parameter α is calculated using the least squares method, where n is the node;

[0036] 3.3 Deploy a prediction model on each computing node to estimate the computing power of the node; the prediction information is sent to the job manager via heartbeat messages. The job manager takes the longest execution time of each task in the computing node as the execution time of the task based on the predicted execution time of each task in the job on the computing node, and adds up the predicted execution time of the tasks in each stage as the total execution time of the job.

[0037] Preferably, step 6 is: by real-time monitoring of the running time of the job and combining it with the total running time of the job predicted in step (5), calculate the ratio of the workload completed by the job:

[0038]

[0039] Among them, V jobIndicates the ratio of completed workload of the job, T current Indicates the execution time of the job, T total Indicates the total execution time of the job;

[0040] During the task execution process, the workload ratio V of the job completed according to step (6) is obtained. job Dynamically adjust the job priority during operation. job As dynamic grading for assignments.

[0041] The global namespace manager follows the NameNode master-slave architecture, with the Master Node managing the master node and the Slave Node managing the slave nodes. Its specific functions are identical to the NameNode, but the difference lies in the Hadoop cross-domain distributed processing system, which has transformed the metadata management component to generate a new metadata manager. The metadata manager is the system component responsible for managing metadata (including data resource and physical resource metadata) across multiple data centers. Through global resource monitoring, it aggregates metadata from multiple data centers at three levels: local, regional, and global, forming a unified namespace. This provides a global view of all resource metadata in the data center, supporting aggregated management and unified access to data and physical resources across data centers. The metadata manager aggregates local and local resources based on physical machines (virtual machines) and the distributed file system HDFS they comprise. It then maps resources from multiple data centers, at the HDFS granularity, to the data resource and physical resource namespaces maintained by the management node for global resource aggregation.

[0042] Job Manager: The Job Manager is responsible for managing and coordinating jobs across multiple data centers. It is compatible with the Application Manager functionality in the Resource Manager and adds new job management and coordination capabilities across multiple data centers. When a user submits a job in a data center, the Global Resource Manager interacts with the Global Namespace Manager to obtain metadata about the data required for the job and starts the Job Manager on a compute node in that data center. The Job Manager initializes the job and breaks it down into multiple tasks. The Job Manager then requests resources from the Global Resource Manager for running the task. The Scheduler in the Global Resource Manager encapsulates the required resources into a Container object (including resource information such as the memory and number of CPU cores that can be occupied by the task execution) based on its scheduling algorithm and passes it to the Global Resource Manager. The Global Resource Manager then sends the task to the allocated Containers in each data center for execution.

[0043] The global resource monitor is responsible for real-time monitoring of the resource load and data distribution of compute nodes in each data center. By enabling the resource monitor on each compute node in each data center, it monitors the compute node's data storage and resource status (including CPU, memory, bandwidth, and network I / O) in real time and reports to the global resource manager regularly. The global resource manager also regularly interacts with the global namespace manager to send resource information from each data center to the metadata manager, MetaData Server, to manage global data center metadata.

[0044] The computing node analyzer collects five parameters of the computing node, including CPU rate, CPU usage, memory usage, network bandwidth occupancy, and whether the task is local, based on the completed tasks, and uses a multivariate linear regression model to predict the task execution time.

[0045] Scheduler: The custom scheduler is modified based on the fair scheduler. It assigns tasks to the optimal computing nodes according to the real-time performance, performance preference and job type of each computing node in the cluster, and dynamically adjusts the priority of jobs. While ensuring job fairness, it achieves cluster node load balancing, optimal allocation of jobs and nodes, and shorter average running time in the Hadoop cross-domain distributed processing system.

[0046] In particular, if the data center's region is restricted by policies and regulations and does not allow its IT resources to be controlled by untrusted third parties, then the global namespace manager and global resource manager in the Hadoop cross-domain distributed processing system will not function properly. In this case, the functions of the various components in the Hadoop cross-domain distributed processing system will be changed, as follows:

[0047] Global Namespace Manager: At this time, the global namespace manager is only responsible for the management of data resources and physical resources in its data center. The metadata management component uses the resource monitor to realize the step-by-step aggregation of the data resources and physical resources of the data center from the local and regional levels to form a unified namespace, providing a global view of all resources in the data center to support the aggregated management and unified access of the data resources and physical resources of the data center.

[0048] Global Resource Manager: At this time, the global resource manager is only responsible for resource management in its data center. Due to regulatory control, the original data of some data centers is not allowed to leave the domain. Therefore, the resource manager of each data center can only parse the data information of its data center and realize data processing in multiple data centers through collaborative management through the job manager.

[0049] Job Manager: The Job Manager manages and coordinates jobs across multiple data centers. Job managers within the same job across multiple data centers use ZooKeeper for coordination. When a user submits a job request in a data center (the components in this data center are all master components, and the components in the remaining data centers are slave components), the master resource manager parses the data to be processed by the job and broadcasts the job description to the slave resource managers in the other data centers. The slave resource managers then parse the data to be processed and synchronize it with the master resource manager. At the same time, both the master and slave resource managers will generate job managers for the job in their data centers. The master job manager will divide the job into tasks based on the full data information obtained by the master resource manager, and write the job's task allocation information (including job ID, task ID, data center ID, computing node where the data is located, and task status) into the job node of the distributed application coordination service (ZooKeeper). The remaining slave resource managers obtain their corresponding tasks by listening to the taskMap task collection in the ZooKeeper job node for scheduling. After the task in each data center is completed, the taskMap task collection will be updated and the output location information of the data will be reported to its resource manager (master or slave). The resource manager will summarize the output information of the local job execution, and then transmit the intermediate data from the resource manager to the master resource manager via the wide area network or dedicated line to summarize and output the final job data information.

[0050] The Hadoop cross-domain distributed processing system can be used for both global resource scheduling and management of multiple data centers and autonomous resource scheduling and management of multiple data centers. The two modes included in the system can be configured through the global resource scheduling and management configuration parameter (global_manager) of multiple data centers. 1 is enabled, that is, global resource scheduling and management of multiple data centers, and 0 is disabled, that is, autonomous resource scheduling and management of multiple data centers.

[0051] The Hadoop cross-domain distributed processing system is deployed across multiple data centers, communicating between them via a wide area network (WAN) or dedicated lines. Each data center's Hadoop cross-domain distributed processing system includes a global namespace manager and a global resource manager component, along with at least one TiDB database instance for global data management and access across the Hadoop cross-domain distributed processing system. The cross-domain coordination and management of jobs between systems is handled by the job manager within the global resource manager, while cross-domain job scheduling is performed by the scheduler within the global resource manager. Global resource monitoring is performed by the global resource monitor within the global resource manager. The Hadoop cross-domain distributed processing systems across multiple data centers can all be used for global data access, management, and scheduling. Users can submit jobs from any data center, enabling distributed big data processing across data centers.

[0052] Beneficial effects of the present invention:

[0053] By building a new Hadoop cross-domain distributed processing system, we aim to unify the management, access, and scheduling of dispersed data and physical resources across multiple data centers. Our scheduling optimization method allocates tasks submitted to the Hadoop cross-domain distributed processing system to optimal compute nodes across multiple data centers based on global data distribution and resource information. This method leverages data locality and dynamic job execution time prediction to dynamically adjust job priorities based on the amount of completed work. This approach reduces the overall runtime of jobs across data centers while ensuring job fairness, achieving optimal scheduling across multiple data centers. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of a typical single-port time reversal cavity and circulator combination system of the present invention;

[0055] Figure 2 This is a layout diagram for experimental verification in an example of the present invention;

[0056] Figure 3 is the initial signal in the example of the present invention;

[0057] Figure 4 Reconstructing the signal for inversion in the example of the present invention;

[0058] Figure 5 This is a deployment diagram of the Hadoop cross-domain distributed processing system in the present invention;

[0059] Figure 6 This is a first set of experimental effect diagrams of an embodiment of the present invention;

[0060] Figure 7 This is a second set of experimental effect diagrams of an embodiment of the present invention;

[0061] Figure 8 This is the third set of experimental effect diagrams of the embodiments of the present invention. DETAILED DESCRIPTION

[0062] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings.

[0063] See also Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 、 Figure 8 , embodiments of the present invention include: a cross-domain distributed processing system and scheduling optimization method;

[0064] like Figure 2 As shown, the local resource root directory of each physical machine (virtual machine) is linked to a logical node of the distributed file system maintained by the global namespace manager. The root directory of each distributed file system is linked to a logical node of the data center maintained by the global namespace manager. The logical nodes and their organizational relationships are recorded by the management node in the distributed database TiDB, which stores important attributes of multiple data centers. To improve reliability, each data center in the Hadoop cross-domain distributed processing system environment requires a TiDB distributed database, relying on its cross-datacenter multi-replica synchronization mechanism to achieve eventual consistency across data centers. Data resource attributes include the data center's UUID, the UUID of the distributed file system HDFS, the UUID of the physical machine (virtual machine), and data file metadata (such as data size, file format, and access path). Physical resource attributes include the UUID of the data center, the UUID of the distributed file system HDFS, the UUID of the physical machine (virtual machine), and used and available resources (resources include the number of CPU cores, memory, network I / O, etc.). The management node aggregates data resources and physical resources step by step to form a unified global data management space.

[0065] The global resource manager transforms the job manager, scheduler and node manager based on the resource manager to achieve job scheduling optimization of cross-domain distributed processing system. Its architecture is as follows: Figure 3 shown.

[0066] The jobs are divided into multiple queues based on their dynamic scores. For example, the five queues Q1 to Q5 are divided into five queues with a unit of 0.2, and the scores increase in order.

[0067] As a job's tasks run, the higher its completed workload ratio, the higher its score and, accordingly, the lower its priority. For example, when job A is initially submitted for execution, its score is 0, giving it the highest priority. It is placed in queue Q1. As the job continues to run, its completed workload increases. When its score exceeds the Q1 threshold, job A is transferred from queue Q1 to queue Q2, where its priority decreases until it is transferred to queue Q5 and completes its execution.

[0068] After a computation is complete, the Hadoop cross-domain distributed processing system asynchronously optimizes data replicas in the background. The global resource monitor, based on the access information collected for the data used in the computation, considers the computing power and load of each data center node and optimizes the layout of data replicas to reduce data transmission overhead during subsequent task execution.

[0069] Assume that the historical average access frequency of the data block is The number of accesses in the current cycle is M, and the access cycle is T. Then the access frequency of the data block in the next cycle is:

[0070]

[0071] In the formula, 0≤α≤1 is the weight used to determine the impact of the historical data access frequency and the current cycle access frequency on the next cycle access frequency. When setting the weight, the locality of data access is fully considered, so that the access frequency closer to the prediction cycle has the greatest impact on it, that is, the weight is the largest. At the same time, the higher the data access frequency, the higher the data popularity.

[0072] The Global Resource Manager uses the Global Resource Monitor to obtain the historical access frequency of the data required for the task, the number of accesses and access period in the current cycle, and the performance of computing nodes in each data center. Using the aforementioned data access frequency prediction formula, the Global Resource Manager calculates the access frequency for the next cycle and determines whether to create a data replica. If the predicted data access popularity and node performance are within a certain threshold, a node within the threshold is selected to create a replica. For low-popularity data, HDFS erasure coding technology is used to reduce data replication.

[0073] Loop through the job queue until all jobs are completed.

[0074] The cross-domain distributed processing system of the present invention is an improvement on Hadoop 3.x and is compatible with Hadoop 3.x's scheduling algorithm. The cross-domain distributed processing system of the present invention was deployed in three different data centers, with a total of 13 heterogeneous computing nodes. Multi-data center global resource scheduling management was adopted. Three loads with different resource characteristics (Sort, TeraSort, and WordCount) were selected. Three sets of jobs of different sizes were experimentally verified against Hadoop 3.x's default FairScheduler scheduling algorithm and the task scheduling optimization algorithm of the present invention. Each set of load data was submitted to the cluster 10 times for testing, and the average result was taken from 10 runs. The three groups of job sets are: the first group [small: (Sort: 250M, TeraSort: 250M, WordCount: 250M); medium: (Sort: 1G, TeraSort: 1G, WordCount: 1G); large: (WordCount: 10G)], the second group [small: (Sort: 500M, TeraSort: 500M, WordCount: 500M); medium: (Sort: 5G, TeraSort: 5G, WordCount: 5G); large: (WordCount: 50G)], and the third group [small: (Sort: 1G, TeraSort: 1G, WordCount: 1G); medium: (Sort: 10G, TeraSort: 10G, WordCount: 10G); large: (WordCount: 100G)].

[0075] Experimental results

[0076] (1) The first set of experimental results, such as Figure 6 ;

[0077] (2) The second set of experimental results, such as Figure 7 ;

[0078] (3) The third group of experimental results, such as Figure 8 ;

[0079] (4) Experimental summary

[0080] The results of the three experimental groups were averaged, demonstrating that the optimization algorithm effectively addressed issues such as mismatches between tasks and compute node resources, unbalanced node loads, and resource waste, improving overall job execution efficiency. The average CPU utilization decreased by approximately 12.78%, the average memory utilization decreased by approximately 14.43%, the average cluster load decreased by approximately 17.43%, and the average total job runtime decreased by 15.56%.

[0081] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-domain distributed processing system and scheduling optimization method, characterized in that: The specific steps include: Step 1: The user submits a job through any data center client node in the Hadoop cross-domain distributed processing system. After the user submits the job, the global resource manager allocates resources for the job to a computing node in the data center and starts the job manager. Step 2: The global resource manager creates a job manager to manage jobs. The job manager initializes the jobs and divides them into tasks. The scheduler obtains global data distribution and global resource information and calculates the performance of each compute node in multiple data centers based on the compute node network load. Step 3: For the initial task, choose to assign the task to any node that has a copy of the data required by the task or a node with the smallest network load; Step 4: Build a multivariate linear regression model based on the initial task execution logs of the computing nodes in each data center to predict the task execution time, thereby predicting the total execution time of the job; Step 5: During the job execution, calculate the ratio of the completed workload by monitoring the job execution time and combining it with the predicted total execution time. Step 6: Based on the job completed workload ratio, the jobs are divided into multiple queues according to the threshold. The higher the job completed workload ratio, the lower the priority. The workload ratio satisfies the following formula: Among them, V job Indicates the ratio of completed workload of the job, T current Indicates the execution time of the job, T total Indicates the total execution time of the job; Step 7: Adjust the job execution order based on the dynamic priority of the job and continue from step 2 until all tasks are completed; Step 8: After the job is completed, collect the access popularity of the data used in this calculation, and adjust the layout of data copies by comprehensively considering the computing power and load conditions of multiple data center nodes to reduce the data transmission overhead when executing subsequent tasks.

2. A cross-domain distributed processing system and scheduling optimization method according to claim 1, characterized in that: In step 3: (1) The network load of the computing node is: In a multi-data center scenario, due to the existence of cross-data center network transmission, the network bandwidth of the multi-data center is set to WD, as shown below: The network load of a computing node in a Hadoop cross-domain distributed processing system is determined by the amount of data received by the node within a time interval and the available bandwidth. Due to the problem of cross-data center transmission, the available bandwidth between data centers must be considered. The calculation formula is as follows: in, represents the amount of data received by computing node n during the t-th time interval, represents the available bandwidth of computing node n in the tth time interval, represents the available bandwidth of data center d in the tth time interval, Indicates the cumulative amount of data to be processed by computing node n after the end of the tth time interval. The larger the value, the greater the network load of computing node n; (2) The real-time performance of the computing node is: The formula for calculating node real-time performance is as follows: in is the real-time performance of node n at time t, and the weight of the node resource influencing factors β1+β2+β3+β4+β5=1; Indicates the real-time number of available CPU cores of node n at time t, represents the real-time memory available at node n at time t, represents the real-time disk read and write rate of node n at time t, Indicates the real-time disk availability of node n at time t, Represents the real-time network load of node n at time t, which is the above The weights of factors affecting the real-time performance of computing nodes are calculated using the AHP hierarchical analysis method.

3. A cross-domain distributed processing system and scheduling optimization method according to claim 1, characterized in that: Step 5 is: the computing node analyzer collects five parameters of the computing node, namely, CPU rate, CPU usage, memory usage, network bandwidth occupancy, and whether the task is local, based on the completed tasks, and uses a multiple linear regression model to predict the task execution time, as follows: 3.1 Collect information on completed tasks on the computing nodes, including five parameters: CPU rate, CPU usage, memory usage, network bandwidth usage, and whether the data is local. 3.2 Use the collected data to build a multiple linear regression prediction model. The model is as follows: t=f(cf,cu,ru,wu,lt) Where t represents the predicted execution time of a unit task on a compute node, cf represents the CPU rate of the compute node executing the task, cu represents the CPU usage of the compute node executing the task, ru represents the memory usage of the compute node executing the task, wu represents the network bandwidth occupancy of the compute node executing the task, and lt represents whether the data replica of the task is on the compute node. The linear regression prediction model matrix is ​​as follows: Based on the collected data, the value of parameter α is calculated using the least squares method, where n is the node; 3.3 Deploy a prediction model on each computing node to estimate the computing power of the node; the prediction information is sent to the job manager via heartbeat messages. The job manager takes the longest execution time of each task in the computing node as the execution time of the task based on the predicted execution time of each task in the job on the computing node, and adds up the predicted execution time of the tasks in each stage as the total execution time of the job.

4. A cross-domain distributed processing system and scheduling optimization method according to claim 1, characterized in that: The step 6 is: by real-time monitoring of the running time of the job and combining it with the total running time of the job predicted in step (5), the ratio of the workload completed by the job is calculated; During the task execution process, the workload ratio V of the job completed according to step (6) is obtained. job Dynamically adjust the job priority during operation. job As dynamic grading for assignments.

Images