A computing power network task scheduling method based on a "storage-computing separation" architecture

Abstract

The application relates to a computing power network task scheduling method based on a 'compute storage separation' architecture, and belongs to the technical field of computing power scheduling, which comprises the following steps: initializing data and a computing power network environment; submitting a computing power network service composed of multiple sub-services, and deconstructing the computing power network service; analyzing a service request, determining the dependency and correlation of resource requirements, and dividing the resource requirements into resource requirement units; using an NSGA-III algorithm to realize multi-objective optimization and optimal matching; and completing scheduling according to the optimal matching. In the computing power network environment, the strategy preferentially selects a data center with short transmission time and low communication cost to complete the service, while improving the utilization rate of the data center resources on the premise of meeting the service requirements, so that efficient operation of the service and full utilization of the resources are realized.

Description

TECHNICAL FIELD

[0001] The application provides a computing power network task scheduling method based on a "storage-computing separation" architecture, and belongs to the technical field of computing power scheduling. BACKGROUND

[0002] "Storage-computing separation" is a new data architecture, which can be regarded as a data analysis layer, a computing layer and a storage layer from top to bottom, and realizes independent distributed services. By separating the computing layer and the storage layer, the purpose of flexible opening of data to different businesses for data analysis, independent expansion of computing and storage, and resource isolation of computing and storage can be achieved.

[0003] Under the "storage-computing separation" architecture, different data center resources can be freely combined to meet the needs of different businesses, and at the same time, storage systems and data analysis services can also be freely combined across domains to realize the free flow of data between different systems. The advantage of this architecture is to improve the scalability, reliability and flexibility of the system, thereby providing more powerful and flexible support for the development of businesses. At present, many important "storage-computing separation" architectures have appeared at home and abroad, including HDFS of Huawei, Alluxio of open source, Colossus of Google and TAO of Facebook.

[0004] In order to cope with these challenges, researchers have proposed various service scheduling schemes. For example, some researchers propose a task scheduling algorithm for completion time and load balancing, but this strategy does not consider the mutual dependence of tasks, and the successor task may need to obtain the data and information of the predecessor task to execute smoothly. In this case, if only the load balancing between data centers is considered for scheduling, it may cause the task to be blocked, thereby affecting the progress of the entire task flow. Some researchers propose an improved task scheduling algorithm in a cloud computing environment, which collects tasks in batches to avoid the length of the collected task exceeding the limit of the resource usage state. For each batch of tasks, a local optimal solution is sought to reduce the completion time and improve resource utilization, but it ignores that under the "storage-computing separation" architecture, when tasks need to be completed in different data centers, the transmission of data and information may require a large amount of transmission cost and transmission time.

[0005] Therefore, the application proposes a computing power network task scheduling method based on a "storage-computing separation" architecture, which aims to more flexibly manage the resources of data centers, maximize the utilization of resources, reduce resource scheduling costs, and thereby improve the utilization of data center resources and service processing efficiency. SUMMARY

[0006] In view of the deficiencies of the prior art, the application provides a computing power network task scheduling method based on a "storage-computing separation" architecture.

[0007] The application adopts the following technical solutions:

[0008] A computing power network task scheduling method based on a "storage-computing separation" architecture first establishes a computing power network service scheduling model, including a service list, a data center resource list, a service pedigree establishment part, a service request analysis part, a resource unit division part and a multi-objective optimizer, the service list contains m sub-services, and the data center resource list contains n data centers;

[0009] In a computing power network environment, service scheduling plays a crucial role. It determines the execution order of services and the allocation of resources, directly affecting the efficient operation of services and the full utilization of resources. In order to achieve optimal scheduling of services submitted by users, the application designs a computing power network service scheduling model, the core of which is to allocate sub-services to different data centers for execution, where the sub-services can be represented as S=[sevice1, sevice2, …, service m ], m represents the number of sub-services, to achieve optimal service scheduling. In the computing power network service scheduling model, all resources in the computing power network are first initialized to obtain a computing power network resource list. Then, after the user submits a service sequence, the communication relationship of different services is extracted to establish a service pedigree. Next, the service request is analyzed, and the demand unit is divided according to the association and dependency relationship between the resources of the service request. Finally, the service demand and the data center resource list are put into the multi-objective optimizer for optimal matching to achieve optimal service scheduling. Through the computing power network service scheduling model, efficient and accurate scheduling of services submitted by users can be achieved, thereby realizing full utilization of resources and efficient operation of services. This will provide strong support for the development and application of computing power networks.

[0010] The scheduling method includes the following steps:

[0011] Step 1: initialize data and initialize the computing power network environment;

[0012] Step 2: submit a computing power network service composed of multiple sub-services, and deconstruct the computing power network service;

[0013] Step 3: analyze the service request, determine the dependency and association relationship of resource demand, and divide the resource demand into resource demand units;

[0014] Step 4: Use the NSGA-III algorithm to perform multi-objective optimization and achieve optimal matching;

[0015] Step 5: Complete the scheduling based on the optimal match.

[0016] Preferably, step 1 includes:

[0017] 1.1: Initialize Data

[0018] Data centers in different regions provide different computing, storage, and network resources, and bear the heavy responsibility of data computation, storage, and forwarding. A data center can be represented as DC = [DC1, DC2, ..., DC]. n ], where n represents the number of data centers; for each data center DC i DC i = {BI, CPU, GPU, VM, CO, RAM, HD, DB}, where BI, CPU, GPU, VM, CO, RAM, HD, and DB represent the basic information of the data center, CPU, GPU, virtual machine, container, memory, hard disk, and database, respectively. Different attribute indicators are defined for each resource type in the data center to display the resource availability, as shown in Table 1.

[0019] Table 1: Different attribute indicators are defined for each resource type.

[0020]

[0021]

[0022] 1.2: Computing Network Initialization

[0023] Access data center resources through the API provided by the cloud service provider to achieve data center access; obtain data center API access permissions, consult the API documentation, and understand how to use the API interface to obtain the required information; use the API to call the corresponding resource interface to obtain the corresponding information; write the obtained information into the data center resource list to complete the computing power network initialization;

[0024] 1.3: Constructing the computing power network topology

[0025] Read cloud service provider documentation or APIs to obtain association or connection information between data centers; use the iperf tool to test the network bandwidth and communication latency between each pair of data centers; based on the previously obtained connection information, network bandwidth, and communication latency, create a dictionary describing the relationship between each pair of data centers, including three key-value pairs: connection relationship, network bandwidth, and communication latency; draw the computing power network topology matrix based on the dictionary.

[0026] Preferably, step 2 is implemented by establishing a service lineage, and the service lineage is established according to the user-provided service.

[0027] Service lineage definition: In the service lineage, each sub-service can be a predecessor, successor or both, which can help people better understand the relationship between services, so as to better plan and organize services. At the same time, it can also improve the repeatability and maintainability between services, because in the service lineage, each service has a clear defined position and relationship, which can make the dependency relationship between services more clear and explicit.

[0028] At the same time, a new value Cd is introduced in the service lineage to represent the communication density between sub-services. The initial value is read according to the service sequence document submitted by the user in the initial stage, and the monitor_service() monitoring function is used to continuously monitor and update during the subsequent service execution.

[0029] The specific process of implementing the service lineage is as follows:

[0030] 2.1: Read the service sequence document submitted by the user to obtain the predecessor and successor of each sub-service, as well as the corresponding communication density information;

[0031] 2.2: For each service, traverse each sub-service and take it as a node of the directed graph;

[0032] 2.3: For each service, establish an adjacent node list to store the information of its predecessor and successor, and take the corresponding communication density as the weight of the directed graph edge;

[0033] 2.4: Define a monitoring function named "monitor_service()" to monitor whether the communication density between sub-services changes in real time during service execution. If it changes, the system will update the corresponding service lineage information in time.

[0034] Preferably, step 3 specifically includes:

[0035] 3.1: Determine the dependency and association relationship of resource demand, and divide the resource demand into resource demand units;

[0036] In the computing power network, there may be dependencies and association relationships between the resources required for service execution. For example, under the premise of ensuring data access speed and stability, cloud hosts need one or more cloud disks to store data, load balancers need to be in the same network as cloud hosts and can access their IP addresses, object storage needs to be deployed in the same data center as CDN and can provide data acceleration and distribution services. Therefore, in order to better meet the needs of services, a triple G(G1, G2, G3) is used to describe the resource requirement unit, where G1 represents the sub-service name, G2 represents the data center resource with dependency relationship, and G3 represents the resource requirement amount; by grouping G, it can be obtained which resources a service needs to have mutual dependencies in order to obtain and schedule resources;

[0037] 3.2: Different resources in the data center are divided into different resource units in order to realize the collaborative use and management of resources between various data centers; for example, CPU resources can be divided into different core number and frequency units, that is:

[0038] CPU = {(2, 2GHz), (4, 2GHz), (4, 3GHz), (8, 3GHz),...}

[0039] Where each element represents a different CPU resource unit, consisting of two parameters: core number and frequency.

[0040] Similarly, memory resources can be divided into different capacity and bandwidth units, that is:

[0041] RAM = {(2GB, 1600MHz), (4GB, 1600MHz), (8GB, 1600MHz), (8GB, 2400MHz),...}

[0042] Where each element represents a different memory resource unit, consisting of two parameters: capacity and bandwidth.

[0043] 3.3: Determine the resource requirement amount.

[0044] Preferably, step 3.3 specifically includes:

[0045] A, create an n x m two-dimensional matrix A, each element of the matrix A is a list, used to store various attribute requirements of resources;

[0046] B, for each service i, traverse its requirement for each resource and add its various attribute requirements to A i,j , where j is the index of the resource;

[0047] C, the final A is the generated two-dimensional matrix:

[0048]

[0049] where A i,j represents the list of various attribute requirements of the i-th service to the j-th resource.

[0050] Preferably, the objective function and the constraint condition in step 4 are set as:

[0051] Optimization objective 1: transmission time f1(x)

[0052] In the service pedigree, there is data and information transmission between sub-services with front and back execution relationship. When two sub-services are processed in different data centers, data transmission needs to pass through l data centers. In the execution process of the service, the higher the communication efficiency between data centers, the less the transmission time of information and data, and then

[0053]

[0054] where x = (x1, x2, …, x l ) represents the sequence of data centers passed through by the service when data and information transmission is performed, and the communication efficiency from data center x i to x i+1 is

[0055]

[0056] where Dt is the data transmission amount, DI is the transmission distance, NB and NL are the network bandwidth and network delay, respectively;

[0057] Optimization objective 2: communication cost f2(x)

[0058] In the execution process of the service, the smaller the communication density between services, the smaller the same data communication cost, and then

[0059]

[0060] where Cd is the communication density, is the network bandwidth cost and delay cost when communication between adjacent data centers, and l represents the number of possible data centers passed through;

[0061] Optimization objective 3: resource unit utilization rate f3(x)

[0062] For the case of dividing resources such as CPU and memory into different units, the utilization rate of each resource unit is calculated, and the average value of them is obtained to get the overall resource unit utilization rate, and then

[0063]

[0064] where nr denotes the number of resource units, u i denotes the utilization rate of the i-th resource unit;

[0065] Define the target vector F = [f1(x), f2(x), -f3(x)], which represents that the target is to minimize the transmission time f1(x) and communication cost f2(x), and maximize the resource unit utilization rate f3(x);

[0066] Define the constraint condition:

[0067] Constraint condition 1:

[0068] R_total >= R_demand

[0069] Wherein, R_total represents the total amount of all available resources in the data center, and R_demand represents the amount of resources required by various services in the data center;

[0070] Constraint condition 2:

[0071] h(x) = max(0, t(x) - T)

[0072] Wherein, t(x) is the service completion time, and T is the service timeliness deadline, and if t(x) > T, h(x) represents that the service completion time exceeds the timeliness deadline.

[0073] Preferably, the specific steps of using the NSGA-III algorithm to realize multi-objective optimization to find the optimal matching are as follows:

[0074] 4.1: Input parameters are the computing power network resource list netList, the service hierarchy serviceHierarchy, and the demand matrix demandMatrix of the service;

[0075] 4.2: Traverse each service according to the order of the service hierarchy, and traverse each level of the data center according to the order of the computing power network resource list netList, and check whether the data center resources meet the demand matrix of the current service;

[0076] 4.3: If the data center meets the demand matrix of the current service, use the NSGA-III function to optimize the selection;

[0077] 4.4: Generate an initial population Pt, and set parameters including population size and maximum iteration number;

[0078] 4.5: Improve the reference point selection strategy

[0079] 4.6: Cross and mutate Pt to generate Qt, and merge Pt and Qt to obtain Rt;

[0080] 4.7: Fast non-dominated sorting of Rts;

[0081] 4.8: Selection mechanism based on reference points to filter suitable individuals to form new offspring Pt+1;

[0082] 4.9: Determine whether the maximum number of iterations is reached, if yes, go to step 4.10, if no, return to step 4.6;

[0083] 4.10: Select the optimal data center from the final population, and update the optimal data center to the mapping table mapping;

[0084] 4.11: Output the result as the optimal matching list of services and data center resources.

[0085] Preferably, step 4.5 improves the reference point selection strategy and the specific implementation steps are as follows:

[0086] a. Use the DBSCAN clustering algorithm to generate an initial reference point set P, and randomly select a reference point p as a seed point in the reference point set P;

[0087] b. For each objective function, calculate the distance d between each reference point and the seed point, and arrange the distances in ascending order;

[0088]

[0089] where (x1, y1, z1) and (x2, y2, z2) are the coordinates of the reference point and the seed point, respectively;

[0090] c. For each reference point, calculate its weight w:

[0091] w[j] = 1 / (d[j] + 2)

[0092] where d[j] is the distance between the reference point and the seed point;

[0093] d. For each reference point, calculate its global density p':

[0094] p'[j] = 1 / (N * S[j])

[0095] where N is the size of the reference point set P, and S[j] is the range of the objective function where the reference point p[j] is located;

[0096] e. For each reference point, calculate its probability distribution p:

[0097] p[j] = (w[j] * p'[j]) / ∑(w[k] * p'[k])

[0098] where ∑ is the summation symbol, and k is the index of all reference points;

[0099] f. Randomly generate a random number r between 0 and 1, and calculate the cumulative probability distribution F[j] of each reference point:

[0100] F[j] = ∑(p[k l ])

[0101] Wherein, k l is the index of all reference points less than or equal to j;

[0102] g. Find the first reference point j such that r is less than or equal to F[j], and generate a new reference point q:

[0103] q[k] = p[j][k m ]

[0104] Wherein, k m is the index of all target functions;

[0105] h. Add the new reference point q to the reference point set P.

[0106] Preferably, the implementation process of step 5 is as follows:

[0107] After obtaining the optimal matching list of services and data centers in step 4, the services are deployed to the selected data centers. In order to ensure the high performance, availability of the services and to ensure compliance with the service level agreement (SLA) requirements, the performance indicators of the services such as response time, throughput, error rate, etc. need to be monitored in real time, and timely measures are taken to solve the problems, so as to complete the optimal scheduling of the services and improve the user experience and satisfaction.

[0108] The details of the present application can be referred to the prior art.

[0109] The present application has the following beneficial effects:

[0110] The present application provides a computing power network task scheduling method based on a "storage-computation separation" architecture, which can select data centers with short transmission time and low communication cost to execute tasks under the premise of meeting service requirements, while improving the utilization rate of data center resource units.

[0111] At the same time, the present application also symbolically defines the data center resources and service requirements (such as Table 1), improves the efficiency and reliability of the data center, reduces the failure caused by improper resource management, improves the utilization rate of resources, and reduces the complexity of maintenance and management. BRIEF DESCRIPTION OF DRAWINGS

[0112] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.

[0113] Figure 1 A schematic diagram of a computing power network service scheduling model;

[0114] Figure 2 Flowchart for constructing the computing power network topology;

[0115] Figure 3 To establish a service hierarchy flowchart;

[0116] Figure 4 This is a flowchart of a computing network task scheduling method based on a "memory-compute separation" architecture. Detailed Implementation

[0117] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. However, this is not the only description; all aspects not described in detail herein are based on conventional techniques in the art.

[0118] Example 1

[0119] A computing network task scheduling method based on a "memory-compute separation" architecture is proposed. First, a computing network service scheduling model is established, including a service list, a data center resource list, a service hierarchy establishment section, a service request parsing section, a resource unit partitioning section, and a multi-objective optimizer. The service list contains m sub-services, and the data center resource list contains n data centers, each providing different types and quantities of resources. The service hierarchy establishment section determines the relationships and communication densities between services based on user-provided service requests, constructing a hierarchical service hierarchy. The service request parsing section traverses all sub-services, recording resource demand types and quantities, and establishing a service demand matrix. The resource unit partitioning section partitions resource units based on resource dependencies and relationships for overall scheduling. The multi-objective optimizer searches for the optimal match among data centers that meet service demands based on three objectives: transmission time, communication cost, and resource unit utilization.

[0120] In a computing network environment, service scheduling plays a crucial role. It determines the execution order of services and the allocation of resources, directly impacting the efficient operation of services and the full utilization of resources. To achieve optimal scheduling of user-submitted services, this invention designs a computing network service scheduling model, such as... Figure 1 As shown, its core is to distribute sub-services to different data centers for execution, where these sub-services can be represented as S = [service1, service2, ..., service2]. m], m represents the number of sub-services to achieve optimal service scheduling. In the computing power network service scheduling model, first, all resources in the computing power network are initialized to obtain a list of computing power network resources. Then, after the user submits the service sequence, the communication relationship of different services is extracted to establish the service pedigree. Next, the service request is parsed, and the demand unit is divided according to the association and dependency relationship between the resources of the service request. Finally, the service demand and the data center resource list are put into the multi-objective optimizer for optimal matching to achieve optimal service scheduling. Through the computing power network service scheduling model, efficient and accurate scheduling of the services submitted by the user can be realized, thereby realizing full utilization of resources and efficient operation of services. This will provide strong support for the development and application of computing power networks.

[0121] The scheduling method comprises the following steps:

[0122] Step 1: initialize data, initialize the computing power network environment;

[0123] Step 2: submit a computing power network service composed of multiple sub-services, and deconstruct the computing power network service;

[0124] Step 3: parse the service request, determine the dependency and association relationship of resource demand, and divide the resource demand into resource demand units;

[0125] Step 4: use the NSGA-III algorithm to realize multi-objective optimization and optimal matching;

[0126] Step 5: complete the scheduling according to the optimal matching.

[0127] Embodiment 2

[0128] A computing power network task scheduling method based on a "compute-store-separation" architecture, as described in Embodiment 1, except that step 1 comprises:

[0129] 1.1: initialize data

[0130] Different regional data centers provide different computing resources, storage resources and network resources, and shoulder the heavy responsibility of data computing, storage and forwarding. A data center can be represented as DC = [DC1, DC2, … DCn], where n represents the number of data centers; for each data center DCi, DCi = [Ci, C2, … Cm], where m represents the number of computing resources provided by the data center; for each computing resource Cj, Cj = [Rj, Tj, Nj], where Rj represents the computing power of the computing resource, Tj represents the storage capacity of the computing resource, and Nj represents the network bandwidth of the computing resource. n ], where n represents the number of data centers; for each data center DC i , DC i{BI, CPU, GPU, VM, CO, RAM, HD, DB}, wherein BI, CPU, GPU, VM, CO, RAM, HD, and DB represent the basic information of the data center, CPU, GPU, virtual machine, container, memory, hard disk, and database respectively; in this embodiment, different attribute indexes are defined for each resource type of the data center, and the availability of the resources is displayed, as shown in Table 2:

[0131] Table 2: Attribute indexes defined for each resource type in this embodiment

[0132]

[0133]

[0134] 1.2: Initialization of the computing power network

[0135] The data center resources are accessed through the API provided by the cloud service provider to realize the access of the data center; the API access permission of the data center is obtained, the API document is consulted, and it is understood how to use the API interface to obtain the required information; the corresponding resource interface is called by using the API to obtain the corresponding information; the obtained information is written into the data center resource list, and the initialization of the computing power network is completed;

[0136] For example:

[0137]

[0138]

[0139] 1.3: Construction of the topology structure of the computing power network

[0140] As shown in Figure 2 , the association or connection relationship information between the data centers is obtained by reading the cloud service provider document or API; the network bandwidth and communication delay between the two data centers are tested by using the iperf tool; according to the connection relationship information, network bandwidth and communication delay obtained in the foregoing, a dictionary is created to describe the relationship between the two data centers, including three key-value pairs: connection relationship, network bandwidth and communication delay; the topology structure matrix of the computing power network is drawn according to the dictionary.

[0141] The implementation process is as follows:

[0142]

[0143]

[0144] Embodiment 3

[0145] A computing power network task scheduling method based on a "compute storage separation" architecture, as described in embodiment 2, except that step 2 is implemented by establishing a service lineage. The service lineage is established according to the user-provided service.

[0146] Service lineage definition: In the service lineage, each sub-service can be a predecessor, successor, or both, which can help people better understand the relationship between services, thereby better planning and organizing services. At the same time, it can also improve the repeatability and maintainability between services, because in the service lineage, each service has a clearly defined position and relationship, which can make the dependency relationship between services more clear and explicit.

[0147] At the same time, a new value Cd is introduced in the service lineage to represent the communication density between sub-services. The initial value is read according to the service sequence document submitted by the user in the initial stage, and the monitor_service() monitoring function is used to continuously monitor and update during the subsequent service execution.

[0148] As Figure 3 It can be seen that the construction process of the service lineage inputs the service information provided by the user, and outputs the service lineage list, while continuously monitoring and updating the communication density during the subsequent execution of the service. The specific process of implementing the service lineage is as follows:

[0149] 2.1: Read the service sequence document submitted by the user to obtain the predecessor and successor services of each sub-service, as well as their corresponding communication density information;

[0150] 2.2: For each service, traverse each sub-service and treat it as a node of a directed graph;

[0151] 2.3: For each service, establish an adjacency node list to store the information of its predecessor and successor services, and take the corresponding communication density as the weight of the directed graph edge;

[0152] 2.4: Define a monitoring function called "monitor_service()" to monitor in real time whether the communication density between sub-services changes during service execution. If it changes, the system will update the corresponding service lineage information in time, as shown in Figure 3 .

[0153] Service lineage code implementation process:

[0154]

[0155]

[0156] Embodiment 4

[0157] A computing power network task scheduling method based on a "compute storage separation" architecture, as described in embodiment 3, except that step 3 specifically includes:

[0158] 3.1: Determine the dependencies and correlations of resource requirements, and divide the resource requirements into resource requirement units;

[0159] In the computing power network, there may be dependencies and correlations between the resources required for service execution. For example, under the premise of ensuring data access speed and stability, cloud hosts need one or more cloud disks to store data, load balancers need to be in the same network as cloud hosts and can access their IP addresses, and object storage needs to be deployed in the same data center as CDN and can provide data acceleration and distribution services. Therefore, in order to better meet the needs of services, a triple G(G1, G2, G3) is used to describe the resource requirement unit, where G1 represents the sub-service name, G2 represents the data center resource with dependency relationship, and G3 represents the resource requirement amount; by grouping G, we can get which resources a service needs to have mutual dependencies in order to obtain and schedule resources;

[0160] 3.2: Divide the different resources of the data center into different resource units in order to realize the collaborative use and management of resources between different data centers; for example, CPU resources can be divided into different core and frequency units, that is:

[0161] CPU = {(2, 2GHz), (4, 2GHz), (4, 3GHz), (8, 3GHz),...}

[0162] Where each element represents a different CPU resource unit, consisting of two parameters: core number and frequency.

[0163] Similarly, memory resources can be divided into different capacity and bandwidth units, that is:

[0164] RAM = {(2GB, 1600MHz), (4GB, 1600MHz), (8GB, 1600MHz), (8GB, 2400MHz),...}

[0165] Where each element represents a different memory resource unit, consisting of two parameters: capacity and bandwidth.

[0166] 3.3: Determine the resource requirement amount.

[0167] Step 3.3 specifically includes:

[0168] A, create an n x m two-dimensional matrix A, each element of the matrix A is a list, used to store various attribute requirements of resources;

[0169] B. For each service i, traverse its demand for each resource, and add its various attribute requirements to A i,j where j is the index of the resource;

[0170] C. The final A is the generated two-dimensional matrix:

[0171]

[0172] where A i,j represents the list of various attribute requirements of the ith service for the jth resource.

[0173] Embodiment 5

[0174] A computing power network task scheduling method based on a "compute-store separation" architecture, as described in Embodiment 4, except that in step 4, the objective function and the constraint condition are set as:

[0175] Optimization objective 1: transmission time f1(x)

[0176] In the service lineage, there is data and information transmission between sub-services with a front-back execution relationship. When two sub-services are processed in different data centers, data transmission needs to pass through l data centers. In the execution process of the service, the higher the communication efficiency between data centers, the less the transmission time of information and data, and the smaller the communication cost.

[0177]

[0178] where x = (x1, x2, …, x l ) represents the sequence of data centers passed through by the service during data and information transmission, and the communication efficiency from data center x i to x i+1 is

[0179]

[0180] where Dt is the data transmission volume, DI is the transmission distance, NB and NL are the network bandwidth and network delay, respectively;

[0181] Optimization objective 2: communication cost f2(x)

[0182] In the execution process of the service, the smaller the communication density between services, the smaller the same data communication cost, and the smaller the communication cost.

[0183]

[0184] where Cd is the communication density, l represents the number of data centers that can be passed through;

[0185] Optimization target 3: resource unit utilization rate f3(x)

[0186] For the case of dividing different units of resources such as CPU, memory, etc., the utilization rate of each resource unit is calculated respectively, and the average value of them is obtained to get the overall resource unit utilization rate, that is,

[0187]

[0188] Wherein, n r represents the number of resource units, u i represents the utilization rate of the i-th resource unit;

[0189] Define the target vector F = [f1(x), f2(x), -f3(x)], which represents that the target is to minimize the transmission time f1(x) and the communication cost f2(x), and to maximize the resource unit utilization rate f3(x);

[0190] Define the constraint condition:

[0191] Constraint condition 1:

[0192] R_total>=R_demand

[0193] Wherein, R_total represents the total amount of all available resources in the data center, and R_demand represents the amount of resources required by various services in the data center;

[0194] Constraint condition 2:

[0195] h(x)=max(0,t(x)-T)

[0196] Wherein, t(x) is the service completion time, and T is the service timeliness deadline. If t(x)>T, h(x) represents that the service completion time exceeds the timeliness deadline.

[0197] Preferably, the specific steps of using NSGA-III algorithm to realize multi-objective optimization to find the optimal matching are as follows:

[0198] 4.1: The input parameters are the computing power network resource list netList, the service hierarchy serviceHierarchy, and the demand matrix demandMatrix of the service;

[0199] 4.2: According to the order of service hierarchy, traverse each service, and according to the order of computing power network resource list netList, traverse each level of data center, check whether the data center resources meet the demand matrix of the current service;

[0200] For example:

[0201]

[0202]

[0203] 4.3: If the data center meets the demand matrix of the current service, use the NSGA-III function for optimization selection;

[0204] 4.4: Generate an initial population Pt, set parameters including population size and maximum number of iterations, etc.

[0205] For example:

[0206] Table 3 is the parameter setting table of this embodiment

[0207] Parameter Value Population size 300 Subpopulation size 100 Maximum number of iterations 300 Crossover probability 0.95 Mutation probability 0.05

[0208] 4.5: Improve the reference point selection strategy

[0209] 4.6: Cross and mutate Pt to generate Qt, and merge Pt and Qt to get Rt;

[0210] For example: Binary crossover algorithm (SBX) is used for crossover and polynomial mutation method is used for mutation.

[0211] ① For each pair of parent individuals, a cross point is randomly generated.

[0212] ② For each cross point, the distance between the two parent individuals is calculated.

[0213] ③ For the ith chromosome, the crossover probability p i is calculated, and two random numbers r1 and r2 are generated.

[0214] ④ If r1≤p i , perform simulated binary crossover.

[0215] ⑤ If r1>p i , the child individual is the same as the parent individual.

[0216] ⑥ Repeat the above steps until all chromosomes have been crossed.

[0217] Polynomial mutation method:

[0218] ① Randomly select an individual that needs to be mutated.

[0219] ② For each gene of the individual, decide whether to perform mutation operation on the gene according to the predefined mutation probability.

[0220] (5) If it is decided to mutate the gene, an integer k is randomly generated, and the polynomial mutation function f(x) = 0.2(2x - 1)k is calculated k , where ax is the current value of the gene.

[0221] (7) The result of the polynomial mutation function is taken as the new gene value, and is limited to the value range of the gene.

[0222] Steps (5) to (7) are repeated until all genes that need to be mutated have been mutated.

[0223] 4.7: Fast non-dominated sorting of Rt is performed;

[0224] 4.8: A suitable individual is selected to form a new offspring Pt+1 based on a reference point selection mechanism;

[0225] For example:

[0226]

[0227]

[0228] 4.9: Determine whether the maximum number of iterations has been reached. If so, proceed to step 4.10. If not, return to step 4.6;

[0229] 4.10: Select the optimal data center from the final population, and update the optimal data center to the mapping table mapping;

[0230] 4.11: Output the result as a list of optimal service and data center resource matching.

[0231] Example 6

[0232] A computing power network task scheduling method based on a "compute-store separation" architecture, as described in Example 5, except that step 4.5 improves the reference point selection strategy, the specific implementation steps are:

[0233] a. Use the DBSCAN clustering algorithm to generate an initial reference point set P, and randomly select a reference point p as a seed point in the reference point set P;

[0234] b. For each objective function, calculate the distance d between each reference point and the seed point, and arrange the distances in ascending order;

[0235]

[0236] where (x1, y1, z1) and (x2, y2, z2) are the coordinates of the reference point and the seed point, respectively.

[0237] c. For each reference point, calculate its weight w:

[0238] w[j] = 1 / (d[j] + 2)

[0239] where d[j] is the distance between the reference point and the seed point;

[0240] d. For each reference point, calculate its global density p':

[0241] p'[j] = 1 / (N * S[j])

[0242] where N is the size of the reference point set P, and S[j] is the range of the objective function where the reference point p[j] is located;

[0243] e. For each reference point, calculate its probability distribution p:

[0244] p[j] = (w[j] * p'[j]) / ∑(w[k] * p'[k])

[0245] where ∑ is the summation symbol, and k is the index of all reference points;

[0246] f. Randomly generate a random number r between 0 and 1, and calculate the cumulative probability distribution F[j] of each reference point:

[0247] F[j] = ∑(p[k l ])

[0248] where k l is the index of all reference points less than or equal to j;

[0249] g. Find the first reference point j that makes r less than or equal to F[j], and generate a new reference point q:

[0250] q[k] = p[j][k m ]

[0251] where k m is the index of all objective functions;

[0252] h. Add the new reference point q to the reference point set P.

[0253] Example 7

[0254] A computing power network task scheduling method based on the "storage-computing separation" architecture, as described in example 6, the difference is that the implementation process of the computing power network task scheduling method based on the "storage-computing separation" architecture:

[0255]

[0256] The implementation process of step 5 is:

[0257] After obtaining the list of optimal matches between the services and data centers obtained in step 4, the services are deployed to the selected data centers. To ensure high performance, availability of the services and to ensure compliance with service level agreement (SLA) requirements, it is necessary to monitor the performance indicators of the services in real time, such as response time, throughput, error rate, etc., and take timely measures to solve problems, thereby completing the optimal scheduling of the services and improving user experience and satisfaction.

[0258] For example:

[0259] Using the Prometheus open source monitoring system, the performance indicators of various services are monitored, and through the collection, storage and query of time series data, as well as the powerful query and visualization functions, users can better understand the performance status of service execution.

[0260] The above describes the preferred embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered within the scope of protection of the present application.

Claims

1. A computing power network task scheduling method based on a "compute-store separation" architecture, characterized in that, Firstly, a computing power network service scheduling model is established, including a service list, a data center resource list, a service pedigree establishment part, a service request analysis part, a resource unit division part, and a multi-objective optimizer. The service list includes m sub-services, and the data center resource list includes n data centers; The scheduling method includes the following steps: Step 1: initialize data and initialize computing power network environment; Step 2: submit a computing power network service composed of multiple sub-services, and deconstruct the computing power network service; Step 3: analyze the service request, determine the dependency and correlation of resource demand, and divide the resource demand into resource demand units; Step 4: use NSGA-III algorithm to realize multi-objective optimization and perform optimal matching; Step 5: complete scheduling according to the optimal matching; In step 4, the objective function and constraint condition are set as: Optimization objective 1: transmission time f1(x) In the service pedigree, there is data and information transmission between sub-services with front and rear execution relationship. When two sub-services are processed in different data centers, data transmission needs to pass through l data centers. In the execution process of the service, the higher the communication efficiency between data centers, the less the transmission time of information and data, then where x = (xl, x2,..., x l ) represents the sequence of data centers through which the service proceeds data and information transfer, the communication efficiency from data center x i to x i+1 is Where Dt is the data transmission amount, DI is the transmission distance, NB and NL are the network bandwidth and network delay, respectively; Optimization objective 2: communication cost f2(x) In the execution process of the service, the smaller the communication density between services, the smaller the same data communication cost, then wherein Cd is the communication density, is the network bandwidth cost and delay cost when communicating between adjacent data centers, and / denotes the number of data centers that can be passed through. Optimization objective 3: resource unit utilization rate f3(x) For different resource unit division cases, the utilization rate of each resource unit is calculated, and the average value of them is obtained to get the overall resource unit utilization rate, then wherein n r denotes the number of resource units, u i denotes the utilization of the i-th resource unit; Define the objective vector F = [f1(x), f2(x), -f3(x)], which represents that the objective is to minimize the transmission time f1(x) and the communication cost f2(x), and to maximize the resource unit utilization rate f3(x); Define the constraint condition: Constraint condition 1: R_total >= R_demand Where R_total represents the total amount of all available resources in the data center, and R_demand represents the resource amount required by various services in the data center; Constraint condition 2: h(x) = max(0, t(x) - T) Where t(x) is the service completion time, and T is the service timeliness deadline. If t(x) > T, h(x) represents that the service completion time exceeds the timeliness deadline.

2. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 1, characterized in that, Step 1 includes: 1.1: initialize data Data centers are represented as DC = [DC1, DC2, ..., DC] n ], where n represents the number of data centers; for each data center DC i DC i ={BI, CPU, GPU, VM, CO, RAM, HD, DB}, where BI, CPU, GPU, VM, CO, RAM, HD, and DB represent the basic information of the data center, CPU, GPU, virtual machine, container, memory, hard disk, and database, respectively; different attribute indicators are defined for each resource type in the data center to show the availability of the resources; 1.2: initialize computing power network Access data center resources through the API provided by the cloud service provider to realize data center access; obtain data center API access permission, consult API documentation, understand how to use API interface to obtain the required information; use API to call the corresponding resource interface to obtain the corresponding information; write the obtained information into the data center resource list to complete the initialization of the computing power network; 1.3: build computing power network topology structure Read the cloud service provider document or API to obtain the association or connection relationship information between data centers; test the network bandwidth and communication delay between each pair of data centers through the iperf tool; create a dictionary to describe the relationship between each pair of data centers, including three key-value pairs: connection relationship, network bandwidth, and communication delay; and draw the computing power network topology matrix according to the dictionary.

3. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 2, characterized in that, Step 2 is implemented by establishing a service lineage, and the specific process is as follows: 2.1: Read the service sequence document submitted by the user to obtain the predecessor and successor services of each sub-service, as well as their corresponding communication density information; 2.2: For each service, traverse each of its sub-services and treat them as nodes in a directed graph; 2.3: For each service, create an adjacency node list to store the information of its predecessor and successor services, and use the corresponding communication density as the weight of the directed graph edge; 2.4: Define a monitoring function called "monitor_service()" to monitor the communication density between sub-services in real time during service execution. If changes occur, the system will update the corresponding service lineage information in a timely manner.

4. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 3, characterized in that, Step 3 specifically includes: 3.1: Determine the dependency and association relationship of resource requirements, and divide the resource requirements into resource requirement units; Use a triple G(G1, G2, G3) to describe the resource requirement unit, where G1 represents the sub-service name, G2 represents the data center resource with dependency relationship, and G3 represents the resource requirement amount; 3.2: Divide the different resources of the data center into different resource units to realize the collaborative use and management of resources among various data centers; 3.3: Determine the resource requirement amount.

5. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 4, characterized in that, Step 3.3 specifically includes: A. Create an n x m two-dimensional matrix A, where each element of the matrix A is a list used to store various attribute requirements of resources; B. For each service i, iterate through its demand for each resource, adding its various attribute demands to A i,j where j is the index of the resource. C. The final A is the generated two-dimensional matrix: where A i,j denotes a list of various attribute requirements of the ith service to the jth resource.

6. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 5, characterized in that, The specific steps of using NSGA-III algorithm to realize multi-objective optimization to find the optimal matching are as follows: 4.1: Input parameters are the computing power network resource list netList, service lineage serviceHierarchy, and service demand matrix demandMatrix; 4.2: Traverse each service according to the order of the service lineage, and traverse each level of data center according to the order of the computing power network resource list netList, to check whether the data center resources meet the demand matrix of the current service; 4.3: If the data center meets the demand matrix of the current service, use the NSGA-III function to optimize the selection; 4.4: Generate an initial population Pt, and set parameters including population size and maximum iteration number; 4.5: Improve the reference point selection strategy 4.6: Perform crossover and mutation on Pt to generate Qt, and merge Pt and Qt to obtain Rt; 4.7: Perform fast non-dominated sorting on Rt; 4.8: Use the reference point-based selection mechanism to select suitable individuals to form a new offspring Pt+1; 4.9: Determine whether the maximum iteration number is reached. If it is reached, proceed to step 4.

10. If not, return to step 4.6; 4.10: Select the optimal data center from the final population, and update the optimal data center to the mapping table mapping; 4.11: Output the result as the service and data center resource optimal matching list.

7. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 6, characterized in that, Step 4.5 improves the reference point selection strategy implementation steps as follows: a. Use the DBSCAN clustering algorithm to generate an initial reference point set P, and randomly select a reference point p as a seed point in the reference point set P; b. For each objective function, calculate the distance d between each reference point and the seed point, and sort the distances in ascending order; Where (x1, y1, z1), (x2, y2, z2) are the coordinates of the reference point and the seed point respectively; c. For each reference point, calculate its weight w: w[j] = 1 / (d[j] + 2) Where d[j] is the distance between the reference point and the seed point; d. For each reference point, calculate its global density p': p'[j] = 1 / (N * S[j]) Where N is the size of the reference point set P, and S[j] is the range of the objective function where the reference point p[j] is located; e. For each reference point, calculate its probability distribution p: p[j] = (w[j] * p'[j]) / ∑(w[k] * p'[k]) Where ∑ is the summation symbol, and k is the index of all reference points; f. Randomly generate a random number r between 0 and 1, and calculate the cumulative probability distribution F[j] of each reference point: F[j] = ∑(p[k l ]) where k l is the index of all reference points less than or equal to j; g. Find the first reference point j that makes r less than or equal to F[j], and generate a new reference point q: q[k] = p[j][k m ] where k m is the index of all objective functions; h. Add the new reference point q to the reference point set P.

8. The computing power network task scheduling method based on the "storage-computing separation" architecture according to claim 7, characterized in that, The implementation process of step 5 is as follows: After obtaining the service and data center optimal matching list obtained in step 4, deploy the service to the selected data center, monitor the performance indicators of the service in real time, and take timely measures to solve the problem, thereby completing the optimal scheduling of the service and improving user experience and satisfaction.

Images