Computing power internet scheduling service method and system based on network performance comprehensive weight decision

Abstract

The present invention relates to the technical field of computing power resource scheduling, and provides a computing power Internet scheduling service method and system based on network performance comprehensive weight decision. On the basis of the resource requirements of a user, a computing power network service platform combines storage platforms and computing power platforms that are selected and do not belong to the same data center one by one to obtain a plurality of storage platform and computing power platform combinations; on the basis of values of different network performance metrics between the storage platform and the computing power platform in each combination, subjective weight vectors of different network performance metrics, objective weight vectors of different network performance metrics, and the resource requirements of the user, the computing power network service platform constructs an objective function under a storage and computing separation scenario or a data circulation scenario, and solves the function by means of an optimization algorithm to obtain an optimal storage platform and computing power platform combination that meets the requirements of the user; and the computing power network service platform deploys job and computing data submitted by the user to the optimal storage platform and computing power platform combination for storage and computing. The resource utilization rate and the task scheduling efficiency are improved.

Description

Computing power Internet scheduling service method and system based on network performance comprehensive weight decision Technical Field

[0001] The present invention relates to the technical field of computing power resource scheduling, and in particular to a computing power Internet scheduling service method and system based on network performance comprehensive weight decision-making. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] In the current digital age, the demand for computing and data processing continues to grow across all industries, with increasing requirements for computing power and storage capacity. With the dramatic increase in data volumes across scientific research, industrial production, and social welfare, the number of tasks such as large-scale numerical computation, big data analysis, and artificial intelligence data mining has also increased significantly. The diversity of computing scenarios has created diverse demands for computing power, storage, and network resources, and the demand for computing power capacity and quantity is also increasing. However, traditional single-type computing platforms often fail to meet the needs of complex applications. There is an urgent need to build a computing power network that can aggregate and coordinate the use of computing power platforms across various locations. Large-scale tasks, such as large-scale model training and scientific computing, require the support of big data, powerful algorithms, and high computing power. With trillions of training parameters, the coordinated computing of multiple computing platforms is imperative.

[0004] However, existing wide-area resource collaborative scheduling methods have some limitations when faced with complex and changing network environments and task requirements. These methods usually use static strategies or simple rules to allocate tasks, ignoring the differences in network paths and the heterogeneity of resources, and therefore cannot fully adapt to practical applications. This simple task allocation strategy may cause tasks to be assigned to network paths or resources with poor performance, thereby reducing the efficiency and performance of the system. For example, if a computing power platform is located at the edge of the network topology or at a network bottleneck node, then task allocation based on simple rules may cause tasks to encounter high latency or bandwidth bottlenecks during network transmission, affecting the completion time and quality of the task.

[0005] Summary of the Invention

[0006] In order to address the deficiencies of the prior art, the present invention provides a computing power Internet scheduling service method and system based on network performance comprehensive weight decision-making, which improves resource utilization and task scheduling efficiency.

[0007] On the one hand, a computing power Internet scheduling service method based on network performance comprehensive weight decision is provided, the method comprising:

[0008] The computing network service platform selects several storage platforms that meet the data storage requirements submitted by users. The data centers corresponding to the storage platforms that meet the storage requirements are called target group data centers.

[0009] The computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements. If so, it outputs the data center number. If not, it selects a computing power platform that meets the user's computing power requirements from non-target group data centers based on the user's computing power requirements.

[0010] The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several combinations of storage platforms and computing power platforms; and calculates the values ​​of different network performance indicators between the storage platform and computing power platform under each combination;

[0011] The computing network service platform determines the optimal combination of storage platform and computing power platform based on the values ​​of different network performance indicators between the storage platform and computing power platform in each combination; the computing network service platform deploys the jobs and computing data submitted by users to the optimal storage platform and computing power platform combination for storage and calculation.

[0012] On the other hand, it provides a computing power Internet scheduling service system based on the comprehensive weight decision of network performance;

[0013] The computing power Internet scheduling service system based on network performance comprehensive weight decision-making is applied to a computing network service platform, which receives resource requirements submitted by users. The computing network service platform is also connected to several data centers, each of which includes a computing power platform and a storage platform;

[0014] The Suanwang service platform receives the resource requirements submitted by the user and determines whether the user has sufficient quota. If so, it proceeds to the next step. If not, it returns a message indicating that the user has insufficient quota.

[0015] The computing network service platform selects several storage platforms that meet the data storage requirements submitted by users. The data centers corresponding to the storage platforms that meet the storage requirements are called target group data centers.

[0016] The computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements. If so, it outputs the data center number. If not, it selects a computing power platform that meets the user's computing power requirements from non-target group data centers based on the user's computing power requirements.

[0017] The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several combinations of storage platforms and computing power platforms; and calculates the values ​​of different network performance indicators between the storage platform and computing power platform under each combination;

[0018] The computing network service platform determines whether to enter the storage and computing separation scenario or the data flow scenario based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, thereby determining the optimal storage platform and computing power platform combination that meets user needs;

[0019] The computing network service platform deploys the jobs and computing data submitted by users to the optimal storage platform and computing power platform combination for storage and computing.

[0020] One of the above technical solutions has the following advantages or beneficial effects:

[0021] 1. Improve task completion efficiency: By comprehensively considering network performance indicators and task requirements, the system can dynamically assign tasks to computing platforms and network paths with better performance, avoiding the situation where tasks are assigned to resources with lower performance. This can effectively reduce task execution time and improve task completion efficiency.

[0022] 2. Optimize resource utilization: The system comprehensively considers the heterogeneity and performance differences of storage and computing resources across a wide area, rationally allocating tasks to different resources. This fully utilizes the computing power, storage capacity, and network bandwidth of each data center, improving resource utilization efficiency and avoiding resource waste.

[0023] 3. Provide flexible resource scheduling strategies: Based on a comprehensive weighted decision-making model, users can set weight parameters for different performance indicators and flexibly select scheduling scenarios based on task requirements and priorities. This allows users to implement personalized resource scheduling based on specific needs, meeting diverse requirements in different scenarios.

[0024] 4. Enhanced system scalability and adaptability: The system builds a computing power internet and computing power service platform to centrally manage and collaboratively utilize computing power resources distributed across a wide area. This provides a highly scalable system architecture that can adapt to growing computing needs and execute tasks of varying scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] FIG1 is a system architecture diagram of a first embodiment of the present invention;

[0027] FIG2 is a flow chart of a method according to a first embodiment of the present invention. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0029] Explanation of terms:

[0030] 1. Data Center: A data center is an IT infrastructure service center that includes a computing platform and a storage platform.

[0031] 2. Computing power platform: It has rich computing power resources and can provide a large amount of computing power for task calculations. It is mainly used to offload task calculations.

[0032] 3. Storage platform: It has rich storage resources and can provide storage space for user data. It is mainly responsible for the storage of computing data.

[0033] 4. Computing Internet: The computing Internet is a new type of information infrastructure that enables interconnection among cloud computing centers, intelligent computing centers, supercomputing centers, and other data centers across the country through the Internet. It brings together distributed computing power, storage, network, and other resources across the entire network for unified management and scheduling, provides diversified resources and services, improves task computing efficiency and data processing capabilities, and meets users' business needs.

[0034] 5. Multi-objective optimization algorithm (e.g., NSGA-III): This algorithm introduces the concept of evolutionary objective space decomposition. The basic idea is to transform the multi-objective optimization problem into multiple single-objective optimization sub-problems. By decomposing the objective space, the problem is transformed into multiple smaller-scale sub-problems for solution.

[0035] Example 1

[0036] This embodiment provides a computing power Internet scheduling service method based on network performance comprehensive weight decision;

[0037] As shown in FIG1 and FIG2, a computing power Internet scheduling service method based on network performance comprehensive weight decision-making includes:

[0038] S101: The computing network service platform receives resource requirements submitted by the user. The computing network service platform determines whether the user has sufficient quota based on the resource requirements. If the user has sufficient quota, the platform proceeds to S102. If the user does not have sufficient quota, the platform returns a quota insufficient message to the user.

[0039] S102: The computing network service platform selects a number of storage platforms that meet the data storage requirements submitted by the user. The data centers corresponding to the storage platforms that meet the storage requirements are referred to as target group data centers.

[0040] S103: The computing network service platform determines whether there is a computing platform in the target group data center that meets the user's computing power requirements. If so, the computing network service platform outputs the number of the data center. If not, the computing network service platform selects a computing platform that meets the user's computing power requirements from non-target group data centers based on the user's computing power requirements.

[0041] S104: The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain a plurality of combinations of storage platforms and computing power platforms; and calculates the values ​​of different network performance indicators between the storage platform and the computing power platform for each combination;

[0042] S105: The computing network service platform determines whether to enter a storage and computing separation scenario or a data flow scenario based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, thereby determining the optimal storage platform and computing power platform combination that meets user needs;

[0043] S106: The computing network service platform deploys the jobs and computing data submitted by users to the optimal storage platform and computing power platform combination for storage and computing.

[0044] In this solution, an application process of the above S101 to S106 is specifically as follows:

[0045] 1. Determine requirements

[0046] First, it is necessary to clarify the user's needs, including but not limited to storage requirements, computing requirements, data access frequency, data security, scalability and cost budget.

[0047] 2. Select a storage platform

[0048] Based on the determined requirements, the process from S101 to S106 described above is used to select a suitable storage platform (a storage platform in the optimal storage platform and computing power platform combination). Common storage platforms may include:

[0049] Object storage: It is easy to use, high-performance, and low-cost, and is suitable for storing and managing large amounts of unstructured data.

[0050] NFS file storage: It is shared, transparent, and high-performance, and is suitable for distributed file storage that requires sharing and accessing files.

[0051] HDFS file storage: A distributed file system with scalability, fault tolerance, and high throughput, suitable for storing large-scale data sets and processing big data.

[0052] The selected storage platform may be one of the common storage platforms mentioned above.

[0053] 3. Choose a computing power platform

[0054] Based on computing requirements, select an appropriate computing platform (the computing platform from the optimal storage platform and computing platform combination) using the processes described above from S101 to S106. Common computing platforms include:

[0055] Cloud computing platform: suitable for dynamic resource expansion and pay-as-you-go computing needs.

[0056] Intelligent computing cluster: Suitable for large-scale training and inference computing scenarios of artificial intelligence such as voice, image, and video.

[0057] High-performance computing (HPC) clusters: Suitable for scenarios requiring large amounts of computing resources, such as scientific computing and engineering computing.

[0058] The selected computing power platform can be one of the common computing power platforms mentioned above.

[0059] 4. Synergy between data storage and computing

[0060] When determining the optimal combination of storage and computing platforms, it is also necessary to ensure synergy between the storage and computing platforms to improve efficiency and performance. This can be addressed from the following aspects:

[0061] Data transfer: Ensures efficient data transfer between storage platforms and computing platforms. Computing network service platforms typically provide data transfer services.

[0062] Data access: Choose a data access method that supports high throughput and low latency. For example, place the storage platform and computing power platform in the same area to reduce data transmission delays and costs.

[0063] Data security: Ensure the security of data during transmission and storage, using encryption and access control mechanisms.

[0064] 5. Optimize performance

[0065] During the application process, optimization strategies can also be used to improve the storage and computing performance of the optimal storage platform and computing power platform combination. This can be addressed from the following aspects:

[0066] Caching: Use a caching mechanism on the computing platform to reduce frequent access to the storage platform. For example, use Redis or Memcached to cache frequently used data.

[0067] Data partitioning: Data is partitioned for storage and calculation to improve parallel processing capabilities.

[0068] Resource management: Use containerization and orchestration tools such as Docker and Kubernetes to efficiently manage computing resources.

[0069] 6. Monitoring and Management

[0070] During the application process, the performance of the optimal storage platform and computing power platform combination can be continuously monitored and optimized and adjusted. This can be done from the following aspects:

[0071] Monitoring tools: Use tools such as Prometheus to monitor resource usage and performance.

[0072] Automatic scaling: Dynamically adjusts computing and storage resources based on load conditions to ensure efficient operation and cost control.

[0073] After the above processing, the jobs and computing data submitted by users of the computing network service platform can be deployed to the optimal storage platform and computing power platform combination for storage and calculation.

[0074] Furthermore, all data centers constitute the computing power internet, and non-target group data centers refer to data centers in the computing power internet other than the target group data centers.

[0075] As shown in Figure 1, the computing network service platform includes a database, a global scheduling system, and a job execution system; the database is used to store key data, such as user, resource, and job information;

[0076] The global scheduling system mainly provides offloading strategies for tasks and computing data, and calculates the optimal storage platform and computing power platform;

[0077] The job execution system mainly calculates the optimal storage platform and computing power platform based on the global scheduling system to execute tasks and data calculations. The computing power platform has abundant computing power resources and can provide a large amount of computing power for task calculations, mainly used to offload task calculations;

[0078] The storage platform has abundant storage resources and can provide storage space for user data, and is mainly used for storing data. Let the data center be D={D1, D2, ..., D i}, where D i ={(S1,C1),(S2,C2),...,(S n ,C n )}, where S1, S2, ..., S n For storage platforms, C1, C2, ..., Cn The computing power platform, storage platform storage type storetype, the remaining storage size is storesize surplus , the storage platform is S i , the remaining memory size of the computing power platform is memory surplus , the remaining number of CPUs is cpu surplus , the computing power platform is C i ;

[0079] Let the user's unique identifier be ID and the user's memory quota be memory quota , the cpu quota is cpu quota , the type of user storage quota is storetype quota , the size of the user storage quota is storesize quota The memory size requested by the user is memory request , the number of CPUs is CPU request ; The storage type requested by the user is storetype request , calculate the data size as storesize request .

[0080] Furthermore, S101: the computing network service platform receives resource requirements submitted by users, wherein the resource requirements include: virtual machine CPU memory capacity and core number, number of virtual machines, storage capacity, whether location priority, whether network priority, whether capacity priority, and whether low price priority, etc.

[0081] Furthermore, the S101: computing network service platform determines whether the user has sufficient quota based on the resource demand, specifically including: if the user's memory quota is greater than the memory size requested by the user, the user's CPU quota is greater than the number of CPUs requested by the user, the user's storage quota type meets the storage quota type requested by the user, and the storage size of the user's storage quota is greater than the size of the computing data requested by the user, then it indicates that the user has sufficient quota, otherwise it indicates that the user does not have sufficient quota.

[0082] For example, the computing network service platform receives the user's resource requirements (cpu request ,memory request ,storetype request ,storesize request ), determine whether the user has sufficient quota based on the user's information ID, that is:

[0083] memory quota >memory request ,

[0084] cpuquota >cpu request ,

[0085] storetype quota =storetype request ,

[0086] storesize quota >storesize request ,

[0087] If the conditions are met, proceed to the next step. If not, the user quota is insufficient information is returned.

[0088] Furthermore, in step S102, the computing network service platform selects a plurality of storage platforms that meet the data storage requirements according to the data storage requirements submitted by the user, wherein the storage platforms that meet the storage requirements are:

[0089] A storage platform whose storage type meets user requests and whose storage capacity exceeds user data storage requirements.

[0090] For example, the computing network service platform receives the storage type required by the user's computing data as storetype request , calculate the data size as storesize request , according to storetype=storetype request , storesize surplus >storesize request The storage platform that meets the calculation data is selected as S a ={S a1 ,S a2 ,...,S an}, the data center where the storage platform is located is D a ={D a1 ,D a2 ,...,D an}.

[0091] Furthermore, the step S103: the computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements, including:

[0092] If the remaining memory size of the computing power platform is greater than the memory required by the user's computing power, and the number of remaining CPUs of the computing power platform is greater than the number of CPUs required by the user, then the target group's data center has a computing power platform that meets the user's computing power requirements. Otherwise, it means that there is no computing power platform that meets the user's computing power requirements.

[0093] Further, S103: If not, based on the user's computing power requirements, a computing power platform that meets the computing power requirements is screened out from the data centers of the non-target group, where the data centers of the non-target group refer to the remaining data centers in all data centers except the target group data center.

[0094] It should be understood that the global scheduling system of the computing network service platform includes a filtering layer and a decision layer. The filtering layer performs screening and selects all suitable computing power platforms C in other data centers according to the computing power requirements of user information. a ={C a1 ,C a2 ,...,C an}, that is, to meet the memory surplus >memory request , cpu surplus >cpu request If there is no computing power platform, it means that the current computing network system does not have a suitable computing power platform. If there is, all suitable computing power platforms C are returned. a ={C a1 ,C a2 ,...,C an} and the corresponding data center D a ={D a1 ,D a2 ,...,D an}.

[0095] Furthermore, S104: the computing network service platform combines the screened storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several groups of storage platform and computing power platform combinations; calculates the values ​​of different network performance indicators between the storage platform and the computing power platform under each combination, where the different network performance indicators include: network bandwidth, network latency, network jitter and network packet loss rate between the storage platform and the computing power platform.

[0096] For example, the filtering layer selects combinations of storage and computing platforms that do not belong to the same data center. These selected combinations will be sent to the decision layer of the global scheduling system of the computing network service platform for further processing and decision-making.

[0097] Furthermore, the S104: calculating the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination specifically includes:

[0098] Periodically calculate the computing power platform C that is not in the same data center a ={C a1 ,C a2 ,...,C an} and storage platform S a ={Sa1 ,S a2 ,...,S an}The network bandwidth between two pairs is B={B1,B2,...,B n}、Network delay L={L1,L2,...,L n}、Network jitter J={J1,J2,...,J n} and network packet loss rate P={P1,P2,...,P n} data, each computing power platform C a ={C a1 ,C a2 ,...,C an} and storage platform S a ={S a1 ,S a2 ,...,S an}Test multiple data between two pairs;

[0099] The acquired network performance data is preprocessed using the truncated mean algorithm: first, the data is sorted in descending order; second, the maximum and minimum values, which are 2% of the total data, are removed; and finally, the average of the remaining values ​​is calculated as the final network performance data; the final network performance data is written into the computing network service platform database.

[0100] Table 1 Network performance data

[0101] Furthermore, the S105: computing network service platform determines the best storage platform and computing power platform combination according to the values ​​of different network performance indicators between the storage platform and computing power platform in each combination, including:

[0102] The computing network service platform constructs the objective function for storage and computing separation scenarios or data flow scenarios based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, the subjective weight vectors of different network performance indicators, the objective weight vectors of different network performance indicators, and the user's resource requirements. The objective function can be solved based on the optimization algorithm to obtain the optimal storage platform and computing power platform combination that meets user needs.

[0103] Furthermore, the S105: subjective weight vectors of different network performance indicators includes: calculating the subjective weight vectors of different network performance indicators according to a subjective weight decision algorithm.

[0104] Furthermore, the subjective weight vectors of different network performance indicators in S105 specifically include:

[0105] S105-11: Constructing the AHP hierarchy: Identify the goals and criteria of the decision problem and organize them into a hierarchical structure;

[0106] There are two levels: the first level is to select the storage platform and computing power platform, and the second level is the network quality indicators, including: bandwidth, latency, jitter, and packet loss rate;

[0107] S105-12: Compare the relative importance of indicators: For each pair of criteria in the hierarchy, perform pairwise comparisons to determine their relative importance. Use a scale of 1-9, where 1 indicates equal importance, 3 indicates slight difference in importance, 5 indicates moderate difference in importance, 7 indicates strong difference in importance, and 9 indicates extreme difference in importance. Fill in the comparison matrix based on expert judgment or experience.

[0108] Table 2 Expert experience

[0109] S105-13: Construct comparison matrix: Construct comparison matrix A, where a ij Indicates the importance of criterion i relative to criterion j; the comparison matrix for j network quality indicators is expressed as:

[0110] S105-14: Normalize the comparison matrix to ensure that the sum of each column is 1: First, for each column, calculate the sum of the elements in the column:

[0111] Divide each element by the sum of the corresponding column to obtain a normalized comparison matrix:

[0112] S105-15: Calculate weight vector: Calculate the average value of each row of the normalized comparison matrix to obtain the subjective weight vector V j =(V1,V2,V3,V4);

[0113] S105-16: Calculate the consistency ratio (CR) to test the consistency: if the CR is less than 0.1, the matrix is ​​judged to have reasonable consistency.

[0114] Furthermore, the S105: objective weight vectors of different network performance indicators includes: calculating the objective weight vectors of different network performance indicators according to an entropy method.

[0115] Furthermore, the objective weight vectors of different network performance indicators in S105 specifically include:

[0116] S105-21: Standardized Data: First, the raw data is standardized to unify the value ranges of different indicators. Two normalization methods, positive and negative indicators, are used to map the value of each indicator to a range between 0 and 1. The standardization formula is as follows:

[0117] When the indicator is positive:

[0118] When the indicator is negative:

[0119] Among them, X′ ij is the normalized value, X ij is the original value, and are the minimum and maximum values ​​of index j respectively;

[0120] S105-22: Calculate the probability matrix P ij :

[0121] S105-23: Calculate the entropy value of each performance indicator, entropy value E j The calculation formula is as follows:

[0122] Among them, E j is the entropy value of indicator j, n is the number of candidate solutions, P ij is a probability matrix;

[0123] S105-24: Calculate the weight of the indicator. The formula for calculating the weight of the indicator is as follows:

[0124] Among them, W j is the weight of indicator j, m is the number of performance indicators;

[0125] Finally, the objective weight vector W of each performance indicator is obtained j =β=(β i , β2, β3, β4) T .

[0126] Furthermore, the S105 further includes:

[0127] By establishing a least squares objective function to construct a comprehensive weight decision model, the comprehensive weight vector formula of the performance index is obtained:

[0128] F j =λV j +(1-λ)W j (9)

[0129] F j =(λV1+(1-λ)W1, λV2+(1-λ)W2, λV3+(1-λ)W3, λV4+(1-λ)W4) (10)

[0130] Among them, F jis the comprehensive weight of the jth network performance indicator, 0<λ<1 is the weighting factor of the subjective and objective preference coefficients, and the computing network service platform can use the comprehensive weight vector formula to weight the weight vector.

[0131] Furthermore, S105 specifically includes: the computing network service platform determines the optimal storage platform and computing power platform combination that meets the user's needs based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, the subjective weight vectors of the different network performance indicators, the objective weight vectors of the different network performance indicators, and the user's resource requirements. This process also includes: determining whether to enter a storage and computing separation scenario or a data flow scenario. After determining which scenario to enter, a corresponding objective function is constructed based on the determined scenario, and the objective function is solved to obtain the optimal storage platform and computing power platform combination that meets the user's needs.

[0132] The criteria for determining whether to enter the storage and computing separation scenario or the data flow scenario are:

[0133] Obtain the network validity threshold H and real network validity K of the computing network service platform j The comparison results between:

[0134] When the network validity threshold H of the computing network service platform is less than or equal to the actual network validity K j , enter the storage and computing separation scenario.

[0135] When the network validity threshold H of the computing network service platform is greater than the actual network validity K j , enter the data flow scene.

[0136] The network effectiveness threshold H of the computing network service platform is calculated by statistically analyzing bandwidth, delay, jitter, packet loss rate, and time to form a normal distribution curve. The maximum slope value in the curve is used as the threshold.

[0137] Real network effectiveness K j , the specific formula is:

[0138] Among them, B r 、D r 、J r 、P r They are the actual values ​​of bandwidth, delay, jitter, and packet loss rate of the network performance indicators between the current storage platform and the computing power platform, respectively. max 、D max 、J max 、P max They are respectively the maximum values ​​of the network performance indicators width, delay, jitter, and packet loss rate between the current storage platform and the computing power platform recorded by the computing network service platform.

[0139] Among them, 0<λ<1 is the weighting factor of subjective and objective preference coefficients, W1, W2, W3 and W4 are objective weight vectors, and V1, V2, V3 and V4 are subjective weight vectors.

[0140] Prioritize screening of all storage platform and computing platform combinations that meet the conditions to enter the storage and computing separation scenario. The condition is that the network effectiveness threshold H of the computing network service platform is less than or equal to the actual network effectiveness K between the storage platform and the computing platform. j If no combination meets the conditions for entering the storage and computing separation scenario, the data flow scenario will be entered.

[0141] Furthermore, the objective function in the storage and computing separation scenario is constructed and solved to obtain the optimal storage platform and computing power platform combination that meets user needs, specifically including:

[0142] S105-31: Determine whether the user has selected a single option or multiple options among the four options of location priority, network priority, capacity priority, and low price priority. If the user has selected a single option, that is, one of the three options is selected, the process proceeds to S105-32. If the user has selected multiple options, that is, multiple options are selected from the four options of location priority, network priority, capacity priority, and low price priority, the process proceeds to step S105-33.

[0143] S105-32: If a single selection is made, a corresponding single-objective optimization function is constructed and solved, and the data center where the corresponding storage platform and computing power platform are located is output. The storage platform and computing power platform are then determined as the optimal storage platform and computing power platform combination:

[0144] a) Location Priority: Construct and solve a single-objective optimization function corresponding to location priority, that is, calculate the Euclidean distance of all combinations of computing power platforms and storage platforms that meet user requirements;

[0145] Among them, l f represents a location-prioritized combination of computing power and storage platforms, x compute and y compute Indicates the latitude and longitude of the city where the computing power platform is located, which is a two-dimensional row vector, x storage and y storage Indicates the latitude and longitude of the city where the storage platform is located, as a two-dimensional row vector;

[0146] b) Network Priority: Construct and solve a single-objective optimization function corresponding to network priority, that is, calculate the network performance parameters between all combinations of computing power platforms and storage platforms;

[0147] Among them, the delay T delay , bandwidth T tp , packet loss rate T lr , jitter Tdv , is a four-dimensional row vector representing the network performance parameters between the storage platform and the computing power platform, i f Represents a network-first computing platform and storage platform combination;

[0148] c) Capacity Priority: Construct and solve a single-objective optimization function corresponding to capacity priority. This function obtains the remaining resource capacity of all combinations of computing power platforms and storage platforms. For example, the computing power platform selects the one with the largest remaining capacity (remaining CPU and remaining memory); the storage platform selects the one with the largest remaining storage capacity.

[0149] Among them, CPU avaiable Indicates the remaining effective CPU amount of the computing power platform, MEM avaiable Indicates the remaining effective memory of the computing power platform, v f Indicates a capacity-first computing platform;

[0150] For example, the storage platform selects the one with the largest remaining capacity;

[0151] Among them, STMEM avaiable Indicates the remaining available storage capacity of the storage platform, v g Indicates a capacity-first storage platform;

[0152] d) Low price priority: Construct and solve a single-objective optimization function corresponding to low price priority. This function calculates all combinations of computing power platforms and storage platforms and selects the one with the lowest computing power platform price.

[0153] Among them, p f Indicates a computing power platform that prioritizes low prices. Indicates the unit price of memory in the computing power platform. Indicates the CPU unit price of the computing power platform, and m represents time (hours);

[0154] Calculate all combinations of computing power platforms and storage platforms and select the one with the lowest storage platform price;

[0155] Among them, p g Indicates a storage platform that prioritizes low prices. Indicates the storage unit price of the storage platform, and m represents time (hours);

[0156] S105-33: If multiple selections are made, a corresponding multi-objective optimization function is constructed and solved, and the optimal storage platform and computing power platform combination is output:

[0157] a) Location- and network-priority: Combining location and network information, a multi-objective optimization function is constructed and solved;

[0158] Among them, l if represents a combination of computing power platform and storage platform with location priority and network priority, x compute and y compute Indicates the latitude and longitude of the city where the computing power platform is located, which is a two-dimensional row vector, x storage and y storage Represents the longitude and latitude of the city where the storage platform is located, which is a two-dimensional row vector; the delay T delay , bandwidth T tp , packet loss rate T lr , jitter T dv , is a four-dimensional row vector, representing the network performance parameters between the storage platform and the computing power platform;

[0159] b) Location and capacity priority: Combine location and capacity requirements to construct and solve a multi-objective optimization function;

[0160] Among them, l vf represents a combination of computing power platform and storage platform with location priority and capacity priority, x compute and y compute Indicates the latitude and longitude of the city where the computing power platform is located, which is a two-dimensional row vector, x storage and y storage Indicates the latitude and longitude of the city where the storage platform is located, which is a two-dimensional row vector; CPU avaiable Indicates the remaining effective CPU amount of the computing power platform, MEM avaiable Indicates the remaining effective memory of the computing power platform, STMEM avaiable Indicates the remaining available storage capacity of the storage platform;

[0161] c) Location + Low Price Priority: Combine location and low price requirements to construct and solve a multi-objective optimization function;

[0162] Among them, l pf represents a combination of computing power platform and storage platform with location priority and capacity priority, x compute and y compute Indicates the latitude and longitude of the city where the computing power platform is located, x storage and y storage Indicates the latitude and longitude of the city where the storage platform is located, as a two-dimensional row vector; Indicates the unit price of memory in the computing power platform. Indicates the CPU unit price of the computing power platform, Indicates the storage unit price of the storage platform, and m represents time (hours);

[0163] d) Network + Capacity Priority: Combine network and capacity requirements to construct and solve a multi-objective optimization function;

[0164] Among them, i vf It represents a combination of computing power platform and storage platform with location priority and capacity priority, and the delay T delay , bandwidth T tp , packet loss rate T lr , jitter T dv , is a four-dimensional row vector, representing the network performance parameters between the storage platform and the computing power platform; CPU avaiable Indicates the remaining effective GPU amount of the computing power platform, MEM avaiable Indicates the remaining effective memory of the computing power platform, STMEM avaiable Indicates the remaining available storage capacity of the storage platform;

[0165] e) Network + Low Price Priority: Combine network and low price requirements to build and solve a multi-objective optimization function;

[0166] Among them, i pf Indicates a combination of computing power platform and storage platform with network priority and low price priority, with latency T delay , bandwidth T tp , packet loss rate T lr , jitter T dv , is a four-dimensional row vector, representing the network performance parameters between the computing power platform and the storage platform; Indicates the unit price of memory in the computing power platform. Indicates the CPU unit price of the computing power platform, Indicates the storage unit price of the storage platform, and m represents time (hours);

[0167] f) Capacity + Low Price Priority: Combine capacity and low price requirements to construct and solve a multi-objective optimization function;

[0168] Among them, v pf Represents a combination of computing power and storage platforms that prioritizes capacity and low price; CPU avaiable Indicates the remaining effective CPU amount of the computing power platform, MEM avaiable Indicates the remaining effective memory of the computing power platform, STMEM avaiable Indicates the remaining available storage capacity of the storage platform; Indicates the unit price of memory in the computing power platform. Indicates the CPU unit price of the computing power platform, Indicates the storage unit price of the storage platform, and m represents time (hours);

[0169] For example, suppose that the storage platform S is obtained by calculating the multi-objective optimization function (location priority and network priority) a =S1 (belonging to data center D1) and computing power platform C a =C5 (belonging to data center D5) combination is the most suitable.

[0170] Furthermore, the multi-objective optimization function in the data flow scenario is constructed and solved to obtain the optimal storage platform and computing power platform combination that meets user needs, specifically including:

[0171] S105-41: Obtain a combination of storage platforms and computing platforms that meets the user's resource requirements, referred to as an alternative combination;

[0172] S105-42: Determine whether the number of candidate combinations exceeds one. If so, proceed to the next step S105-44; if not, output the number information of the unique candidate combination, and determine the unique candidate combination as the optimal storage platform and computing power platform combination;

[0173] S105-43: Calculate the data transfer time from the storage platform to the computing power platform in each combination, add the priority strategy requirements selected by the user to the corresponding objective function, solve the objective function, and output the optimal storage platform and computing power platform combination, wherein the priority strategy is at least one of the four options selected by the user: location priority, network priority, capacity priority, and low price priority. How to determine the corresponding objective function based on the priority strategy selected by the user has been described above and will not be repeated here.

[0174] Furthermore, S105-43: the requirements of the priority strategy selected by the user are added to the corresponding objective function. If the objective function is a multi-objective optimization function, the objective optimization function can be solved based on the multi-objective optimization algorithm to output the optimal storage platform and computing power platform combination.

[0175] The multi-objective optimization algorithm may be NSGA-III.

[0176] The decision layer sends the output results after entering the scene to the job execution system.

[0177] The job execution system in the computing network service platform executes tasks based on the output results of the decision layer of the global scheduling system.

[0178] The job execution system deploys the user's jobs and computing data to the appropriate storage and computing platforms for execution, based on the optimal combination of storage and computing platforms selected by the global scheduling system. Simultaneously, the job execution system monitors the progress and status of the job execution and provides real-time feedback to the user.

[0179] The present invention realizes the centralized scheduling of computing resources distributed in a wide area network. By establishing a unified resource scheduling service system, computing resources in different geographical locations and network environments can be effectively scheduled.

[0180] This invention provides an intelligent resource scheduling algorithm that can dynamically select the optimal resource combination for task scheduling based on user needs, the status and performance indicators of each computing resource. This can maximize the utilization of computing resources and improve the efficiency of task execution.

[0181] The present invention also includes a performance optimization mechanism for wide area networks. By analyzing and evaluating network performance indicators across different storage and computing platform combinations, the system can select combinations with lower network latency and greater bandwidth, thereby reducing communication delays during task execution and improving data transmission efficiency.

[0182] Example 2

[0183] This embodiment provides a computing power Internet scheduling service system based on network performance comprehensive weight decision-making;

[0184] The computing power Internet scheduling service system based on network performance comprehensive weight decision-making is applied to a computing network service platform, which is connected to several data centers, each of which includes a computing power platform and a storage platform;

[0185] The Suanwang service platform receives the submitted resource requirements and determines whether the user has sufficient quota. If so, it proceeds to the next step. If not, it returns a message indicating that the user has insufficient quota.

[0186] The computing network service platform selects several storage platforms that meet the data storage requirements submitted by users. The data centers corresponding to the storage platforms that meet the storage requirements are called target group data centers.

[0187] The computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements. If so, it outputs the data center number. If not, it selects a computing power platform that meets the user's computing power requirements from non-target group data centers based on the user's computing power requirements.

[0188] The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several combinations of storage platforms and computing power platforms; and calculates the values ​​of different network performance indicators between the storage platform and computing power platform under each combination;

[0189] The computing network service platform constructs an objective function for storage-computing separation scenarios or data flow scenarios based on the values ​​of different network performance indicators between the storage platform and computing power platform in each combination, the subjective weight vectors of different network performance indicators, the objective weight vectors of different network performance indicators, and the user's resource requirements. It solves the objective function and obtains the optimal storage platform and computing power platform combination that meets user needs.

[0190] The computing network service platform deploys the jobs and computing data submitted by users to the optimal storage platform and computing power platform combination for storage and computing.

[0191] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A computing power Internet scheduling service method based on network performance comprehensive weight decision, characterized by: include: The computing network service platform selects several storage platforms that meet the storage requirements based on the data storage requirements submitted by the user. The data centers corresponding to the storage platforms that meet the storage requirements are called target group data centers. The computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements. If so, it outputs and returns the number of the data center; if not, it selects a computing power platform that meets the user's computing power requirements from the non-target group data center based on the user's computing power requirements; The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several groups of storage platform and computing power platform combinations; calculates the values ​​of different network performance indicators between the storage platform and the computing power platform under each combination; The computing network service platform determines the best combination of storage platform and computing power platform based on the values ​​of different network performance indicators between the storage platform and computing power platform in each combination; the computing network service platform deploys the jobs and computing data submitted by users to the best combination of storage platform and computing power platform for storage and calculation.

2. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 1 is characterized in that: The computing network service platform determines the best combination of storage platform and computing power platform based on the values ​​of different network performance indicators between the storage platform and computing power platform under each combination, including: The computing network service platform constructs the objective function in the storage and computing separation scenario or the data flow scenario based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, the subjective weight vectors of different network performance indicators, the objective weight vectors of different network performance indicators, and the user's resource requirements, and solves the objective function to obtain the optimal storage platform and computing power platform combination that meets user needs.

3. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 1 is characterized in that: The computing network service platform further includes: The computing network service platform receives the resource requirements submitted by the user, and determines whether the user has sufficient quota based on the resource requirements. If so, the platform proceeds to the next step; if not, the platform returns information indicating that the user has insufficient quota.

4. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 3 is characterized in that: The computing network service platform determines whether the user has sufficient quota based on the resource demand, including: If the user's memory quota is larger than the memory size requested by the user, the user's CPU quota is larger than the number of CPUs requested by the user, the user's storage quota type meets the storage quota type requested by the user, and the storage size of the user's storage quota is larger than the size of the computing data requested by the user, it means that the user has sufficient quota; otherwise, it means that the user does not have sufficient quota.

5. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 1 is characterized in that: The computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements, including: If the remaining memory size of the computing power platform is larger than the memory required by the user's computing power, and the number of remaining CPUs of the computing power platform is greater than the number of CPUs required by the user's computing power, it means that the target group data center has a computing power platform that meets the user's computing power requirements. Otherwise, it means that there is no computing power platform that meets the user's computing power requirements.

6. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 2 is characterized in that: The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several groups of storage platform and computing power platform combinations; Calculate the values ​​of different network performance indicators between the storage platform and the computing power platform under each combination; the different network performance indicators include: network bandwidth, network latency, network jitter, and network packet loss rate between the storage platform and the computing power platform; The subjective weight vectors of different network performance indicators include: calculating the subjective weight vectors of different network performance indicators according to a subjective weight decision algorithm; The objective weight vectors of different network performance indicators include: calculating the objective weight vectors of different network performance indicators according to an entropy method.

7. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 2 is characterized in that: The computing network service platform constructs an objective function in a storage and computing separation scenario or a data flow scenario according to the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, the subjective weight vectors of different network performance indicators, the objective weight vectors of different network performance indicators, and the user's resource requirements, and solves the objective function to obtain the best storage platform and computing power platform combination that meets the user's needs, including: The computing network service platform determines whether to enter the storage and computing separation scenario or the data flow scenario based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, the subjective weight vectors of different network performance indicators, the objective weight vectors of different network performance indicators, and the user's resource requirements; According to the scenario judged to be entered, an objective function corresponding to the scenario is constructed, and the objective function is solved to obtain the best storage platform and computing power platform combination that meets the user's needs; The basis for judging whether to enter the storage and computing separation scenario or the data flow scenario is: Obtain the network validity threshold H and real network validity K of the computing network service platform j The comparison results between: When the network validity threshold H of the computing network service platform is less than or equal to the actual network validity K j , enter the storage and computing separation scenario; When the network validity threshold H of the computing network service platform is greater than the actual network validity K j , enter the data flow scene.

8. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 2 is characterized in that: Construct the objective function in the storage and computing separation scenario, solve the objective function, and obtain the best storage platform and computing power platform combination that meets user needs, including: Determine whether the user selects one or more of the four options: location priority, network priority, capacity priority, and low price priority; If the user selects one from location priority, network priority, capacity priority, and low price priority, then the corresponding single-objective optimization function is constructed and solved, and the data center where the corresponding storage platform and computing power platform are located is output, and the storage platform and computing power platform are determined as the optimal storage platform and computing power platform combination; If the user selects multiple strategies from the four strategies of location priority, network priority, capacity priority, and low price priority, the corresponding multi-objective optimization function is constructed and solved to output the optimal combination of storage platform and computing power platform.

9. The computing power Internet scheduling service method based on network performance comprehensive weight decision as claimed in claim 2 is characterized in that: The objective function in the data flow scenario is constructed, and the objective function is solved to obtain the best storage platform and computing power platform combination that meets the user's needs, specifically including: Obtain a storage platform and computing power platform combination that meets the user's resource requirements as an alternative combination, determine whether the number of alternative combinations exceeds one, and if so, proceed to the next step; if not, output the number information of the unique combination, and determine the unique combination as the optimal storage platform and computing power platform combination; Calculate the data transfer time from the storage platform to the computing power platform in each combination, add the priority strategy requirements selected by the user to the corresponding objective function, solve the objective function, and output the optimal storage platform and computing power platform combination. The priority strategy is at least one of the four options selected by the user: location priority, network priority, capacity priority, and low price priority.

10. A computing power Internet scheduling service system based on network performance comprehensive weight decision-making, characterized by: The computing power Internet scheduling service system based on network performance comprehensive weight decision is applied to a computing network service platform, and the computing network service platform is connected to several data centers, each of which includes a computing power platform and a storage platform; The Suanwang service platform receives the resource requirements submitted by the user and determines whether the user has sufficient quota. If yes, it proceeds to the next step; if not, it returns the information that the user has insufficient quota. The computing network service platform selects several storage platforms that meet the storage requirements according to the user's data storage requirements; the data centers corresponding to the several storage platforms that meet the storage requirements are called target group data centers; The computing network service platform determines whether the target group data center has a computing power platform that meets the user's computing power requirements. If so, it outputs and returns the number of the data center; if not, it selects a computing power platform that meets the user's computing power requirements from the non-target group data center based on the user's computing power requirements; The computing network service platform combines the selected storage platforms and computing power platforms that do not belong to the same data center one by one to obtain several groups of storage platform and computing power platform combinations; calculates the values ​​of different network performance indicators between the storage platform and the computing power platform under each combination; The computing network service platform determines whether to enter the storage and computing separation scenario or the data flow scenario based on the values ​​of different network performance indicators between the storage platform and the computing power platform in each combination, so as to determine the best storage platform and computing power platform combination that meets user needs; The computing network service platform deploys the jobs and computing data submitted by users to the optimal storage platform and computing power platform combination for storage and computing.

Images